Confirmed. Reducing...

Created attachment 35158 [details]
reduced testcase

markus@x4 tmp % g++ -std=gnu++1y -flto -g -O2 -r -nostdlib tcp_client.ii
lto1: internal compiler error: Segmentation fault
0xa2766f crash_signal
        ../../gcc/gcc/toplev.c:383
0x10ed7b0 htab_hash_string
        ../../gcc/libiberty/hashtab.c:839
0x6e66df external_ref_hasher::hash(external_ref const*)
        ../../gcc/gcc/dwarf2out.c:7763
0x6e66df hash_table<external_ref_hasher, xcallocator, false>::find_slot(external_ref const*, insert_option)
        ../../gcc/gcc/hash-table.h:652
0x6e66df lookup_external_ref
        ../../gcc/gcc/dwarf2out.c:7791
0x6e6770 optimize_external_refs_1
        ../../gcc/gcc/dwarf2out.c:7829
0x6e6797 optimize_external_refs_1
        ../../gcc/gcc/dwarf2out.c:7833
0x6e6c06 optimize_external_refs
        ../../gcc/gcc/dwarf2out.c:7882
0x6e6e3d output_comp_unit
        ../../gcc/gcc/dwarf2out.c:9122
0x70ac89 dwarf2out_finish
        ../../gcc/gcc/dwarf2out.c:24800
Please submit a full bug report,
with preprocessed source if appropriate.
Please include the complete backtrace with any bug report.
See <http://gcc.gnu.org/bugs.html> for instructions.

From quick look, the problem is that we attempt to generate DIE for a RECORD_TYPE __lambda5 in partition 2.  That RECORD_TYPE has TYPE_CONTEXT the main FUNCTION_DECL, which is apparently emitted in partition 1 and so lookup_decl_die for it returns NULL (there is not even a stub DW_TAG_subprogram emitted in the other partition).
Wonder why do we partition a single CU input at all, and why we decide to put a lambda function into a different partition from the one containing it.

So this should be more reproducible with -flto-partition=max and also with
nested functions?

I wonder if we need to call debug_hooks->function_decl for function
decls in lto_read_decls similar to how we do for type_decls?  At least
with early debug we'd re-create a DIE for it at LTRANS time.

Doesn't help.

Btw, I always wondered why dwarf2out.c doesn't create such stubs itself when it requires a parent (via force_decl_die).

(In reply to Richard Biener from comment #6)
> Btw, I always wondered why dwarf2out.c doesn't create such stubs itself when
> it requires a parent (via force_decl_die).

Most probably because dwarf2out.c has never been written with the current state of LTO in mind, it assumes the FE or the middle-end gives it sane and complete trees of everything needed.
With early debug info, if done right for LTO, supposedly the problems goes away again, so either we add some hack for this and accept there are hundreds of other ways how to get it crashed with -flto -g (perhaps regression on some particular testcase, but generally all gcc versions with -flto -g contained lots of various ICEs), or just add this to other cases which show -flto -g is unsupportable until early debug info is added to GCC 6.

Well, it worked reasonably well (not ICEing) since GCC 4.8 (or 4.9) at least.

*** Bug 65243 has been marked as a duplicate of this bug. ***

Reproduces also with -flto-partition=max and GCC 4.9 succeeds with that.  Suggesting this may be a real regression and not just a latent issue popping up(?)

Honza, can you please have a look?

And can somebody bisect this with -flto-partition=max (and or reduce further)?

Created attachment 35191 [details]
reduced testcase

Further reduced with -flto-partition=max, but still 4.3k.

Created attachment 35193 [details]
reduced testcase

Started with r220011.

(In reply to Markus Trippelsdorf from comment #13)
> Started with r220011.

But that doesn't make too much sense and indeed when I 
use an earlier version of the reduced testcase it still 
crashes with -flto-partition=max on r220010...

It really started with r219076 aka ipa-inline sreal conversion.

Yep, it seems the problem is triggered on and off with random changed in the inliner's decisions...

Lambda functions aren't really nested functions in the tree-nested.c sense, but still are so closely related to the functions they are nested in that IMHO it would be desirable not to split them off into separate partitions, not just temporarily for GCC 5 for debug info reasons.
To find out lambdas, I think walking the list of argument types of a function, for each argument type strip of POINTER_TYPE_P and if it is a RECORD_TYPE after stripping those, check if TYPE_CONTEXT of that is some FUNCTION_DECL and in that case try to put the function into the same partition as that other FUNCTION_DECL.

Gone latent with inliner heuristic changes,  -fno-inline -flto-partition=max is now required to reproduce the ICE (with -O -g, of course).

Which also ICEs with 4.9!

Also crashes with 4.8.

markus@x4 tmp % < tcp_client.ii
template <typename> struct __and_;
template <typename> class function;
template <typename _Functor> class _Base_manager {
protected:
  static _Functor *_M_get_pointer(int) {}
};

template <typename, typename> class A;
template <typename _Res, typename _Functor, typename... _ArgTypes>
class A<_Res(_ArgTypes...), _Functor> : _Base_manager<_Functor> {
public:
  static _Res _M_invoke(const int &) {
    (*_Base_manager<_Functor>::_M_get_pointer(0))();
  }
};

template <typename, typename> using __check_func_return_type = int;
template <typename _Res, typename... _ArgTypes>
class function<_Res(_ArgTypes...)> {
  template <typename> using _Invoke = decltype(0);
  template <typename _Functor>
  using _Callable = __and_<__check_func_return_type<_Invoke<_Functor>, _Res>>;
  template <typename, typename> using _Requires = int;

public:
  template <typename _Functor, typename = _Requires<_Callable<_Functor>, void>>
  function(_Functor);
  using _Invoker_type = _Res (*)(const int &);
  _Invoker_type _M_invoker;
};

template <typename _Res, typename... _ArgTypes>
template <typename _Functor, typename>
function<_Res(_ArgTypes...)>::function(_Functor) {
  _M_invoker = A<_Res(), _Functor>::_M_invoke;
}

class B {
public:
  void configuration();
  void run(int, int, function<void()>);
};
main() {
  B app;
  app.run(0, 0, [&] { app.configuration(); });
}

markus@x4 tmp % /usr/x86_64-pc-linux-gnu/gcc-bin/4.8.4/g++ -std=gnu++1y -flto -g -O2 -r -nostdlib -fno-inline -flto-partition=max tcp_client.ii
lto1: internal compiler error: in build_abbrev_table, at dwarf2out.c:7478

I'm running into this when building firefox beta tree with lto. The release tree is working alright, but on beta it crashes with this bug's stacktrace. Is it only with -g? I'm using -g0 for the most part.

Now that I understand the bug, of course I notice (too late) that my way of setting -g0 wasn't taking effect in the beta tree, the way it was before. So that explains that, but still this crash is happening with just -flto=4 (and default -flto-partition=balanced) for me. That means, at the moment, it's not latent for building firefox.

When doing LTO early-debug work I stumbled over one reason we can crash here
which I fixed with

Index: dwarf2out.c
===================================================================
*** dwarf2out.c (revision 222165)
--- dwarf2out.c (working copy)
*************** resolve_addr (dw_die_ref die)
*** 23950,23957 ****
                && DECL_EXTERNAL (tdecl)
                && DECL_ABSTRACT_ORIGIN (tdecl) == NULL_TREE)
              {
!               force_decl_die (tdecl);
!               tdie = lookup_decl_die (tdecl);
              }
            if (tdie)
              {
--- 23950,23964 ----
                && DECL_EXTERNAL (tdecl)
                && DECL_ABSTRACT_ORIGIN (tdecl) == NULL_TREE)
              {
!               /* Creating a full DIE for tdecl is overly expensive and
!                  at this point (with early debug active) even wrong
!                  as it can end up generating new type DIEs we didn't
!                  output.  */
!               tdie = new_die (DW_TAG_subprogram, comp_unit_die (), NULL_TREE);
!               add_AT_flag (tdie, DW_AT_external, 1);
!               add_AT_flag (tdie, DW_AT_declaration, 1);
!               add_linkage_attr (tdie, tdecl);
!               equate_decl_number_to_die (tdecl, tdie);
              }
            if (tdie)
              {

change the comment to "at this point (with LTO) even wrong" and it still applies.  The point is, if we didn't create a DIE for the external decl
it isn't a good idea to create a full DIE for it for the reference.  Just
create a DIE with enough info for the debugger (I suppose that's the
linkage name?  I really didn't check whether that's ok for the purpose
of DW_TAG_GNU_call_site - Jakub?  (not sure if we have guality checks
that cover this very case)

ISTR that for the case I ran into the call was to some builtin function.

The above patch fixes this bug.

Which means that -gstrict-dwarf should be a workaround for this and the dups.

It's not a latent bug for me:

lto1: internal compiler error: Segmentation fault
0xa2f1af crash_signal
	../../gcc/toplev.c:383
0x1129f20 htab_hash_string
	../../libiberty/hashtab.c:839
0x6d264f external_ref_hasher::hash(external_ref const*)
	../../gcc/dwarf2out.c:7763
0x6d264f hash_table<external_ref_hasher, xcallocator, false>::find_slot(external_ref const*, insert_option)
	../../gcc/hash-table.h:652
0x6d264f lookup_external_ref
	../../gcc/dwarf2out.c:7791
0x6d26ed optimize_external_refs_1
	../../gcc/dwarf2out.c:7829
0x6d2718 optimize_external_refs_1
	../../gcc/dwarf2out.c:7833
0x6d2b56 optimize_external_refs
	../../gcc/dwarf2out.c:7882
0x6d2d8d output_comp_unit
	../../gcc/dwarf2out.c:9122
0x6f8c69 dwarf2out_finish
	../../gcc/dwarf2out.c:24801
Please submit a full bug report,
with preprocessed source if appropriate.
Please include the complete backtrace with any bug report.
See <http://gcc.gnu.org/bugs.html> for instructions.
lto-wrapper: fatal error: /home/avi/gcc5/bin/g++5 returned 1 exit status
compilation terminated.


g++5 (GCC) 6.0.0 20150417 (experimental)

Created attachment 35343 [details]
patch

Patch I am testing.

Patch fixes the problem for me (though the linker still fails)

So the context I put the stub DIE in is technically wrong.  Thus some more analysis results.  The reason why we end up populating the limbo_die_list
from the force_decl_die path is because we refuse to put the context
(the __lambda2 type) into its context (the main function which we forced
a DIE for) because of

      /* A declaration DIE doesn't count; nested types need to go in the
         specification.  */
      if (context_die && is_declaration_die (context_die))
        context_die = NULL;

(In reply to Jakub Jelinek from comment #17)
> Lambda functions aren't really nested functions in the tree-nested.c sense,
> but still are so closely related to the functions they are nested in that
> IMHO it would be desirable not to split them off into separate partitions,
> not just temporarily for GCC 5 for debug info reasons.

Agreed.  decl_function_context of the lambda function should give you the containing function.

On Fri, 17 Apr 2015, jason at gcc dot gnu.org wrote:

> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=65549
> 
> Jason Merrill <jason at gcc dot gnu.org> changed:
> 
>            What    |Removed                     |Added
> ----------------------------------------------------------------------------
>                  CC|                            |jason at gcc dot gnu.org
> 
> --- Comment #29 from Jason Merrill <jason at gcc dot gnu.org> ---
> (In reply to Jakub Jelinek from comment #17)
> > Lambda functions aren't really nested functions in the tree-nested.c sense,
> > but still are so closely related to the functions they are nested in that
> > IMHO it would be desirable not to split them off into separate partitions,
> > not just temporarily for GCC 5 for debug info reasons.
> 
> Agreed.  decl_function_context of the lambda function should give you the
> containing function.

As far as I understood imposing constraints like this to the partitioner
is hard because either you implement the same constraints into the inliner
as well or you eventually blow up partition sizes.

I'm also not 100% sure the issue we hit in dwarf2out is specific
to lambdas or nested functions - it might very well trigger with
regular C++ classes as well, just harder to reproduce.

So at least for branches I think my patch is the way to go - eventually
guarded with in_lto_p (if you really insist on that...)

That's the prominent issue with LTO in Ada because there are a lot of nested functions in Ada.  We use a local patch that fixes up the limbo list again:

@@ -24063,6 +24080,11 @@ dwarf2out_finish (const char *filename)
   }
 #endif
   resolve_addr (comp_unit_die ());
+  /* resolve_addr calls force_decl_die, which can put new DIEs on the limbo
+     list in LTO mode when nested functions are put in a different partition
+     than that of their parent function.  */
+  if (in_lto_p)
+    fixup_limbo_list ();
   move_marked_base_types ();
 
   for (node = deferred_asm_name; node; node = node->next)

Author: rguenth
Date: Tue Jun  2 12:33:02 2015
New Revision: 224029

URL: https://gcc.gnu.org/viewcvs?rev=224029&root=gcc&view=rev
Log:
2015-06-02  Richard Biener  <rguenther@suse.de>

	PR debug/65549
	* dwarf2out.c (lookup_context_die): New function.
	(resolve_addr): Avoid forcing a full DIE for the
	target of a DW_TAG_GNU_call_site during late compilation.
	Instead create a stub DIE without a type if we have a
	context DIE present.

	* g++.dg/lto/pr65549_0.C: New testcase.

Added:
    trunk/gcc/testsuite/g++.dg/lto/pr65549_0.C
Modified:
    trunk/gcc/ChangeLog
    trunk/gcc/dwarf2out.c
    trunk/gcc/testsuite/ChangeLog

Fixed on trunk sofar.

Author: rguenth
Date: Wed Jun  3 07:39:06 2015
New Revision: 224059

URL: https://gcc.gnu.org/viewcvs?rev=224059&root=gcc&view=rev
Log:
2015-06-03  Richard Biener  <rguenther@suse.de>

	Backport from mainline
	2015-06-02  Richard Biener  <rguenther@suse.de>

	PR debug/65549
	* dwarf2out.c (lookup_context_die): New function.
	(resolve_addr): Avoid forcing a full DIE for the
	target of a DW_TAG_GNU_call_site during late compilation.
	Instead create a stub DIE without a type if we have a
	context DIE present.

	* g++.dg/lto/pr65549_0.C: New testcase.

	2015-06-01  Richard Biener  <rguenther@suse.de>

	PR tree-optimization/66280
	* tree-vect-slp.c (vect_detect_hybrid_slp_stmts): Fix pattern
	def-use walking.

	* g++.dg/torture/pr66280.C: New testcase.
	* g++.dg/torture/pr66280-2.C: Likewise.

	2015-05-27  Richard Biener  <rguenther@suse.de>

	PR tree-optimization/66272
	Revert parts of
	2014-08-15  Richard Biener  <rguenther@suse.de>

	PR tree-optimization/62031
	* tree-data-ref.c (dr_analyze_indices): Do not set
	DR_UNCONSTRAINED_BASE.
	(dr_may_alias_p): All indirect accesses have to go the
	formerly DR_UNCONSTRAINED_BASE path.
	* tree-data-ref.h (struct indices): Remove
	unconstrained_base member.
	(DR_UNCONSTRAINED_BASE): Remove.

	* gcc.dg/torture/pr66272.c: New testcase.

	2015-05-21  Richard Biener  <rguenther@suse.de>

	PR c++/66211
	* match.pd: Guard pattern optimzing (int)(float)int
	conversions to apply only on GIMPLE.

	* g++.dg/conversion/pr66211.C: New testcase.
	* gcc.dg/tree-ssa/forwprop-18.c: Adjust.

	2015-05-13  Richard Biener  <rguenther@suse.de>

	PR tree-optimization/66123
	* tree-ssa-dom.c (propagate_rhs_into_lhs): Check if we found
	a taken edge.

	* gcc.dg/torture/pr66123.c: New testcase.

Added:
    branches/gcc-5-branch/gcc/testsuite/g++.dg/conversion/pr66211.C
    branches/gcc-5-branch/gcc/testsuite/g++.dg/lto/pr65549_0.C
    branches/gcc-5-branch/gcc/testsuite/g++.dg/torture/pr66280-2.C
    branches/gcc-5-branch/gcc/testsuite/g++.dg/torture/pr66280.C
    branches/gcc-5-branch/gcc/testsuite/gcc.dg/torture/pr66123.c
    branches/gcc-5-branch/gcc/testsuite/gcc.dg/torture/pr66272.c
Modified:
    branches/gcc-5-branch/gcc/ChangeLog
    branches/gcc-5-branch/gcc/dwarf2out.c
    branches/gcc-5-branch/gcc/match.pd
    branches/gcc-5-branch/gcc/testsuite/ChangeLog
    branches/gcc-5-branch/gcc/testsuite/gcc.dg/tree-ssa/forwprop-18.c
    branches/gcc-5-branch/gcc/tree-data-ref.c
    branches/gcc-5-branch/gcc/tree-data-ref.h
    branches/gcc-5-branch/gcc/tree-ssa-dom.c
    branches/gcc-5-branch/gcc/tree-vect-slp.c

Author: rguenth
Date: Wed Jun  3 11:03:26 2015
New Revision: 224072

URL: https://gcc.gnu.org/viewcvs?rev=224072&root=gcc&view=rev
Log:
2015-06-03  Richard Biener  <rguenther@suse.de>

	Backport from mainline
	2015-05-26  Michael Matz  <matz@suse.de>

	PR middle-end/66251
	* tree-vect-stmts.c (vect_create_vectorized_demotion_stmts): Always set
	STMT_VINFO_VEC_STMT, also with SLP.

	* gcc.dg/vect/pr66251.c: New test.

	2015-05-22  Richard Biener  <rguenther@suse.de>

	PR tree-optimization/66251
	* tree-vect-stmts.c (vectorizable_conversion): Properly
	set STMT_VINFO_VEC_STMT even for the SLP case.

	* gfortran.fortran-torture/compile/pr66251.f90: New testcase.

	2015-05-27  Richard Biener  <rguenther@suse.de>

	PR tree-optimization/66272
	Revert parts of
	2014-08-15  Richard Biener  <rguenther@suse.de>

	PR tree-optimization/62031
	* tree-data-ref.c (dr_analyze_indices): Do not set
	DR_UNCONSTRAINED_BASE.
	(dr_may_alias_p): All indirect accesses have to go the
	formerly DR_UNCONSTRAINED_BASE path.
	* tree-data-ref.h (struct indices): Remove
	unconstrained_base member.
	(DR_UNCONSTRAINED_BASE): Remove.

	* gcc.dg/torture/pr66272.c: New testcase.

	2015-05-13  Richard Biener  <rguenther@suse.de>

	PR tree-optimization/66123
	* tree-ssa-dom.c (propagate_rhs_into_lhs): Check if we found
	a taken edge.

	* gcc.dg/torture/pr66123.c: New testcase.

	2015-06-02  Richard Biener  <rguenther@suse.de>

	PR debug/65549
	* dwarf2out.c (lookup_context_die): New function.
	(resolve_addr): Avoid forcing a full DIE for the
	target of a DW_TAG_GNU_call_site during late compilation.
	Instead create a stub DIE without a type if we have a
	context DIE present.

	* g++.dg/lto/pr65549_0.C: New testcase.

	2015-03-23  Richard Biener  <rguenther@suse.de>

	PR tree-optimization/65518
	* tree-vect-stmts.c (vectorizable_load): Reject single-element
	interleaving cases we generate absymal code for.

	* gcc.dg/vect/pr65518.c: New testcase.

Added:
    branches/gcc-4_9-branch/gcc/testsuite/g++.dg/lto/pr65549_0.C
    branches/gcc-4_9-branch/gcc/testsuite/gcc.dg/torture/pr66123.c
    branches/gcc-4_9-branch/gcc/testsuite/gcc.dg/torture/pr66272.c
    branches/gcc-4_9-branch/gcc/testsuite/gcc.dg/vect/pr65518.c
    branches/gcc-4_9-branch/gcc/testsuite/gcc.dg/vect/pr66251.c
    branches/gcc-4_9-branch/gcc/testsuite/gfortran.fortran-torture/compile/pr66251.f90
Modified:
    branches/gcc-4_9-branch/gcc/ChangeLog
    branches/gcc-4_9-branch/gcc/dwarf2out.c
    branches/gcc-4_9-branch/gcc/testsuite/ChangeLog
    branches/gcc-4_9-branch/gcc/tree-data-ref.c
    branches/gcc-4_9-branch/gcc/tree-data-ref.h
    branches/gcc-4_9-branch/gcc/tree-ssa-dom.c
    branches/gcc-4_9-branch/gcc/tree-vect-stmts.c

Fixed.

Author: rguenth
Date: Thu Jun 11 13:39:56 2015
New Revision: 224375

URL: https://gcc.gnu.org/viewcvs?rev=224375&root=gcc&view=rev
Log:
2015-06-11  Richard Biener  <rguenther@suse.de>

	Backport from mainline, guarded with in_lto_p
	2015-06-02  Richard Biener  <rguenther@suse.de>

	PR debug/65549
	* dwarf2out.c (lookup_context_die): New function.
	(resolve_addr): Avoid forcing a full DIE for the
	target of a DW_TAG_GNU_call_site during late compilation.
	Instead create a stub DIE without a type if we have a
	context DIE present.

	Backport from mainline
	2014-04-04  Jan Hubicka  <hubicka@ucw.cz>

	PR ipa/59626
	* lto-cgraph.c (input_overwrite_node): Check that partitioning
	flags are set only during streaming.
	* ipa.c (process_references, symtab_remove_unreachable_nodes):
	Drop bodies of always inline after early inlining.
	(symtab_remove_unreachable_nodes): Remove always_inline attribute.

	* gcc.dg/lto/pr59626_0.c: New testcase.
	* gcc.dg/lto/pr59626_1.c: New testcase.

Added:
    branches/gcc-4_8-branch/gcc/testsuite/gcc.dg/lto/pr59626_0.c
    branches/gcc-4_8-branch/gcc/testsuite/gcc.dg/lto/pr59626_1.c
Modified:
    branches/gcc-4_8-branch/gcc/ChangeLog
    branches/gcc-4_8-branch/gcc/dwarf2out.c
    branches/gcc-4_8-branch/gcc/ipa.c
    branches/gcc-4_8-branch/gcc/testsuite/ChangeLog