Bug 41858 - -flto -O2 crashes when many source files.
Summary: -flto -O2 crashes when many source files.
Alias: None
Product: gcc
Classification: Unclassified
Component: lto (show other bugs)
Version: 4.5.0
: P3 major
Target Milestone: ---
Assignee: Richard Biener
Keywords: lto
Depends on:
Reported: 2009-10-28 21:37 UTC by Basile Starynkevitch
Modified: 2009-10-29 16:06 UTC (History)
2 users (show)

See Also:
Host: x86_64-unknown-linux-gnu
Target: x86_64-unknown-linux-gnu
Build: x86_64-unknown-linux-gnu
Known to work:
Known to fail:
Last reconfirmed: 2009-10-28 23:12:19

a small program which generates many files to exhibit the bug (1.02 KB, text/x-csrc)
2009-10-28 21:40 UTC, Basile Starynkevitch

Note You need to log in before you can comment on or make changes to this bug.
Description Basile Starynkevitch 2009-10-28 21:37:09 UTC
When compiling a lot (e.g. 4000) source files with
   gcc-trunk -flto -O2 [A-Z]*.c -o big
the lto component crashes with sigsegv.

gcc version 4.5.0 20091028 (experimental) (GCC) 

that it the trunk rev 153680
Comment 1 Basile Starynkevitch 2009-10-28 21:40:35 UTC
Created attachment 18927 [details]
a small program which generates many files to exhibit the bug

The attached main.c file generates many files.

Put it in a new directory (empty).

Compile it with 
   gcc -O main.c -o main

Run it to generate many files, for example
  ./main 4000

it should display messages ending with

generated function #3904 in E_150
generated function #3968 in Q_152
generated 4000 files last is W_153.c
try to compile all 4000 files with:
	 gcc-trunk -flto -O2 -DGENERATEDALL main.c [A-Z]*.c -o big

Now, run the suggest command (replacing gcc-trunk with what is appropriate for you).

Comment 2 Basile Starynkevitch 2009-10-28 21:47:49 UTC
The attachment main.c is a source file generator and test case.

Put it in an empty directory.

First, compile it just with
  gcc main.c -o main
[you could add other flags to gcc, you can use any flavor of gcc]

Then run it once to generate many files.
  ./main 4000

Then it shows a lot of messages, ending with
 generated function #3968 in Q_152
generated 4000 files last is W_153.c
try to compile all 4000 files with:
	 gcc-trunk -flto -O2 -DGENERATEDALL main.c [A-Z]*.c -o big

Do what it tells, so run the gcc trunk as said.
Comment 3 Richard Biener 2009-10-28 22:10:22 UTC
Well, it looks like you generated a pretty degenerate callgraph.  Trying
to reproduce.
Comment 4 Basile Starynkevitch 2009-10-28 22:22:27 UTC
The linker is gold on Debian/Sid/Amd64 
% /usr/bin/ld -v
GNU gold (GNU Binutils for Debian 2.20) 1.9

It could be that it is that gold which crashes. Apparently, it open a lot of files, and when getting the MFILE errno, something has a sigegv. Perhaps that something is lto1.
Comment 5 Richard Biener 2009-10-28 23:10:07 UTC
Confirmed.  There's a file-descriptor leak (well, we kepp to many of them open).
Comment 6 Richard Biener 2009-10-28 23:12:19 UTC
I have a patch.
Comment 7 Basile Starynkevitch 2009-10-29 16:06:26 UTC
(In reply to comment #6)
> I have a patch.

Richard's patch on http://gcc.gnu.org/ml/gcc/2009-10/msg00602.html
seems to solve the original problem I've got 

I hope that Richard's patch will be accepted in 4.5.
I am marking this PR as a resolved bug (or should I wait for the patch to be accepted?)

Comment 8 Richard Biener 2009-10-30 15:59:25 UTC
Subject: Bug 41858

Author: rguenth
Date: Fri Oct 30 15:58:57 2009
New Revision: 153758

URL: http://gcc.gnu.org/viewcvs?root=gcc&view=rev&rev=153758
2009-10-30  Richard Guenther  <rguenther@suse.de>

	PR lto/41858
	* lto-streamer.h (struct lto_file_decl_data): Remove fd member.

	* lto.c (lto_file_read): Do not set file_data->fd.
	(lto_read_section_data): Use a single-entry file-descriptor cache.
	Do not check the result of xmalloc.
	(free_section_data): Do not use file_data->fd.
	(lto_read_all_file_options): Likewise.