Bug 41858

Summary:	-flto -O2 crashes when many source files.
Product:	gcc	Reporter:	Basile Starynkevitch <bstarynk>
Component:	lto	Assignee:	Richard Biener <rguenth>
Status:	RESOLVED FIXED
Severity:	major	CC:	bstarynk, gcc-bugs
Priority:	P3	Keywords:	lto
Version:	4.5.0
Target Milestone:	---
Host:	x86_64-unknown-linux-gnu	Target:	x86_64-unknown-linux-gnu
Build:	x86_64-unknown-linux-gnu	Known to work:
Known to fail:		Last reconfirmed:	2009-10-28 23:12:19
Attachments:	a small program which generates many files to exhibit the bug

Description Basile Starynkevitch 2009-10-28 21:37:09 UTC

When compiling a lot (e.g. 4000) source files with
   gcc-trunk -flto -O2 [A-Z]*.c -o big
the lto component crashes with sigsegv.

Version:
gcc version 4.5.0 20091028 (experimental) (GCC) 

that it the trunk rev 153680

Comment 1 Basile Starynkevitch 2009-10-28 21:40:35 UTC

Created attachment 18927 [details]
a small program which generates many files to exhibit the bug

The attached main.c file generates many files.

Put it in a new directory (empty).

Compile it with 
   gcc -O main.c -o main

Run it to generate many files, for example
  ./main 4000

it should display messages ending with

generated function #3904 in E_150
generated function #3968 in Q_152
generated 4000 files last is W_153.c
try to compile all 4000 files with:
	 gcc-trunk -flto -O2 -DGENERATEDALL main.c [A-Z]*.c -o big


Now, run the suggest command (replacing gcc-trunk with what is appropriate for you).

Regards.

Comment 2 Basile Starynkevitch 2009-10-28 21:47:49 UTC

The attachment main.c is a source file generator and test case.

Put it in an empty directory.

First, compile it just with
  gcc main.c -o main
[you could add other flags to gcc, you can use any flavor of gcc]

Then run it once to generate many files.
  ./main 4000

Then it shows a lot of messages, ending with
 generated function #3968 in Q_152
generated 4000 files last is W_153.c
try to compile all 4000 files with:
	 gcc-trunk -flto -O2 -DGENERATEDALL main.c [A-Z]*.c -o big

Do what it tells, so run the gcc trunk as said.

Comment 3 Richard Biener 2009-10-28 22:10:22 UTC

Well, it looks like you generated a pretty degenerate callgraph.  Trying
to reproduce.

Comment 4 Basile Starynkevitch 2009-10-28 22:22:27 UTC

The linker is gold on Debian/Sid/Amd64 
% /usr/bin/ld -v
GNU gold (GNU Binutils for Debian 2.20) 1.9

It could be that it is that gold which crashes. Apparently, it open a lot of files, and when getting the MFILE errno, something has a sigegv. Perhaps that something is lto1.

Comment 5 Richard Biener 2009-10-28 23:10:07 UTC

Confirmed.  There's a file-descriptor leak (well, we kepp to many of them open).

Comment 6 Richard Biener 2009-10-28 23:12:19 UTC

I have a patch.

Comment 7 Basile Starynkevitch 2009-10-29 16:06:26 UTC

(In reply to comment #6)
> I have a patch.
> 

Richard's patch on http://gcc.gnu.org/ml/gcc/2009-10/msg00602.html
seems to solve the original problem I've got 
http://gcc.gnu.org/ml/gcc-patches/2009-10/msg01741.html

I hope that Richard's patch will be accepted in 4.5.
I am marking this PR as a resolved bug (or should I wait for the patch to be accepted?)

Comment 8 Richard Biener 2009-10-30 15:59:25 UTC

Subject: Bug 41858

Author: rguenth
Date: Fri Oct 30 15:58:57 2009
New Revision: 153758

URL: http://gcc.gnu.org/viewcvs?root=gcc&view=rev&rev=153758
Log:
2009-10-30  Richard Guenther  <rguenther@suse.de>

	PR lto/41858
	* lto-streamer.h (struct lto_file_decl_data): Remove fd member.

	lto/
	* lto.c (lto_file_read): Do not set file_data->fd.
	(lto_read_section_data): Use a single-entry file-descriptor cache.
	Do not check the result of xmalloc.
	(free_section_data): Do not use file_data->fd.
	(lto_read_all_file_options): Likewise.

Modified:
    trunk/gcc/ChangeLog
    trunk/gcc/lto-streamer.h
    trunk/gcc/lto/ChangeLog
    trunk/gcc/lto/lto.c