Bug 79399 - GCC fails to compile big source at -O0
Summary: GCC fails to compile big source at -O0
Alias: None
Product: gcc
Classification: Unclassified
Component: middle-end (show other bugs)
Version: unknown
: P3 normal
Target Milestone: ---
Assignee: Not yet assigned to anyone
Depends on:
Blocks: yarpgen
  Show dependency treegraph
Reported: 2017-02-06 23:27 UTC by Dmitry Babokin
Modified: 2021-11-01 23:07 UTC (History)
2 users (show)

See Also:
Known to work:
Known to fail:
Last reconfirmed: 2017-02-07 00:00:00

gcc7-pr79399.patch (537 bytes, patch)
2017-02-07 10:46 UTC, Jakub Jelinek
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Dmitry Babokin 2017-02-06 23:27:58 UTC
The bug report is not very typical, but it may help fixing the real issue, which is hard to find and debug in other corricumstances. I don't mind if it's closed as "will not be fixed", but still hope it could be useful.

The test case is huge auto generated C++ file - 260Mb of source code (1.7Mb when compressed). The code is scalar. So I would expect that GCC compiles it without issues at least at -O0. But it fails trying to allocate 18446744071569032860 bytes, which is... 1.8*10^19. Which is too much in any case. This is unreasonable value. But if converted to hex format it's 0xffffffff806a469c. Probably someone tries to allocate more than 2^32 bytes in 64bit compiler and has an issue with passing 64 bit size (size_t actually) and uses 32 bit value instead?

> /usr/bin/time --format="Max %M kb\nreal %E\nuser %U\nsys %S" g++ -std=c++11 -w -O0 -march=nehalem -o gcc_no_opt_func.o -c func.cpp

cc1plus: out of memory allocating 18446744071569032860 bytes after a total of 9836802048 bytes
Max 24979592 kb
real 22:27.38
user 1321.54
sys 24.45

The test case is, again, 1.7Mb when compressed, so I can't attach it to the bug report. But it's available here:


Compiler that I'm using is couple days old trunk:
> gcc -v
Using built-in specs.
Target: x86_64-pc-linux-gnu
Configured with: ../gcc_svn/configure --with-arch=corei7 --with-cpu=corei7 --enable-clocale=gnu --with-system-zlib --enable-shared --with-demangler-in-ld --enable-cloog-backend=isl --with-fpmath=sse --prefix=/gcc/bin_svn/ --enable-languages=c,c++,lto : (reconfigured) ../gcc_svn_intel/configure --with-arch=corei7 --with-cpu=corei7 --enable-clocale=gnu --with-system-zlib --enable-shared --with-demangler-in-ld --enable-cloog-backend=isl --with-fpmath=sse --prefix=/gcc/bin_svn/ --enable-languages=c,c++,lto --no-create --no-recursion : (reconfigured) ../gcc_svn/configure --with-arch=corei7 --with-cpu=corei7 --enable-clocale=gnu --with-system-zlib --enable-shared --with-demangler-in-ld --enable-cloog-backend=isl --with-fpmath=sse --prefix=/gcc/bin_svn/ --enable-languages=c,c++,lto --no-create --no-recursion : (reconfigured) ../gcc_svn/configure --with-arch=corei7 --with-cpu=corei7 --enable-clocale=gnu --with-system-zlib --enable-shared --with-demangler-in-ld --enable-cloog-backend=isl --with-fpmath=sse --prefix=/gcc/bin_svn/ --enable-languages=c,c++,lto --no-create --no-recursion
Thread model: posix
gcc version 7.0.1 20170203 (experimental) (GCC)
Comment 1 Jakub Jelinek 2017-02-07 08:16:29 UTC
You would save a lot of pain and even could compile it at -O2 if the generator generated many, but much smaller, functions, for such sizes it certainly pays off.
What I see is after allocating ~ 24GB of memory ira-costs.c (init_costs) tries to allocate 124 * 17380077 bytes, but both values are ints, so the multiplication result is int too and is negative, while it should have allocated just slightly over 2GB.  So the following patch should cure this, though not sure if that is the only problem (and will see if I manage to compile, have only 32GB in my box).

--- gcc/ira-costs.c.jj	2017-01-16 12:28:35.000000000 +0100
+++ gcc/ira-costs.c	2017-02-07 09:07:00.744696695 +0100
@@ -1704,7 +1704,7 @@ find_costs_and_classes (FILE *dump_file)
 				  process_bb_node_for_costs, NULL);
 	  memcpy (total_allocno_costs, costs,
-		  max_struct_costs_size * ira_allocnos_num);
+		  (size_t) max_struct_costs_size * ira_allocnos_num);
@@ -2205,7 +2205,7 @@ static void
 init_costs (void)
   init_subregs_of_mode ();
-  costs = (struct costs *) ira_allocate (max_struct_costs_size
+  costs = (struct costs *) ira_allocate ((size_t) max_struct_costs_size
 					 * cost_elements_num);
   pref_buffer = (enum reg_class *) ira_allocate (sizeof (enum reg_class)
 						 * cost_elements_num);
@@ -2235,8 +2235,9 @@ ira_costs (void)
   allocno_p = true;
   cost_elements_num = ira_allocnos_num;
   init_costs ();
-  total_allocno_costs = (struct costs *) ira_allocate (max_struct_costs_size
-						       * ira_allocnos_num);
+  total_allocno_costs
+    = (struct costs *) ira_allocate ((size_t) max_struct_costs_size
+				     * ira_allocnos_num);
   initiate_regno_cost_classes ();
   calculate_elim_costs_all_insns ();
   find_costs_and_classes (ira_dump_file);
Comment 2 Markus Trippelsdorf 2017-02-07 08:22:25 UTC
32GB is not enough. gcc-6 is using ~42GB on gcc112 and is still running for over an hour...
And it is probably a good idea to use a "--enable-checking=release" compiler for this testcase.
Comment 3 Jakub Jelinek 2017-02-07 08:32:21 UTC
Ok, then I guess I'll just bootstrap/regtest my patch, it is a way forward without testing it further.

2017-02-07  Jakub Jelinek  <jakub@redhat.com>

	PR middle-end/79399
	* ira-int.h (struct target_ira_int): Change x_max_struct_costs_size
	type from int to size_t.

--- gcc/ira-int.h.jj	2017-01-01 12:45:39.000000000 +0100
+++ gcc/ira-int.h	2017-02-07 09:29:32.694102809 +0100
@@ -782,7 +782,7 @@ struct target_ira_int {
   /* Initialized once.  It is a maximal possible size of the allocated
      struct costs.  */
-  int x_max_struct_costs_size;
+  size_t x_max_struct_costs_size;
   /* Allocated and initialized once, and used to initialize cost values
      for each insn.  */

is better, this field is followed by a pointer, so it just uses a padding on 64-bit hosts for something useful and on 32-bit hosts won't make any changes.
Comment 4 Dmitry Babokin 2017-02-07 08:39:13 UTC
The purpose of the generator is to break a compiler, that's why it's a single function in this case, but not many. Though with smaller functions we break compilers too.

This is the generator: https://github.com/01org/yarpgen
Comment 5 Jakub Jelinek 2017-02-07 08:47:58 UTC
Ah, ok then.  Note that e.g. vectors use unsigned int for length, so as long as you need more than 4G elements in any vector, the game is over, but probably testcase of this size doesn't reach that even close.  Anyway, thanks for the report.  Can you try yourself with the #c3 patch?
Comment 6 Dmitry Babokin 2017-02-07 09:10:21 UTC
I've started the compilation, it should take more than an hour to finish. Will report back when it's done.
Comment 7 Dmitry Babokin 2017-02-07 10:23:24 UTC
It crashed.
> /usr/bin/time --format="Max %M kb\nreal %E\nuser %U\nsys%S" g++ -std=c++11 -w -O0 -march=nehalem -o gcc_no_opt_func.o -c func.cpp
func.cpp: In function ‘void foo()’:
func.cpp:26656:1: internal compiler error: Segmentation fault
0xd62f9f crash_signal
0xbb902e find_costs_and_classes
0xbba119 ira_costs()
0xbb3569 ira_build()
0xba9d42 ira
0xba9d42 execute
Please submit a full bug report,
with preprocessed source if appropriate.
Please include the complete backtrace with any bug report.
See <http://gcc.gnu.org/bugs.html> for instructions.
Max 30306372 kb
real 40:09.53
user 2349.01
Comment 8 Richard Biener 2017-02-07 10:42:26 UTC
/* It is the current size of struct costs.  */
static int struct_costs_size;

so similar issue.
Comment 9 Jakub Jelinek 2017-02-07 10:46:56 UTC
Created attachment 40684 [details]

Comment 10 Jakub Jelinek 2017-02-07 21:51:53 UTC
Author: jakub
Date: Tue Feb  7 21:51:21 2017
New Revision: 245256

URL: https://gcc.gnu.org/viewcvs?rev=245256&root=gcc&view=rev
	PR middle-end/79399
	* ira-int.h (struct target_ira_int): Change x_max_struct_costs_size
	type from int to size_t.
	* ira-costs.c (struct_costs_size): Change type from int to size_t.

Comment 11 Dmitry Babokin 2017-02-08 00:18:21 UTC
With new patch it compiled successfully. It took 41G of memory and 5:25 hours to complete.
Comment 12 Jakub Jelinek 2017-02-08 07:43:51 UTC
Thanks for testing that.
Comment 13 Jakub Jelinek 2017-02-15 07:04:37 UTC
Author: jakub
Date: Wed Feb 15 07:04:05 2017
New Revision: 245470

URL: https://gcc.gnu.org/viewcvs?rev=245470&root=gcc&view=rev
	Backported from mainline
	2017-02-07  Jakub Jelinek  <jakub@redhat.com>
		    Richard Biener  <rguenther@suse.de>

	PR middle-end/79399
	* ira-int.h (struct target_ira_int): Change x_max_struct_costs_size
	type from int to size_t.
	* ira-costs.c (struct_costs_size): Change type from int to size_t.

Comment 14 Jakub Jelinek 2017-05-30 07:57:42 UTC
Author: jakub
Date: Tue May 30 07:57:10 2017
New Revision: 248641

URL: https://gcc.gnu.org/viewcvs?rev=248641&root=gcc&view=rev
	Backported from mainline
	2017-02-07  Jakub Jelinek  <jakub@redhat.com>
		    Richard Biener  <rguenther@suse.de>

	PR middle-end/79399
	* ira-int.h (struct target_ira_int): Change x_max_struct_costs_size
	type from int to size_t.
	* ira-costs.c (struct_costs_size): Change type from int to size_t.