You would save a lot of pain and even could compile it at -O2 if the generator generated many, but much smaller, functions, for such sizes it certainly pays off.
What I see is after allocating ~ 24GB of memory ira-costs.c (init_costs) tries to allocate 124 * 17380077 bytes, but both values are ints, so the multiplication result is int too and is negative, while it should have allocated just slightly over 2GB.  So the following patch should cure this, though not sure if that is the only problem (and will see if I manage to compile, have only 32GB in my box).

--- gcc/ira-costs.c.jj	2017-01-16 12:28:35.000000000 +0100
+++ gcc/ira-costs.c	2017-02-07 09:07:00.744696695 +0100
@@ -1704,7 +1704,7 @@ find_costs_and_classes (FILE *dump_file)
 				  process_bb_node_for_costs, NULL);
 
 	  memcpy (total_allocno_costs, costs,
-		  max_struct_costs_size * ira_allocnos_num);
+		  (size_t) max_struct_costs_size * ira_allocnos_num);
 	}
       else
 	{
@@ -2205,7 +2205,7 @@ static void
 init_costs (void)
 {
   init_subregs_of_mode ();
-  costs = (struct costs *) ira_allocate (max_struct_costs_size
+  costs = (struct costs *) ira_allocate ((size_t) max_struct_costs_size
 					 * cost_elements_num);
   pref_buffer = (enum reg_class *) ira_allocate (sizeof (enum reg_class)
 						 * cost_elements_num);
@@ -2235,8 +2235,9 @@ ira_costs (void)
   allocno_p = true;
   cost_elements_num = ira_allocnos_num;
   init_costs ();
-  total_allocno_costs = (struct costs *) ira_allocate (max_struct_costs_size
-						       * ira_allocnos_num);
+  total_allocno_costs
+    = (struct costs *) ira_allocate ((size_t) max_struct_costs_size
+				     * ira_allocnos_num);
   initiate_regno_cost_classes ();
   calculate_elim_costs_all_insns ();
   find_costs_and_classes (ira_dump_file);

32GB is not enough. gcc-6 is using ~42GB on gcc112 and is still running for over an hour...
And it is probably a good idea to use a "--enable-checking=release" compiler for this testcase.

Ok, then I guess I'll just bootstrap/regtest my patch, it is a way forward without testing it further.

Actually:
2017-02-07  Jakub Jelinek  <jakub@redhat.com>

	PR middle-end/79399
	* ira-int.h (struct target_ira_int): Change x_max_struct_costs_size
	type from int to size_t.

--- gcc/ira-int.h.jj	2017-01-01 12:45:39.000000000 +0100
+++ gcc/ira-int.h	2017-02-07 09:29:32.694102809 +0100
@@ -782,7 +782,7 @@ struct target_ira_int {
 
   /* Initialized once.  It is a maximal possible size of the allocated
      struct costs.  */
-  int x_max_struct_costs_size;
+  size_t x_max_struct_costs_size;
 
   /* Allocated and initialized once, and used to initialize cost values
      for each insn.  */

is better, this field is followed by a pointer, so it just uses a padding on 64-bit hosts for something useful and on 32-bit hosts won't make any changes.

The purpose of the generator is to break a compiler, that's why it's a single function in this case, but not many. Though with smaller functions we break compilers too.

This is the generator: https://github.com/01org/yarpgen

Ah, ok then.  Note that e.g. vectors use unsigned int for length, so as long as you need more than 4G elements in any vector, the game is over, but probably testcase of this size doesn't reach that even close.  Anyway, thanks for the report.  Can you try yourself with the #c3 patch?

I've started the compilation, it should take more than an hour to finish. Will report back when it's done.

It crashed.
> /usr/bin/time --format="Max %M kb\nreal %E\nuser %U\nsys%S" g++ -std=c++11 -w -O0 -march=nehalem -o gcc_no_opt_func.o -c func.cpp
func.cpp: In function ‘void foo()’:
func.cpp:26656:1: internal compiler error: Segmentation fault
 }
 ^
0xd62f9f crash_signal
        ../../gcc_svn/gcc/toplev.c:333
0xbb902e find_costs_and_classes
        ../../gcc_svn/gcc/ira-costs.c:1697
0xbba119 ira_costs()
        ../../gcc_svn/gcc/ira-costs.c:2242
0xbb3569 ira_build()
        ../../gcc_svn/gcc/ira-build.c:3420
0xba9d42 ira
        ../../gcc_svn/gcc/ira.c:5236
0xba9d42 execute
        ../../gcc_svn/gcc/ira.c:5541
Please submit a full bug report,
with preprocessed source if appropriate.
Please include the complete backtrace with any bug report.
See <http://gcc.gnu.org/bugs.html> for instructions.
Max 30306372 kb
real 40:09.53
user 2349.01
sys53.94

/* It is the current size of struct costs.  */
static int struct_costs_size;


so similar issue.

Created attachment 40684 [details]
gcc7-pr79399.patch

Indeed.

Author: jakub
Date: Tue Feb  7 21:51:21 2017
New Revision: 245256

URL: https://gcc.gnu.org/viewcvs?rev=245256&root=gcc&view=rev
Log:
	PR middle-end/79399
	* ira-int.h (struct target_ira_int): Change x_max_struct_costs_size
	type from int to size_t.
	* ira-costs.c (struct_costs_size): Change type from int to size_t.

Modified:
    trunk/gcc/ChangeLog
    trunk/gcc/ira-costs.c
    trunk/gcc/ira-int.h

With new patch it compiled successfully. It took 41G of memory and 5:25 hours to complete.

Thanks for testing that.

Author: jakub
Date: Wed Feb 15 07:04:05 2017
New Revision: 245470

URL: https://gcc.gnu.org/viewcvs?rev=245470&root=gcc&view=rev
Log:
	Backported from mainline
	2017-02-07  Jakub Jelinek  <jakub@redhat.com>
		    Richard Biener  <rguenther@suse.de>

	PR middle-end/79399
	* ira-int.h (struct target_ira_int): Change x_max_struct_costs_size
	type from int to size_t.
	* ira-costs.c (struct_costs_size): Change type from int to size_t.

Modified:
    branches/gcc-6-branch/gcc/ChangeLog
    branches/gcc-6-branch/gcc/ira-costs.c
    branches/gcc-6-branch/gcc/ira-int.h

Author: jakub
Date: Tue May 30 07:57:10 2017
New Revision: 248641

URL: https://gcc.gnu.org/viewcvs?rev=248641&root=gcc&view=rev
Log:
	Backported from mainline
	2017-02-07  Jakub Jelinek  <jakub@redhat.com>
		    Richard Biener  <rguenther@suse.de>

	PR middle-end/79399
	* ira-int.h (struct target_ira_int): Change x_max_struct_costs_size
	type from int to size_t.
	* ira-costs.c (struct_costs_size): Change type from int to size_t.

Modified:
    branches/gcc-5-branch/gcc/ChangeLog
    branches/gcc-5-branch/gcc/ira-costs.c
    branches/gcc-5-branch/gcc/ira-int.h