[Bug middle-end/80930] New: REE pass causes high memory usage via df_mir_alloc() with ASAN+UBSAN turned on

["sirl at gcc dot gnu.org" <gcc-bugzilla at gcc dot gnu dot org>]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=80930

            Bug ID: 80930
           Summary: REE pass causes high memory usage via df_mir_alloc()
                    with ASAN+UBSAN turned on
           Product: gcc
           Version: 7.1.1
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: middle-end
          Assignee: unassigned at gcc dot gnu.org
          Reporter: sirl at gcc dot gnu.org
  Target Milestone: ---

We have an inhouse C source where the memory usage is excessive (> 88GB, then
OOM killed) with GCC-7/x86_64 (7.1.1@r248575). With GCC-6 (6.3.1@r248575) the
memory usage is a bit better, roughly 58GB.
When stopped in gdb shortly before OOM, the backtrace looks like this

(gdb) bt
#0  bitmap_elt_insert_after (indx=1076, elt=<optimized out>,
head=0x7fff3aa62bd0) at ../../gcc/bitmap.c:409
#1  bitmap_set_range(bitmap_head*, unsigned int, unsigned int) () at
../../gcc/bitmap.c:1233
#2  0x0000000000c9827b in df_mir_alloc(bitmap_head*) () at
../../gcc/df-problems.c:1914
#3  0x0000000000c8d454 in df_analyze_problem (n_blocks=299166,
postorder=0x17168b2e0, blocks_to_consider=<optimized out>, dflow=0x16b830fb0)
at ../../gcc/df-core.c:1165
#4  df_analyze_1 () at ../../gcc/df-core.c:1226
#5  df_analyze() () at ../../gcc/df-core.c:1294
#6  0x000000000109ab81 in find_and_remove_re () at ../../gcc/ree.c:1216
#7  rest_of_handle_ree () at ../../gcc/ree.c:1310
#8  (anonymous namespace)::pass_ree::execute(function*) () at
../../gcc/ree.c:1338
#9  0x0000000000deace1 in execute_one_pass(opt_pass*) () at
../../gcc/passes.c:2465
#10 0x000000000062c20c in execute_pass_list_1 (pass=0x1c94570) at
../../gcc/passes.c:2554
#11 0x000000000062c1dc in execute_pass_list_1 (pass=0x1c943f0) at
../../gcc/passes.c:2555
#12 execute_pass_list_1 (pass=0x1c931d0) at ../../gcc/passes.c:2555
#13 execute_pass_list (fn=<optimized out>, pass=<optimized out>) at
../../gcc/passes.c:2565
#14 0x00000000010fcb0d in cgraph_node::expand (this=0x7fffe8ccbcf0) at
../../gcc/cgraphunit.c:2042
#15 expand_all_functions () at ../../gcc/cgraphunit.c:2178
#16 symbol_table::compile() () at ../../gcc/cgraphunit.c:2535
#17 0x0000000000c7973f in symbol_table::finalize_compilation_unit() () at
../../gcc/cgraphunit.c:2625
#18 0x000000000113b6b0 in compile_file() () at ../../gcc/toplev.c:492
#19 0x0000000000bd2afb in do_compile () at ../../gcc/toplev.c:2000
#20 toplev::main(int, char**) () at ../../gcc/toplev.c:2134
#21 0x0000000000bd439b in main (argc=177, argv=0x7fffffffcbd8) at
../../gcc/main.c:39

Using -fno-ree brings down memory usage to 7GB (GCC-7) and 5GB (GCC-6).
The relevant options for the compile are:
 -g -O2 -version -fsanitize=address -fstack-protector-strong
-fsanitize=undefined -fsanitize=bounds-strict -fno-omit-frame-pointer
-fno-common -ffunction-sections -fdata-sections -faggressive-loop-optimizations

I can provide more information and do other testing, but a public testcase will
likely be impossible.