[Bug testsuite/66154] ReserveShadowMemoryRange test suite errors when memory overcommitting is turned off on Linux

["f dot heckenbach at fh-soft dot de" <gcc-bugzilla at gcc dot gnu dot org>]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=66154

--- Comment #1 from Frank Heckenbach <f.heckenbach@fh-soft.de> ---
Somehow this got sent before the text was inserted, so here goes:

Also reported on Debian, but I was asked to report it here:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=785249

When I tried to build gcc on Debian jessie from sources
(debuild -us -uc), I got a number of errors like this:

FAIL: g++.dg/asan/interception-malloc-test-1.C  -O1  output pattern test, is
==25380==ERROR: AddressSanitizer failed to allocate 0xdfff0001000
(15392894357504) bytes at address 0x02008fff7000 (12)
==25380==ReserveShadowMemoryRange failed while trying to map 0xdfff0001000
bytes. Perhaps you're using ulimit -v
, should match malloc call.*(
|
)[^
]*heap-use-after-free

In fact I wasn't using ulimit -v, but I had turned off memory
overcommitting (sysctl: vm.overcommit_memory=2). Obviously I have
less than 14 TB of RAM in my system. ;) After I turned it on, these
errors disappeared.

So I suggest:

- If possible, amend the test to use a more reasonable allocation.
  Not sure what it's actually testing, but I wonder what I can do
  with 14 TB (which is way short of the 64 bit address range) that
  it couldn't do with, say 6 GB (larger than 32 bit range, in case
  that matters). Otherwise:

- In the wording of the message, add "or have turned off memory
  overcommitting" as a possible reason, at least for Linux systems.

- Verify [ `sysctl -n vm.overcommit_memory` -eq 0 ] early in the
  build process, so you don't get to see such errors after a long
  time building.

  Same for a ulimit check if there isn't one already.