[Bug middle-end/78786] New: GCC hangs/out of memory calling sprintf with large precision

["msebor at gcc dot gnu.org" <gcc-bugzilla at gcc dot gnu dot org>]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=78786

            Bug ID: 78786
           Summary: GCC hangs/out of memory calling sprintf with large
                    precision
           Product: gcc
           Version: 7.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: middle-end
          Assignee: unassigned at gcc dot gnu.org
          Reporter: msebor at gcc dot gnu.org
  Target Milestone: ---

As mentioned in the review of a patch for bug 78696
(https://gcc.gnu.org/ml/gcc-patches/2016-12/msg01098.html), due to what looks
like an MPFR bug/inefficiency, GCC hangs on the following input with the "%.*f"
format string.  The mpfr_snprintf function apparently tries to allocate on the
order of INT_MAX bytes of memory or, if it manages to allocate it, attempts to
fill it.  Imposing a limit on the amount of virtual memory available to the
process turns the problem into an ICE.

$ (set -x; cat b.c && ulimit -S -v 1000000 && for f in "%i %f" "%.*f"; do
/build/gcc-svn/gcc/xgcc -B /build/gcc-svn/gcc -DFMT="\"$f\"" -S -Wall -Wextra
b.c; done)
+ cat b.c
char d[1];

void f (double x)
{
  __builtin_sprintf (d + 1, FMT, __INT_MAX__, x);
}
+ ulimit -S -v 1000000
+ for f in '"%i %f"' '"%.*f"'
+ /build/gcc-svn/gcc/xgcc -B /build/gcc-svn/gcc '-DFMT="%i %f"' -S -Wall
-Wextra b.c
b.c: In function ‘f’:
<command-line>:0:5: warning: ‘%i’ directive writing 10 bytes into a region of
size 0 [-Wformat-length=]
b.c:5:29: note: in expansion of macro ‘FMT’
   __builtin_sprintf (d + 1, FMT, __INT_MAX__, x);
                             ^~~
b.c:5:3: note: format output between 20 and 329 bytes into a destination of
size 0
   __builtin_sprintf (d + 1, FMT, __INT_MAX__, x);
   ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ for f in '"%i %f"' '"%.*f"'
+ /build/gcc-svn/gcc/xgcc -B /build/gcc-svn/gcc '-DFMT="%.*f"' -S -Wall -Wextra
b.c
GNU MP: Cannot allocate memory (size=2147483957)
b.c: In function ‘f’:
b.c:3:6: internal compiler error: Aborted
 void f (double x)
      ^
0xedf8f4 crash_signal
        /src/gcc/svn/gcc/toplev.c:333
0x1c606b1 __gmp_default_allocate
        /src/gcc/svn/gmp/memory.c:58
0x1bcf384 mpfr_get_str
        /src/gcc/svn/mpfr/src/get_str.c:2325
0x1bd7fd1 regular_fg
        /src/gcc/svn/mpfr/src/vasprintf.c:1374
0x1bd8aed partition_number
        /src/gcc/svn/mpfr/src/vasprintf.c:1598
0x1bd8f01 sprnt_fp
        /src/gcc/svn/mpfr/src/vasprintf.c:1708
0x1bda091 __gmpfr_vasprintf
        /src/gcc/svn/mpfr/src/vasprintf.c:2031
0x1bd5c0c mpfr_snprintf
        /src/gcc/svn/mpfr/src/printf.c:169
0x18c2019 format_floating_max
        /src/gcc/svn/gcc/gimple-ssa-sprintf.c:1251
0x18c25a1 format_floating
        /src/gcc/svn/gcc/gimple-ssa-sprintf.c:1361
0x18c2ee7 format_floating
        /src/gcc/svn/gcc/gimple-ssa-sprintf.c:1534
0x18c36e0 format_directive
        /src/gcc/svn/gcc/gimple-ssa-sprintf.c:1820
0x18c4e15 compute_format_length
        /src/gcc/svn/gcc/gimple-ssa-sprintf.c:2578
0x18c5c60 handle_gimple_call
        /src/gcc/svn/gcc/gimple-ssa-sprintf.c:2938
0x18c5d31 execute
        /src/gcc/svn/gcc/gimple-ssa-sprintf.c:2966
Please submit a full bug report,
with preprocessed source if appropriate.
Please include the complete backtrace with any bug report.
See <http://gcc.gnu.org/bugs.html> for instructions.