[Bug demangler/79111] New: demangle_template tries to allocate 18446744070799748648 bytes

[ppluzhnikov at google dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=79111

            Bug ID: 79111
           Summary: demangle_template tries to allocate
                    18446744070799748648 bytes
           Product: gcc
           Version: unknown
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: demangler
          Assignee: unassigned at gcc dot gnu.org
          Reporter: ppluzhnikov at google dot com
  Target Milestone: ---

Test case from LLVM libFuzzer.

Using current trunk binutils (libiberty identical to current trunk GCC
r244514):

cxxfilt __H21111111109__nuDD2
cxxfilt: out of memory allocating 18446744070799748648 bytes after a total of
135168 bytes

(gdb) b __libc_malloc if bytes > 10000
Breakpoint 2 at 0x7ffff7893660: file malloc.c, line 2876.
(gdb) c
Continuing.

Breakpoint 2, __GI___libc_malloc (bytes=140737488345896) at malloc.c:2876
2876    malloc.c: No such file or directory.
(gdb) up 2
#2  0x00000000007bd246 in demangle_template (work=work@entry=0x7fffffffdba0,
mangled=mangled@entry=0x7fffffffdb28, tname=tname@entry=0x7fffffffdb40,
trawname=trawname@entry=0x0, is_type=is_type@entry=0,
remember=remember@entry=0)
    at ../../libiberty/cplus-dem.c:2232
2232          work->tmpl_argvec = XNEWVEC (char *, r);
(gdb) p r
$1 = -363725371

(gdb) bt
#0  __GI___libc_malloc (bytes=140737488345896) at malloc.c:2876
#1  0x00000000007d1158 in xmalloc (size=18446744070799748648) at
../../libiberty/xmalloc.c:147
#2  0x00000000007bd246 in demangle_template (work=work@entry=0x7fffffffdba0,
mangled=mangled@entry=0x7fffffffdb28, tname=tname@entry=0x7fffffffdb40,
trawname=trawname@entry=0x0, is_type=is_type@entry=0,
remember=remember@entry=0)
    at ../../libiberty/cplus-dem.c:2232
#3  0x00000000007c05e6 in demangle_signature (work=work@entry=0x7fffffffdba0,
mangled=mangled@entry=0x7fffffffdb28, declp=declp@entry=0x7fffffffdb40) at
../../libiberty/cplus-dem.c:1695
#4  0x00000000007c1435 in internal_cplus_demangle
(work=work@entry=0x7fffffffdba0, mangled=0x7fffffffe0af "_nuDD2",
mangled@entry=0x7fffffffe0a0 "__H21111111109__nuDD2") at
../../libiberty/cplus-dem.c:1261
#5  0x00000000007bc492 in cplus_demangle (mangled=0x7fffffffe0a0
"__H21111111109__nuDD2", options=11) at ../../libiberty/cplus-dem.c:922
#6  0x0000000000405644 in demangle_it (mangled_name=0x7fffffffe0a0
"__H21111111109__nuDD2") at ../../binutils/cxxfilt.c:62
#7  0x000000000040525c in main (argc=2, argv=0x7fffffffdd88) at
../../binutils/cxxfilt.c:227


Should get_count() check for int overflow?