When I did fuzz-testing I had several crashes caused by infinite recursion. I did not find an easy and good solution to fix them. Perhaps, introducing some additional logic which would specifically target invalid inputs could help (like maintaining a hashtable/list/whatever of visited nodes which cannot be visited again during normal result output; substitutions would require some additional handling).
Nevertheless, I think that the testcases are still worth being recorded. Here are some, I believe, distinct examples:
_Z1KIStcvT_E - i.e. something like:
Normally, when demangling templated conversion operator we would print out the template parameter, but in this case this leads to infinite recursion, because we think that the operator itself is the parameter.
More problems with conversion operator:
_ZcvT_IIS0_EE - in substitution
_ZcvT_IZcvT_E1fE - in local name
_Z1gINcvT_EE - in nested name (probably same as std::)
_ZcvT_ILZcvDTT_EEE - template parameter in decltype
Infinite recursion when collapsing ref-qualifiers:
Memory hog with pointers-to-member and arrays:
(output size doubles with each M-o pair)
"pointer-to-member" and "array" are not necessarily consecutive: _ZdvMMMMMMMMMMMMMrrrrA_DTdvfp_fp_Eededilfdfdfdfd
(I first posted this as a comment for an existing bug, which was actually caused by a different issue, sorry for double-posting)
Created attachment 36208 [details]
Created attachment 36211 [details]
Script for testcase reduction
The proposed patch for Bug 70909 - Libiberty Demangler segfaults (4) stops these kind of infinite recursion.
Date: Wed Mar 8 14:28:38 2017
New Revision: 245978
Fix PR demangler/70909 and 67264 (endless demangler recursion)
* include/demangle.h: Add d_printing to struct demangle_component
and pass struct demangle_component as non const.
* cp-demangle.c: Fix endless recursion. Pass
struct demangle_component as non const.
(d_make_empty): Initialize variable.
(d_print_comp_inner): Limit recursion.
(d_print_comp): Decrement variable.
* cp-demint.c (cplus_demangle_fill_component): Initialize
* testsuite/demangle-expected: Add tests.