A stackoverflow in the libiberty demangler causes its host application to crash on a tainted branch instruction. The problem is caused by a self-reference in a mangled type string that is "remembered" for later reference. This leads to an infinite recursion during the demangling.
How to reproduce:
$ valgrind cxxfilt __33%00000000000000000000__S8_00T0000T0
==30184== Memcheck, a memory error detector
==30184== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==30184== Using Valgrind-3.10.1 and LibVEX; rerun with -h for copyright info
==30184== Command: ../../binutils_git5/libiberty/testsuite/test-demangle
==30184== Stack overflow in thread 1: can't grow stack to 0xffe801ff8
$ valgrind cxxfilt __10%0__S4_0T0T0
Analysis: The demangler "remembers" mangled types that can later be referenced in the mangled string. In this case the complete string is remembered in work->typevec before it is demangled in cplus-dem.c:4534. Now, a part of the mangled string ("T0000") references the remembered type stored at work->typevec which is supposed to be demangled as well (cplus-dem.c:3632).
The tool was found with a more efficient version of AFL, called AFL-Fast.
I am preparing a patch.
Date: Thu Aug 4 16:53:18 2016
New Revision: 239143
2016-08-04 Marcel Böhme <email@example.com>
* cplus-dem.c: Prevent infinite recursion when there is a cycle
in the referencing of remembered mangled types.
(work_stuff): New stack to keep track of the remembered mangled
types that are currently being processed.
(push_processed_type): New method to push currently processed
remembered type onto the stack.
(pop_processed_type): New method to pop currently processed
remembered type from the stack.
(work_stuff_copy_to_from): Copy values of new variables.
(delete_non_B_K_work_stuff): Free stack memory.
(demangle_args): Push/Pop currently processed remembered type.
(do_type): Do not demangle a cyclic reference and push/pop
referenced remembered type.
Should be fixed on trunk now.