Fix fir PR71696 in Libiberty Demangler (6)

[Marcel Böhme]

Hi,

This patch is still pending a full review.

Best regards,
- Marcel

> On 4 Jul 2016, at 8:54 PM, Bernd Schmidt <bschmidt@redhat.com> wrote:
> 
> On 06/30/2016 08:46 AM, Marcel Böhme wrote:
>> The attached patch fixes the stack overflow in the demangler due to
>> cycles in the references of “remembered” mangled types
>> (https://gcc.gnu.org/bugzilla/show_bug.cgi?id=71696).
>> 
>> The methods demangle_signature and do_arg in cplus-dem.c allow to
>> “remember” mangled type names that can later be referenced and will
>> also be demangled. The method demangle_args demangles those types
>> following any references. So, if there is a cycle in the referencing
>> (or in the simplest case a self-reference), the method enters
>> infinite recursion.
>> 
>> The patch tracks the mangled types that are currently being demangled
>> in a new variable called work->proctypevec. If a referenced type is
>> currently being demangled, the demangling is marked as not
>> successful.
> 
> I'll defer reviewing these to someone who understands demangling better. You might want to Cc Jason.
> 
> 
> Bernd
> 
>