[Bug other/59195] C++ demangler handles conversion operator incorrectly

[trippels at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=59195

--- Comment #6 from Markus Trippelsdorf <trippels at gcc dot gnu.org> ---
Author: trippels
Date: Fri Nov 27 14:48:21 2015
New Revision: 231020

URL: https://gcc.gnu.org/viewcvs?rev=231020&root=gcc&view=rev
Log:
PR other/61321 - demangler crash on casts in template parameters

The fix for bug 59195:

 [C++ demangler handles conversion operator incorrectly]
 https://gcc.gnu.org/bugzilla/show_bug.cgi?id=59195

unfortunately makes the demangler crash due to infinite recursion, in
case of casts in template parameters.

For example, with:

 template<int> struct A {};
 template <typename Y> void function_temp(A<sizeof ((Y)(999))>) {}
 template void function_temp<int>(A<sizeof (int)>);

The 'function_temp<int>' instantiation above mangles to:

  _Z13function_tempIiEv1AIXszcvT_Li999EEE

The demangler parses this as:

typed name
  template
    name 'function_temp'
    template argument list
      builtin type int
  function type
    builtin type void
    argument list
      template                          (*)
        name 'A'
        template argument list
          unary operator
            operator sizeof
            unary operator
              cast
                template parameter 0    (**)
              literal
                builtin type int
                name '999'

And after the fix for 59195, due to:

 static void
 d_print_cast (struct d_print_info *dpi, int options,
               const struct demangle_component *dc)
 {
 ...
   /* For a cast operator, we need the template parameters from
      the enclosing template in scope for processing the type.  */
   if (dpi->current_template != NULL)
     {
       dpt.next = dpi->templates;
       dpi->templates = &dpt;
       dpt.template_decl = dpi->current_template;
     }

when printing the template argument list of A (what should be "<sizeof
(int)>"), the template parameter 0 (that is, "T_", the '**' above) now
refers to the first parameter of the the template argument list of the
'A' template (the '*' above), exactly what we were already trying to
print.  This leads to infinite recursion, and stack exaustion.  The
template parameter 0 should actually refer to the first parameter of
the 'function_temp' template.

Where it reads "for the cast operator" in the comment in d_print_cast
(above), it's really talking about a conversion operator, like:

  struct A { template <typename U> explicit operator U(); };

We don't want to inject the template parameters from the enclosing
template in scope when processing a cast _expression_, only when
handling a conversion operator.

The problem is that DEMANGLE_COMPONENT_CAST is currently ambiguous,
and means _both_ 'conversion operator' and 'cast expression'.

Fix this by adding a new DEMANGLE_COMPONENT_CONVERSION component type,
which does what DEMANGLE_COMPONENT_CAST does today, and making
DEMANGLE_COMPONENT_CAST just simply print its component subtree.

I think we could instead reuse DEMANGLE_COMPONENT_CAST and in
d_print_comp_inner still do:

 @@ -5001,9 +5013,9 @@ d_print_comp_inner (struct d_print_info *dpi, int
options,
        d_print_comp (dpi, options, dc->u.s_extended_operator.name);
        return;

     case DEMANGLE_COMPONENT_CAST:
       d_append_string (dpi, "operator ");
 -     d_print_cast (dpi, options, dc);
 +     d_print_conversion (dpi, options, dc);
       return;

leaving the unary cast case below calling d_print_cast, but seems to
me that spliting the component types makes it easier to reason about
the code.

g++'s testsuite actually generates three symbols that crash the
demangler in the same way.  I've added those as tests in the demangler
testsuite as well.

And then this fixes PR other/61233 too, which happens to be a
demangler crash originally reported to GDB, at:
https://sourceware.org/bugzilla/show_bug.cgi?id=16957

Bootstrapped and regtested on x86_64 Fedora 20.

Also ran this through GDB's testsuite.  GDB will require a small
update to use DEMANGLE_COMPONENT_CONVERSION in one place it's using
DEMANGLE_COMPONENT_CAST in its sources.

libiberty/
2015-11-27  Pedro Alves  <palves@redhat.com>

        PR other/61321
        PR other/61233
        * demangle.h (enum demangle_component_type)
        <DEMANGLE_COMPONENT_CONVERSION>: New value.
        * cp-demangle.c (d_demangle_callback, d_make_comp): Handle
        DEMANGLE_COMPONENT_CONVERSION.
        (is_ctor_dtor_or_conversion): Handle DEMANGLE_COMPONENT_CONVERSION
        instead of DEMANGLE_COMPONENT_CAST.
        (d_operator_name): Return a DEMANGLE_COMPONENT_CONVERSION
        component if handling a conversion.
        (d_count_templates_scopes, d_print_comp_inner): Handle
        DEMANGLE_COMPONENT_CONVERSION.
        (d_print_comp_inner): Handle DEMANGLE_COMPONENT_CONVERSION instead
        of DEMANGLE_COMPONENT_CAST.
        (d_print_cast): Rename as ...
        (d_print_conversion): ... this.  Adjust comments.
        (d_print_cast): Rewrite - simply print the left subcomponent.
        * cp-demint.c (cplus_demangle_fill_component): Handle
        DEMANGLE_COMPONENT_CONVERSION.

        * testsuite/demangle-expected: Add tests.

Modified:
    trunk/include/demangle.h
    trunk/libiberty/ChangeLog
    trunk/libiberty/cp-demangle.c
    trunk/libiberty/cp-demint.c
    trunk/libiberty/testsuite/demangle-expected