type_in_anonymous_namespace WRT compund types, take 2

[Jan Hubicka <hubicka at ucw dot cz>]

Jason,
this is my second attempt to get type_in_anonymous_namespace_p return true on
compound types built from types in anonymous namespace.  I need this for LTO
TBAA where I do need the anonymous types to be closed upwards.

The original thread is here https://gcc.gnu.org/ml/gcc-patches/2014-07/msg00382.html

I added the cases you mentioned and did testing with my patch on
libreoffice/firefox/gcc.  I learnt two extra things

  1) We need to look at main variants only.  For example if structure
     has pointer to itself, it may happen that some of its variants
     won't get TYPE_STUB_DECL set.
     Example is i.e. http://doxygen.db48x.net/mozilla/html/structMarItem__.html
  2) I also want to check functions arguments/return value.  Perhaps this
     is needed only in cases method is turned to function by tree-sra, but
     I am not quite sure about it.

The patch LTO bootstraps now, but while building firefox I get ICE on the following:

 <record_type 0x7ffff686d930 DI
    size <integer_cst 0x7ffff6ad7b58 type <integer_type 0x7ffff6adb150 bitsizetype> constant 64>
    unit size <integer_cst 0x7ffff6ad7b70 type <integer_type 0x7ffff6adb0a8 sizetype> constant 8>
    align 64 symtab 0 alias set 0 structural equality
    fields <field_decl 0x7ffff686e098 mPtr
        type <pointer_type 0x7ffff6846000 type <record_type 0x7ffff683bd20 OpenSignedAppFileTask>
            unsigned DI size <integer_cst 0x7ffff6ad7b58 64> unit size <integer_cst 0x7ffff6ad7b70 8>
            align 64 symtab 0 alias set 0 canonical type 0x7ffff6846000>
        DI file ../../dist/include/mozilla/RefPtr.h line 225 col 7 size <integer_cst 0x7ffff6ad7b58 64>
        align 1 offset_align 1
        offset <integer_cst 0x7ffff6ad7b88 constant 0>
        bit offset <integer_cst 0x7ffff6ad7bd0 constant 0> context <record_type 0x7ffff686d930>> context <record_type 0x7ffff6848690 RefPtr>>


This is a record that is no class type (i.e. have to TYPE_BINFO etc.) but it has mPtr field that points
to anonymous type structure.  Since the ICE happens at link time, I have bit hard time identifying
where it is built.  I plan to write a checker for it tomorrow.

Is there any chance to convince C++ FE to put those into anonymous namespaces, too?

	* tree.c (type_in_anonymous_namespace_p): Look into main variant;
	handle ARRAY_TYPE, POINTER_TYPE, OFFSET_TYPE and FUNCTION_TYPE
	without own TYPE_STUB_DECL.
Index: tree.c
===================================================================
--- tree.c	(revision 215614)
+++ tree.c	(working copy)
@@ -11821,6 +11821,32 @@ obj_type_ref_class (tree ref)
 bool
 type_in_anonymous_namespace_p (const_tree t)
 {
+  t = TYPE_MAIN_VARIANT (t);
+  /* Coumpound types do not need to have stub decls. Look for the base
+     type.  */
+  while (!TYPE_STUB_DECL (t)
+	 && TREE_TYPE (t)
+	 && (TREE_CODE (t) == ARRAY_TYPE
+	     || POINTER_TYPE_P (t)
+	     || TREE_CODE (t) == OFFSET_TYPE))
+    t = TYPE_MAIN_VARIANT (TREE_TYPE (t));
+
+  /* Methods do not have their STUB_DECLs, but they are anonymous
+     if type they belong to is.  */
+  if (TREE_CODE (t) == METHOD_TYPE)
+    t = TYPE_MAIN_VARIANT (TYPE_METHOD_BASETYPE (t));
+
+  /* We consider function anonymous if any of the types are.  */
+  if (TREE_CODE (t) == FUNCTION_TYPE
+      && !TYPE_STUB_DECL (t))
+    {
+      if (type_in_anonymous_namespace_p (TREE_TYPE (t)))
+	return true;
+      for (tree p = TYPE_ARG_TYPES (t); p; p = TREE_CHAIN (p))
+        if (type_in_anonymous_namespace_p (TREE_VALUE (p)))
+	  return true;
+      return false;
+    }
   /* TREE_PUBLIC of TYPE_STUB_DECL may not be properly set for
      bulitin types; those have CONTEXT NULL.  */
   if (!TYPE_CONTEXT (t))