[Bug rtl-optimization/57422] [4.9 Regression] ICE: SIGSEGV in dominated_by_p with custom flags

[abel at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=57422

Andrey Belevantsev <abel at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |ASSIGNED
           Assignee|unassigned at gcc dot gnu.org      |abel at gcc dot gnu.org

--- Comment #3 from Andrey Belevantsev <abel at gcc dot gnu.org> ---
Created attachment 31465
  --> http://gcc.gnu.org/bugzilla/attachment.cgi?id=31465&action=edit
proposed patch

This is an issue with somewhat surprising reason.  The problem in itself is
that when we're scheduling the fence insn (actually next insn), it gets
register renamed and so its previous form stored as the fence insn no longer
corresponds to the new form.  We do not expect it so there is no code to update
the fence insn in this case.

Starting to add this code that easily fixed the issue, I stopped and wondered
why this situation arises at all -- the fence insn should always be able to be
scheduled as is.  In this case, first, the target availability bit correctly
set to true on this insn is reset because this insn form was already scheduled
on this fence and the bit might be incorrect (see
vec_target_unavailable_vinsns).  Thus we resort to the full recomputation of
the possible registers for the insn, which suddenly do not include its target
register (ax).

Now this happens because ax is incorrectly marked as unavailable due to target
reasons.  And this in turn happens because of the typo in the patch of rev.
172231:

-#if !HARD_FRAME_POINTER_IS_FRAME_POINTER
-      for (i = hard_regno_nregs[HARD_FRAME_POINTER_REGNUM][Pmode]; i--;)
-       SET_HARD_REG_BIT (reg_rename_p->unavailable_hard_regs,
-                          HARD_FRAME_POINTER_REGNUM + i);
-#endif
+      if (!HARD_FRAME_POINTER_IS_FRAME_POINTER)
+        add_to_hard_reg_set (&reg_rename_p->unavailable_hard_regs,
+                            Pmode, HARD_FRAME_POINTER_IS_FRAME_POINTER);

Instead of HARD_FRAME_POINTER_REGNUM, the parameter passed to
add_to_hard_reg_set is HARD_FRAME_POINTER_IS_FRAME_POINTER, which is zero and
equals to ax number :-)  Thus we always mark ax as unavailable in this piece of
code, which was noticed just because for this particular insn this leads to its
renaming and for this insn it should never happen.

The obvious patch restoring HARD_FRAME_POINTER_REGNUM is attached and fixes the
testcase (for the reported revision; the trunk no longer fails).  I will also
add an assert to check that the fence insn never gets renamed.