[Bug target/79807] [5/6/7 Regression] ICE in extract_insn, at recog.c:2311 (error: unrecognizable insn)

[jakub at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=79807

Jakub Jelinek <jakub at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |NEW
   Last reconfirmed|                            |2017-03-02
                 CC|jakub at redhat dot com            |jakub at gcc dot gnu.org,
                   |                            |uros at gcc dot gnu.org
   Target Milestone|---                         |5.5
            Summary|ICE in extract_insn, at     |[5/6/7 Regression] ICE in
                   |recog.c:2311 (error:        |extract_insn, at
                   |unrecognizable insn)        |recog.c:2311 (error:
                   |                            |unrecognizable insn)
     Ever confirmed|0                           |1

--- Comment #1 from Jakub Jelinek <jakub at gcc dot gnu.org> ---
This is not related to PR79568 in any way.
I think it started with r178132.
Short testcase -O0 -mavx -ffloat-store:
typedef double __m128d __attribute__ ((__vector_size__ (16), __may_alias__));
typedef double __m256d __attribute__ ((__vector_size__ (32), __may_alias__));
typedef double __v4df __attribute__ ((__vector_size__ (32)));

__m128d
foo (__m256d __A)
{
  return (__m128d) __builtin_ia32_pd_pd256 ((__v4df)__A);
}

I think the problem is that tons of sse.md instructions accept memory in both
output operand predicate and some input operand predicate, but then have
conditions like:
&& !(MEM_P (operands[0]) && MEM_P (operands[1]))
or
&& (register_operand (operands[0], <MODE>mode)
       || register_operand (operands[1], <MODE>mode))
But, when this is expanded through ix86_expand_*_builtin,
(ix86_expand_args_builtin in particular in this case), we only care about the
predicates, not the condition.
And we have:
  if (rmode == VOIDmode || rmode == tmode)
    {
      if (optimize
          || target == 0
          || GET_MODE (target) != tmode
          || !insn_p->operand[0].predicate (target, tmode))
        target = gen_reg_rtx (tmode);
      real_target = target;
    }
  else
    {
      real_target = gen_reg_rtx (tmode);
      target = lowpart_subreg (rmode, real_target, tmode);
    }
and then:
          /* If we aren't optimizing, only allow one memory operand to
             be generated.  */
          if (memory_operand (op, mode))
            num_memory++;
...
              if (optimize || !match || num_memory > 1)
                op = copy_to_mode_reg (mode, op);

So, for -O0, we can end up with both the target/real_target being MEM as well
as one of the operands.

The big question is if there are any insns we care for -O0 to have MEMs in both
output and input operands.  If not, I have a patch.