PR 55359: ICE on invalid subreg in simplify_subreg

[Richard Sandiford <rdsandiford at googlemail dot com>]

This patch fixes PR 55359, which is an ICE caused by my removal
of a validate_subreg call in:

    http://gcc.gnu.org/ml/gcc-patches/2012-10/msg01353.html

The question then is whether the caller of simplify_(gen_)subreg is
responsible for checking whether a subreg is valid, or whether the
simplification routines should return null for cases they can't handle.
simplify_gen_subreg already contains a validate_subreg call, so the latter
seemed better.

At first I wanted to make simplify_subreg return null if
!validate_subreg, but that hit a case in the x86 backend where we tried
to create an SFmode constant by taking the lowpart of a DImode CONST_INT.
This patch instead takes the more conservative approach of returning
null for offsets that can't be handled.

Tested on x86_64-linux-gnu.  OK to install?

Richard


gcc/
	PR middle-end/55359
	* simplify-rtx.c (simplify_subreg): Return null for invalid offsets.

gcc/testsuite/
	* gcc.target/i386/pr55359.c: New test.

Index: gcc/simplify-rtx.c
===================================================================
--- gcc/simplify-rtx.c	2012-11-19 19:58:34.334226296 +0000
+++ gcc/simplify-rtx.c	2012-11-19 20:34:58.170680961 +0000
@@ -5685,8 +5685,11 @@ simplify_subreg (enum machine_mode outer
   gcc_assert (GET_MODE (op) == innermode
 	      || GET_MODE (op) == VOIDmode);
 
-  gcc_assert ((byte % GET_MODE_SIZE (outermode)) == 0);
-  gcc_assert (byte < GET_MODE_SIZE (innermode));
+  if ((byte % GET_MODE_SIZE (outermode)) != 0)
+    return NULL_RTX;
+
+  if (byte >= GET_MODE_SIZE (innermode))
+    return NULL_RTX;
 
   if (outermode == innermode && !byte)
     return op;
Index: gcc/testsuite/gcc.target/i386/pr55359.c
===================================================================
--- /dev/null	2012-11-17 12:28:45.191709874 +0000
+++ gcc/testsuite/gcc.target/i386/pr55359.c	2012-11-19 20:01:37.744975549 +0000
@@ -0,0 +1,10 @@
+/* { dg-do compile } */
+/* { dg-options "-mavx2 -O2" } */
+
+#include <x86intrin.h>
+
+__m128d
+f (__m256d x)
+{
+  return *((__m128d*) ((double *) &x + 1));
+}