[PATCH] Fix buffer overflow in simplify_immed_subreg

[Jakub Jelinek]

Hi!

The following testcase causes
*** stack smashing detected ***: /usr/libexec/gcc/i386-redhat-linux/4.0.1/cc1 terminated
when gcc is built with -fstack-protector.
The problematic simplify_immed_subreg call is:
simplify_immed_subreg (outermode=V16QImode, op, innermode=V2DImode, byte=0)
where op is
(const_vector:V2DI [
        (const_double 168430090 [0xa0a0a0a] 168430090 [0xa0a0a0a] 0 [0x0] 0
[0x0] 0 [0x0] 0 [0x0])
        (const_double 168430090 [0xa0a0a0a] 168430090 [0xa0a0a0a] 0 [0x0] 0
[0x0] 0 [0x0] 0 [0x0])
    ])
CONST_VECTOR has 2 elements, both VOIDmode CONST_DOUBLEs.
value is a 64 byte long array (max_bitsize / value_bit).
When handling the first CONST_DOUBLE, vp will be set to &value[0] before getting
into the
case CONST_DOUBLE:
so clearing bits up to max_bitsize causes no overflow.
But for the second CONST_DOUBLE, vp will be &value[8] (the first CONST_DOUBLE
covered 64 bits), and so we clear 8 bytes after the value array, which can be stack
canary, return address or whatever else.
When handling CONST_INTs or MODE_FLOAT CONST_DOUBLEs, simplify_immed_subreg just
clears (or sign extends) bits up to elem_bitsize, which makes sense:
          /* CONST_INTs are always logically sign-extended.  */
          for (; i < elem_bitsize; i += value_bit)
            *vp++ = INTVAL (el) < 0 ? -1 : 0;
resp.
              /* It shouldn't matter what's done here, so fill it with
                 zero.  */
              for (; i < elem_bitsize; i += value_bit)
                *vp++ = 0;
Only VOIDmode CONST_DOUBLEs do the surprising:
              /* It shouldn't matter what's done here, so fill it with
                 zero.  */
              for (; i < max_bitsize; i += value_bit)
                *vp++ = 0;
which really can't work.  Really sounds like a typo to me.
The bug has been introduced in 3.4.
Ok to commit to 3.4/4.0/HEAD?

2005-08-20  Jakub Jelinek  <jakub@redhat.com>

	* simplify-rtx.c (simplify_immed_subreg) <case CONST_DOUBLE>: Only clear
	up to elem_bitsize bits, not max_bitsize.

	* gcc.target/i386/sse-4.c: New test.

--- gcc/simplify-rtx.c.jj	2005-08-20 04:24:41.000000000 -0400
+++ gcc/simplify-rtx.c	2005-08-20 16:02:21.613690586 -0400
@@ -3605,7 +3605,7 @@ simplify_immed_subreg (enum machine_mode
 		}
 	      /* It shouldn't matter what's done here, so fill it with
 		 zero.  */
-	      for (; i < max_bitsize; i += value_bit)
+	      for (; i < elem_bitsize; i += value_bit)
 		*vp++ = 0;
 	    }
 	  else
--- gcc/testsuite/gcc.target/i386/sse-4.c.jj	2005-08-20 16:04:34.754403712 -0400
+++ gcc/testsuite/gcc.target/i386/sse-4.c	2005-08-20 16:06:58.996092357 -0400
@@ -0,0 +1,10 @@
+/* This testcase caused a buffer overflow in simplify_immed_subreg.  */
+/* { dg-do compile } */
+/* { dg-options "-O2 -msse2" } */
+
+#include <emmintrin.h>
+
+__m128i foo (__m128i x)
+{
+  return _mm_min_epu8 (x, _mm_set1_epi8 (10));
+}

	Jakub