[Bug target/87928] [7/8/9 Regression] ICE in ix86_compute_frame_layout, at config/i386/i386.c:11161 since r228607

[ubizjak at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87928

Uroš Bizjak <ubizjak at gmail dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |NEW
                 CC|                            |ktietz at gcc dot gnu.org
     Ever confirmed|0                           |1

--- Comment #4 from Uroš Bizjak <ubizjak at gmail dot com> ---
(In reply to Martin Liška from comment #0)
> Following causes ICE:
> 
> $ gcc
> /home/marxin/Programming/gcc/gcc/testsuite/gcc.target/i386/aggregate-ret4.c
> -mabi=ms -O1 -mstackrealign
> during RTL pass: pro_and_epilogue
> /home/marxin/Programming/gcc/gcc/testsuite/gcc.target/i386/aggregate-ret4.c:
> In function ‘bar’:
> /home/marxin/Programming/gcc/gcc/testsuite/gcc.target/i386/aggregate-ret4.c:
> 24:1: internal compiler error: in ix86_compute_frame_layout, at
> config/i386/i386.c:11161
>    24 | }
>       | ^
> 0x73ba80 ix86_compute_frame_layout
> 	/home/marxin/Programming/gcc/gcc/config/i386/i386.c:11161
> 0x111ef6f ix86_finalize_stack_frame_flags
> 	/home/marxin/Programming/gcc/gcc/config/i386/i386.c:12925
> 0x11255e4 ix86_expand_prologue()
> 	/home/marxin/Programming/gcc/gcc/config/i386/i386.c:13035
> 0x145df8a gen_prologue()
> 	/home/marxin/Programming/gcc/gcc/config/i386/i386.md:12984
> 0x1107728 target_gen_prologue
> 	/home/marxin/Programming/gcc/gcc/config/i386/i386.md:18928
> 0xab61c1 make_prologue_seq
> 	/home/marxin/Programming/gcc/gcc/function.c:5713
> 0xab6367 thread_prologue_and_epilogue_insns()
> 	/home/marxin/Programming/gcc/gcc/function.c:5830
> 0xab6a96 rest_of_handle_thread_prologue_and_epilogue
> 	/home/marxin/Programming/gcc/gcc/function.c:6321
> 0xab6a96 execute
> 	/home/marxin/Programming/gcc/gcc/function.c:6363

This happens in i386.c:

--cut here--
  /* 64-bit MS ABI seem to require stack alignment to be always 16,
     except for function prologues, leaf functions and when the defult
     incoming stack boundary is overriden at command line or via
     force_align_arg_pointer attribute.  */
  if ((TARGET_64BIT_MS_ABI && crtl->preferred_stack_boundary < 128)
      && (!crtl->is_leaf || cfun->calls_alloca != 0
          || ix86_current_function_calls_tls_descriptor
          || ix86_incoming_stack_boundary < 128))
    {
      crtl->preferred_stack_boundary = 128;
      crtl->stack_alignment_needed = 128;
    }

  stack_alignment_needed = crtl->stack_alignment_needed / BITS_PER_UNIT;
  preferred_alignment = crtl->preferred_stack_boundary / BITS_PER_UNIT;

  gcc_assert (!size || stack_alignment_needed);
  gcc_assert (preferred_alignment >= STACK_BOUNDARY / BITS_PER_UNIT);
  gcc_assert (preferred_alignment <= stack_alignment_needed);
--cut here--

Where TARGET_64BIT_MS_ABI gets defined as:

#define TARGET_64BIT_MS_ABI (TARGET_64BIT && ix86_cfun_abi () == MS_ABI)

and STACK_BOUNDARY:

#define STACK_BOUNDARY \
  (TARGET_64BIT && ix86_abi == MS_ABI ? 128 : BITS_PER_WORD)

Please note that the former definition depends on ix86_cfun_abi, which depends
on __attribute__(([sysv_abi|ms_abi])), while the later depends on ix86_abi
global variable. In the testcase, ix86_cfun_abi () returns SYSV_ABI due to
function __attribute__ override, while ix86_abi returns MS_ABI.

The following patch redefines STACK_BOUNDARY to:

 #define STACK_BOUNDARY \
- (TARGET_64BIT && ix86_abi == MS_ABI ? 128 : BITS_PER_WORD)
+  (TARGET_64BIT && ix86_cfun_abi () == MS_ABI ? 128 : BITS_PER_WORD)

so, STACK_BOUNDARY now also depends on function __attribute__.

The patch also removes unneeded redefinition of STACK_BOUNDARY from cygming.h.

--cut here--
diff --git a/gcc/config/i386/cygming.h b/gcc/config/i386/cygming.h
index d7c7dd7057bf..0a3c9a0dd401 100644
--- a/gcc/config/i386/cygming.h
+++ b/gcc/config/i386/cygming.h
@@ -268,9 +268,6 @@ do {                                                \
    bytes in one go.  */
 #define CHECK_STACK_LIMIT 4000

-#undef STACK_BOUNDARY
-#define STACK_BOUNDARY (TARGET_64BIT && ix86_abi == MS_ABI ? 128 :
BITS_PER_WORD)
-
 /* By default, target has a 80387, uses IEEE compatible arithmetic,
    returns float values in the 387 and needs stack probes.
    We also align doubles to 64-bits for MSVC default compatibility.  */
diff --git a/gcc/config/i386/i386.h b/gcc/config/i386/i386.h
index 58caab2bb55b..a85b43134092 100644
--- a/gcc/config/i386/i386.h
+++ b/gcc/config/i386/i386.h
@@ -808,7 +808,7 @@ extern const char *host_detect_local_cpu (int argc, const
char **argv);

 /* Boundary (in *bits*) on which stack pointer should be aligned.  */
 #define STACK_BOUNDARY \
- (TARGET_64BIT && ix86_abi == MS_ABI ? 128 : BITS_PER_WORD)
+  (TARGET_64BIT && ix86_cfun_abi () == MS_ABI ? 128 : BITS_PER_WORD)

 /* Stack boundary of the main function guaranteed by OS.  */
 #define MAIN_STACK_BOUNDARY (TARGET_64BIT ? 128 : 32)
--cut here--

Adding Kai to CC. The patch also needs to be tested on Windows targets.