[PATCH] Stack protector i386/x86_64 fixes

[Jakub Jelinek <jakub at redhat dot com>]

Hi!

This patch (on top of
http://gcc.gnu.org/ml/gcc-patches/2005-06/msg01867.html
) fixes a bug in the x86-64 stack_protect_set pattern,
which generates invalid xorl %rax, %rax instruction.
Also, it only uses __stack_chk_fail_local for 32-bit code in i386/*
(x86-64 doesn't need to set up any PIC pointer and using
__stack_chk_fail_local makes programs/libraries tiny bit bigger),
likewise it doesn't use it if visibility is not supported or
for non-pic code.
There is also a tiny optimization, we know that the scratch
register after stack_protect_set will be zero, so it is desirable
to tell the compiler about that, otherwise we ended up e.g. with
xorl %eax, %eax
xorl %eax, %eax
etc. on some testcases.
I have tried to do something similar for stack_protect_test, but
failed.  Telling the compiler it is always 0 is lying, what if the compiler
(unlikely, but who knows) decided to need zero somewhere on the
path to the noreturn function where the register does not
contain zero.  I have tried to write the pattern so that
the compiler could know if the branch does not jump, it is zero,
but it was probably too complicated and did not result in
any code improvements.

	Jakub

Attachment: d-ssp-2
Description: Text document