[PATCH] Fix mips-tfile causing gdb to crash on Tru64

[Roger Sayle <roger at eyesopen dot com>]

I decided to investigate why the binaries produced by recent versions
of GCC cause gdb to segfault on alphaev67-dec-osf5.1.  I was expecting
it to be a GDB issue (reproducible with both gdb 6.4 and gdb 6.5) but
digging deeper the cause was actually recent change to GCC's
mips-tfile.c.

An very innocent looking change by Kelly Cook in revision 99203,
http://gcc.gnu.org/viewcvs?view=rev&revision=99203 changed the
following lines:

-#ifdef HOST_WORDS_BIG_ENDIAN
+#ifdef WORDS_BIG_ENDIAN
   init_file.fdr.fBigendian = 1;
 #endif

The unanticipated consequence of this change is that unlike
HOST_WORDS_BIG_ENDIAN that was either defined or not, its replacement
WORDS_BIG_ENDIAN is always defined, but takes a value of either one
or zero.  The result is that mips-tfile now always generates its
ecoff information claiming to be big-endian, though on alpha it's
actually always little-endian.  This confuses gdb hence the seg-fault.

The underlying problem addressed by the above change is that
mips-tfile doesn't attempt to perform byte-swapping but instead
writes bytes in its native order.  Hence, depending upon the way
mips-tfile is potentially used this needs to be HOST_WORDS_BIG_ENDIAN
or BUILD_WORDS_BIG_ENDIAN either of which may be unrelated to
TARGET_WORDS_BIG_ENDIAN.

I opted for the trivial naive solution below, which is to perform a
trivial run-time endianness test.  However, if someone could tell me
what the correct test should be, I'm happy to change the code below.
I've a suspicion that it should be something like:

#ifdef WORDS_BIG_ENDIAN
	if (WORDS_BIG_ENDIAN)
	  init_file.fdr.fBigendian = 1;
#endif


The following patch has been tested on alphaev67-dec-osf5.1 with a
full bootstrap, all default languages except Ada, and regression
tested with a top-level "make -k check" with no new failures.

Ok for mainline?  [I'm not sure who's responsible for mips-tfile]


Hopefully a working gdb will help track down the java segfault
building libjava on this platform.


2006-09-12  Roger Sayle  <roger@eyesopen.com>

	* mips-tfile.c (initialize_init_file): Correct endianness test.


Index: mips-tfile.c
===================================================================
*** mips-tfile.c	(revision 116809)
--- mips-tfile.c	(working copy)
*************** add_procedure (const char *func_start,
*** 2352,2366 ****
  STATIC void
  initialize_init_file (void)
  {
    memset (&init_file, 0, sizeof (init_file));

    init_file.fdr.lang = langC;
    init_file.fdr.fMerge = 1;
    init_file.fdr.glevel = GLEVEL_2;

! #ifdef WORDS_BIG_ENDIAN
!   init_file.fdr.fBigendian = 1;
! #endif

    INITIALIZE_VARRAY (&init_file.strings, char);
    INITIALIZE_VARRAY (&init_file.symbols, SYMR);
--- 2352,2379 ----
  STATIC void
  initialize_init_file (void)
  {
+   union {
+     unsigned char c[4];
+     int i;
+   } endian_test;
+
    memset (&init_file, 0, sizeof (init_file));

    init_file.fdr.lang = langC;
    init_file.fdr.fMerge = 1;
    init_file.fdr.glevel = GLEVEL_2;

!   /* mips-tfile doesn't attempt to perform byte swapping and always writes
!      out integers in its native ordering.  For cross-compilers, this need
!      not be the same as either the host or the target.  The simplest thing
!      to do is skip the configury and perform an introspective test.  */
!   /* ??? Despite the name, mips-tfile is currently only used on alpha/Tru64
!      and would/may require significant work to be used in cross-compiler
!      configurations, so we could simply admit defeat and hard code this as
!      little-endian, i.e. init_file.fdr.fBigendian = 0.  */
!   endian_test.i = 1;
!   if (endian_test.c[3])
!     init_file.fdr.fBigendian = 1;

    INITIALIZE_VARRAY (&init_file.strings, char);
    INITIALIZE_VARRAY (&init_file.symbols, SYMR);


Roger
--