MAX_FIXED_MODE_SIZE vs. __attribute__(mode) vs. ABIs

[Roger Sayle <roger at eyesopen dot com>]

I've a simplify-rtx patch that addresses a code quality regression on
AVR, but testing it is complicated by the fact that a cross-build to
avr-elf currently fails for mainline due to PR middle-end/18887.

Whilst I don't disagree that libgcc2.h may be generating intrinsics for
the wrong modes on 8-bit targets, the ICE is caused by a much deeper
interaction between the front and back-ends over MAX_FIXED_MODE_SIZE.

Consider, the following code fragment:

typedef int DItype __attribute__ ((mode (DI)));

extern DItype foo();

DItype bar()
{
  return foo() + 1;
}

and particularly what this code is supposed to do on backends that
#define MAX_FIXED_MODE_SIZE 32, such as h8300 or avr, i.e. that
indicate the processor can't hold anything larger than SImode in
a register (or more accurately they don't have enough registers
to hold the entire DImode pseudo at the same time).


The question then is who is responsible to avoiding initial RTL such
as:

(call_insn 10 9 19 (set (reg:DI 18 r18)
        (call (mem:HI (symbol_ref:HI ("foo") ...)


I'm not clear whether it is the front-ends who should check the
GET_MODE_BITSIZE on their __attribute__(mode)) specifications against
MAX_FIXED_MODE_SIZE and either issue a warning or use BLKmode instead,
or if its the backend for not not explicitly disqualifying DImode in
HARD_REGNO_MODE_OK (and other macros), or some other interaction.

Ultimately, we get an ICE in reload because it can't find a regclass
that can hold DImode (surprise, surprise), but its far from clear
whether this is a front-end problem (for introducing DImode in the
first place), a back-end problem (for not being more specific about
disallowing it) or a reload problem (that should be more carefull
before calling abort).

Bizarrely, the above code compiles without problems on gcc-3.4, but
mostly because we did less checking in reload, and the AVR actually
has 8 8bit registers that could be use to hold the DImode value.

Any advice would be appreciated.

Roger
--