I have been adding rotate capability to AVR port and have come this bug in 
optabs.c: expand_binop()

This occurs during a rotate expansion. For example

target  = op0  rotated by op1

In the particular situation (code extract below) it tries a reverse rotate of (bits - op1). Where this expression is then expanded into an integer,
a negation or subtraction depending on type of op1 and target.

The expansion of the subtraction is using the mode of the target - it should be using the mode of op1.
The mode of the rotation  amount need not be the same as the target.

target:DI = Op0:DI rotate op1:HI

In my testcase it is not and I get asserts latter in simplfy_rtx with the mismatched inner/outer modes.

The negation mode looks equally wrong.

I found this on testcase 20010226-1.c with gcc 4.42 (line 55) - however, it would appear to be on 4.5 and perhaps earlier versions.

Revision 126370 added this particular code to optabs.c in July 2007



 /* If we were trying to rotate, and that didn't work, try rotating
    the other direction before falling back to shifts and bitwise-or.  */
 if (((binoptab == rotl_optab
   && optab_handler (rotr_optab, mode)->insn_code != CODE_FOR_nothing)
      || (binoptab == rotr_optab
      && optab_handler (rotl_optab, mode)->insn_code != CODE_FOR_nothing))
     && mclass == MODE_INT)
   {
     optab otheroptab = (binoptab == rotl_optab ? rotr_optab : rotl_optab);
     rtx newop1;
     unsigned int bits = GET_MODE_BITSIZE (mode);

     if (CONST_INT_P (op1))
   newop1 = GEN_INT (bits - INTVAL (op1));
     else if (targetm.shift_truncation_mask (mode) == bits - 1)
   newop1 = negate_rtx (mode, op1);
     else
   newop1 = expand_binop (mode, sub_optab,
                  GEN_INT (bits), op1,
                  NULL_RTX, unsignedp, OPTAB_DIRECT);