[Bug c/91450] New: __builtin_mul_overflow(A,B,R) wrong code if product < 0, *R is unsigned, and !(A&B)

[eggert at cs dot ucla.edu]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=91450

            Bug ID: 91450
           Summary: __builtin_mul_overflow(A,B,R) wrong code if product <
                    0, *R is unsigned, and !(A&B)
           Product: gcc
           Version: 9.2.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: c
          Assignee: unassigned at gcc dot gnu.org
          Reporter: eggert at cs dot ucla.edu
  Target Milestone: ---

GCC generates wrong code for this program:

int a = -2;
int b = 1;

int
main (void)
{
  unsigned long long r;
  return __builtin_mul_overflow (a, b, &r);
}

The program should exit with status 1 because the mathematical product -2 does
not fit in unsigned long long. However, it exits with status 0, for both x86-64
and x86.

Looking at the generated code, the problem occurs when (a<0) != (b<0), i.e.,
the mathematical product is nonpositive. In this case, overflow should be
reported if the product is negative, i.e., if a!=0 & b!=0. However, the
generated code reports overflow if (a&b) != 0. That is an invalid optimization,
as illustrated in the sample program where a is -2 and b is 1 so overflow
occurs, but (a&b) == 0 so __builtin_mul_overflow incorrectly reports no
overflow.