[Bug target/91721] New: Missed optimization for checking nan and comparison

[antoshkka at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=91721

            Bug ID: 91721
           Summary: Missed optimization for checking nan and comparison
           Product: gcc
           Version: 10.0
            Status: UNCONFIRMED
          Keywords: missed-optimization
          Severity: normal
          Priority: P3
         Component: target
          Assignee: unassigned at gcc dot gnu.org
          Reporter: antoshkka at gmail dot com
  Target Milestone: ---

Consider the example:

int doubleToString_0(double a) {
    if ( __builtin_isnan( a ) )
        return 1;
    else if ( a == 0. )
        return 2;

    return 3;
}

A suboptimal assembly with two `ucomisd` is generated for the above sample:

doubleToString_0(double):
  ucomisd xmm0, xmm0
  jp .L4
  ucomisd xmm0, QWORD PTR .LC0[rip]
  jnp .L8
.L5:
  mov eax, 3
  ret
.L8:
  jne .L5
  mov eax, 2
  ret
.L4:
  mov eax, 1
  ret
.LC0:
  .long 0
  .long 0


More optimal solution would be to do only the second `ucomisd` and check flags
for a NaN:

doubleToString_0(double):
  pxor xmm1, xmm1
  ucomisd xmm0, xmm1
  jp .L4
  je .L8
.L5:
  mov eax, 3
  ret
.L8:
  mov eax, 2
  ret
.L4:
  mov eax, 1
  ret