[Bug tree-optimization/80758] New: isnan/isfinite/isinf value propagation

["drepper.fsp+rhbz at gmail dot com" <gcc-bugzilla at gcc dot gnu dot org>]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=80758

            Bug ID: 80758
           Summary: isnan/isfinite/isinf value propagation
           Product: gcc
           Version: 8.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: tree-optimization
          Assignee: unassigned at gcc dot gnu.org
          Reporter: drepper.fsp+rhbz at gmail dot com
  Target Milestone: ---

Consider the following code:

#define isnan(x) __builtin_isnan(x)
#define isfinite(x) __builtin_isfinite(x)

int f(double a, double b)
{
  if (!isfinite(a) || !isfinite(b))
    return 0;
  double c = a + b;
  return isnan(c) ? 0 : 1;
}

For x86-64 with the current trunk version (and probably all previous versions)
the generated code looks something like this:

        .cfi_startproc
        vmovq   .LC0(%rip), %xmm2
        vmovapd %xmm0, %xmm4
        vmovsd  .LC1(%rip), %xmm3
        xorl    %eax, %eax
        vandpd  %xmm2, %xmm4, %xmm4
        vucomisd        %xmm4, %xmm3
        jb      .L5
        vandpd  %xmm1, %xmm2, %xmm2
        vucomisd        %xmm2, %xmm3
        jb      .L5
        vaddsd  %xmm1, %xmm0, %xmm0
        xorl    %eax, %eax
        vucomisd        %xmm0, %xmm0
        setnp   %al
.L5:
        ret
        .cfi_endproc

The issue here is that the sum of two finite values will never be NaN.  It can
be ±Inf but not NaN.  The VRP information should contain necessary information
and use it in the __builtin_isnan code generation.