[Bug c/96793] New: __builtin_floor produces wrong result when rounding direction is FE_DOWNWARD

[chfast at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=96793

            Bug ID: 96793
           Summary: __builtin_floor produces wrong result when rounding
                    direction is FE_DOWNWARD
           Product: gcc
           Version: 10.2.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: c
          Assignee: unassigned at gcc dot gnu.org
          Reporter: chfast at gmail dot com
  Target Milestone: ---

Created attachment 49125
  --> https://gcc.gnu.org/bugzilla/attachment.cgi?id=49125&action=edit
Preprocessed test code

If the rounding direction is set to FE_DOWNWARD by fesetround(),
the __builtin_floor() result is -0 where it should be +0. Eg.
__builtin_floor(0.25) == -0.

The outputs are done with GCC 10.2.0 (x86_64-linux-gnu) with -O2
-frounding-math -lm. Full gcc -v output in cli.log file. This is also
reproducible in GCC 9 and trunk.


The test code to reproduce the bug checks if the result's sign is 0 as expected
(attached as builtin_floor_test.c):

enum { FE_DOWNWARD = 0x400 };

extern int fesetround(int rounding_direction);

__attribute__((noinline))
float builtin_floorf(float value)
{
    return __builtin_floorf(value);
}

int main()
{
    fesetround(FE_DOWNWARD);
    float result = builtin_floorf(0.25f);
    return __builtin_signbitf(result) != 0;
}


The __builtin_floor() generates the following assembly (part of
builtin_floor_test.s):

        movss   .LC1(%rip), %xmm2
        movss   .LC0(%rip), %xmm4
        movaps  %xmm0, %xmm3
        movaps  %xmm0, %xmm1
        andps   %xmm2, %xmm3
        ucomiss %xmm3, %xmm4
        jbe     .L2
        cvttss2sil      %xmm0, %eax
        pxor    %xmm3, %xmm3
        andnps  %xmm1, %xmm2
        cvtsi2ssl       %eax, %xmm3
        movaps  %xmm3, %xmm4
        cmpnless        %xmm0, %xmm4
        movss   .LC2(%rip), %xmm0
        andps   %xmm0, %xmm4
        subss   %xmm4, %xmm3
        movaps  %xmm3, %xmm0
        orps    %xmm2, %xmm0
.L2:
        ret