__builtin_isnanl() and invalid x87 80-bit floating point numbers

[Vincent Lefevre]

On 2017-04-25 17:17:05 +0800, Liu Hao wrote:
> Hello there,
> 
> mingw-w64 is having a potential bug here, but this seems related to GCC too.
> For your reference, details of the mingw-w64 problem can be found here
> <https://sourceforge.net/p/mingw-w64/bugs/602/>.
> 
> As for GCC, the following program:
> ```
> int main(){
>   long double ld = 0;
>   unsigned char data[10] = {
>     0x5b, 0x01, 0x04, 0x5e, 0x85, 0x00, 0x00, 0x00, 0xd8, 0x59
>   };
>   __builtin_memcpy(&ld, data, 10);
>   __builtin_printf("nan? %d\n", __builtin_isnanl(ld));
> }
> ```
> outputs `nan? 1`. This has been confirmed to be the case on GCC 6.3 on x64
> Windows and GCC 4.7 & 4.9 on x64 Linux.
> 
> According to <https://en.wikipedia.org/wiki/Extended_precision#x86_extended_precision_format>,
> the number in question is an invalid operand (effectively a _trap
> representation_), because its exponent is neither all-zeroes nor all-ones
> and its bit 63 is zero. Unlike NaNs, this invalid number can't be copied as
> a `long double`. x87 loads it successfully using the `FLD` instruction, but
> generates an #IA (invalid operand) exception if `FST` is used to store it
> into RAM, which isn't the case for a SNaN or QNaN.

It seems to be regarded as a NaN, not as a trap representation. On
Debian/unstable with an Intel(R) Xeon(R) CPU E5-2609 v3 (x86_64):

------------------------------------------------------------------
#include <stdio.h>

int main (void)
{
  long double ld = 0;
  unsigned char data[10] = {
    0x5b, 0x01, 0x04, 0x5e, 0x85, 0x00, 0x00, 0x00, 0xd8, 0x59
  };
  __builtin_memcpy(&ld, data, 10);
  ld += 1;
  printf ("%Lg\n", ld);
  return 0;
}
------------------------------------------------------------------

cventin:~> gcc-4.9 tst.c -o tst
cventin:~> ./tst
-nan

Or perhaps it's just for older processors? AFAIK, the Intel x87 80-bit
format is regarded as an extended precision format in IEEE 754-2008
(Section 3.7), and trap representations are not allowed since all
encodings must map to a floating-point datum (Section 3.2).

This encoding could either be interpreted like an unnormalized number
(but then transparently normalized) as suggested by IEEE 754-1985, or
it should encode a NaN.

-- 
Vincent Lefèvre <vincent@vinc17.net> - Web: <https://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <https://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)