When building 64 bit with an optimization setting of -O2, -O3, or -Os, some binary comparisons with nan values fail to produce false. All comparisons (==, >, <, etc) where one (or both) of the two values is nan should produce false. There are no warnings or other error messages during compilation. The attached test demonstrates the error with the final assert failing. The max() function should return the maximum of the two values, ignoring nans. i.e., max(5, nan) should be 5. This fails with gcc3.4.6 -O2 -m64. Strangely, switching the order of the final portion of the return to something like: return (a > b ? a : (b > a ? b : (isNaN(b) ? a : b))); causes it to work. Also, replacing isNaN() with a call to the slower, non-inline isnan() also makes the function work.
Created attachment 13353 [details] preprocessed example
Are you sure you did not have -ffast-math on the command line?
-ffast-math was not used. Adding that option actually makes the test work.
I can't reproduce this in any recent versions of GCC. Even in GCC 4.7.3 and GCC 4.6.4 it works and we get no assert.
Thanks for looking at this old bug. I had totally forgotten about it myself! I agree that it does not seem to affect any recent version of GCC. It was likely fixed in GCC 4, and doesn't seem to affect any later versions. There's no good reason to spend time fixing such an old version, so this bug can probably be closed as it is no longer relevant.
Just closing as fixed.