vax double precision broken

[Paul Koning]

Excerpt of message (sent 29 May 2002) by Stephen L Moshier:
> ...
> Regarding the question about single precision, the binary data structure
> that we produce matches what we found on three different vaxen
> at the time the emulator was written.  It is definitely _not_ the same
> as pdp-11 data structure.  From the beginning, vax offered a pdp-11
> emulation mode that gave you both single and double precision
> pdp-11 numbers; but that was not the default.  I am now speculating
> without having the books in front of me, but the default as I
> remember it gave you an IBM-compatible single precision that later was
> the basis of IEEE single precision format.  Vaxen go back to about 1979,
> while IEEE 754 was adopted in 1985.

I commented on this offline, but at Stephen's suggestion I'll repeat
it on the list.  Those of you with VAX expertise or, better yet, VAXen
to run stuff on, please look this over.  I'm in the process of
constructing a patch to real.c to fix the issues that Stephen raised
(working from the real.c cleanup that Roman has been doing) but it
sure would help to have this double-checked by others.

I have no explanation for observations that suggests VAX float is
different from PDP-11 float -- for "F" and "D" float, that is.  I
don't have access to VAX hardware, unfortunately.  I do have DEC
reference manuals, which have this to say:

1. VAX-11 Architecture Handbook, 1979 (I believe that's the first
edition of what PDP11s called "processor handbook" -- 11/780 only).

It describes only the original two float formats, F and D.  Page 9
says "Floating point values are stored using a signed 8-bit excess-128
exponent and a binary normalized fraction, using 4-byte and 8-byte
formats identical to those of the PDP-11."  And indeed the next page
shows diagrams identical to those in the PDP-11 Architecture Manual,
even going as far as to show 16-bit wide layouts.

2. VAX architecture reference manual (1987), page 7-9 shows the four
floating point formats, F, D, G, H.

Diagrams:

	31                16 15 14           7 6         0
	+-------------------+--+--------------+-----------+
	|     fraction      |S | exponent     | fraction  | :A
	+-------------------+--+--------------+-----------+
	F-floating data type

	31                16 15 14           7 6         0
	+-------------------+--+--------------+-----------+
	|     fraction      |S | exponent     | fraction  | :A
	+-------------------+--+--------------+-----------+
	|     fraction      |      fraction               | :A + 4
	+-------------------+-----------------------------+
	63                                               32
	D-floating data type

	31                16 15 14             4 3       0
	+-------------------+--+----------------+---------+
	|     fraction      |S |    exponent    |fraction | :A
	+-------------------+--+----------------+---------+
	|     fraction      |      fraction               | :A + 4
	+-------------------+-----------------------------+
	63                                               32
	G-floating data type

	31                16 15 14                       0
	+-------------------+--+--------------------------+
	|     fraction      |S |      exponent            | :A
	+-------------------+--+--------------------------+
	|     fraction      |      fraction               | :A + 4
	+-------------------+-----------------------------+
	|     fraction      |      fraction               | :A + 8
	+-------------------+-----------------------------+
	|     fraction      |      fraction               | :A + 12
	+-------------------+-----------------------------+
	127                                              96
	H-floating data type

(For F and D those are the same as in the earlier VAX-11 book, except
for presentation -- it is presented 32 bits at a time.)

In all cases, the fraction is normalized with the high order bit not
represented.  For F and D, the MSB of the fraction is bit 6.  For F,
the LSB is bit 16.  For D and G, it's bit 48.  For H, MSB is bit 31,
and LSB is bit 112.

Exponent is excess-128 for F and D, excess-1024 for G, and
excess-16384 for H.  In all cases, exponent field (as coded) == 0 is
used for representing float 0.0 if S is 0, and NaN if S is 1.

---
I just looked at my MIPS architecture manual, that being the closest
thing I have to a description of the IEEE float format.

The single and double IEEE formats do indeed look similar to VAX F and
G formats respectively, but they do NOT match.  There are three key
differences: the halfwords are in the opposite order (sign bit is bit
31 in each case for IEEE); the exponent bias is one less -- 127 and
1023 for IEEE vs. 128 and 1024 for VAX; and the hidden high order bit
is interpreted as the bit just to the left of the binary point (2**0)
in IEEE rather than just to the right (2**-1) as in VAX.  (In
addition, IEEE has both Inf and NaN, while DEC format only has NaN.
And IEEE has denormals; DEC does not.)

All that fits the data in www.netlib.org/port/Mach/r1mach.f: it shows
VAX value for Log10(2) as 0x209B3F9A vs. IEEE as 0x3E9A209B, so the
fraction is the same (1A209B as stored, i.e., 90209B if you include
the hidden bit) and the exponents differ by two, which is 1 due to the
different bias and one due to the different placement of the binary
point. 

     paul