Implementing Universal Character Names in identifiers

[Joseph S. Myers]

On 4 Nov 2002, Kai Henningsen wrote:

> > I don't think there's anything in the language standards to permit
> > normalization to NFC as described there.
> 
> I don't think even considering to do without can be justified. You do  
> Unicode, you normalize. Anything else is insane.

A C- or C++-aware text editor (and it must be to write UCNs in the source
file when the user entered Unicode characters; direct Unicode in the
source is a matter for "phase 0" transformations outside the scope of the
standard) is insane and manifestly broken if it doesn't produce NFC
sequences, of UCNs permitted in identifiers when the Unicode is in
identifiers.

Users will inevitably have to deal with normalized and unnormalized
sequences looking the same, in filesystems which will just use
byte-sequences.  GCC can at least give an error, if a combining character
(not allowed in identifiers) is encountered, suggesting that the input
might not be NFC-normalized (with an appropriate index entry for the term
in the manual).

> IIRC, if you implement them as doubly-indirect, tables of *all* the bit- 
> attributes of the current Unicode version (over the complete 20.x bit  
> range) take up something like 20 KB or so. (The trick is that you first  
> unify stretches of equal bit sequences - mainly all-ones and all-zeroes -  
> and then unify stretches of equal pointers [or indices] to those.)

That's the character properties tables.  What of the tables showing what
characters each character decomposes to?

-- 
Joseph S. Myers
jsm28@cam.ac.uk