FW: Unicode and C++

[Martin v. Loewis]

> I had a dekko at it. They are adding things like Klingon! and scholarly
> and ancient languages. Thats why I said 'living languages' and its a
> stated goal to fit all of the living languages in the BMP only. 

Yes, that was a stated goal. I think it turned out that it cannot be
fully achieved - they had to invent the Han unification. Even with
that, I think it still was not fully achieved. AFAIK, the plane 2
assignments prepared by the PR of China's national body are in-use,
life symbols. They aren't used on a day to day basis, though.

> But yes I do see the point in having one simple 32bit character. But
> it seems so extravagant for most data. Is it a political decision to
> have wchar_t as 32 bit due to old EUC stuff for *NIXes or is there a
> purely technical reason?

I don't exactly know whether any of the EUC sets required 32 bit
characters - but once the ABI said that wchar_t is 32 bits, you cannot
change it afterwards. One technical reason may have been efficiency,
as fetching a 16-bit value is more expensive than fetching a 32-bit
value on some RISC machines.

As for political reasons, I think that ISO 10646 also makes strong
point.

> After all if you can have UTF-8 favouring ASCII then why can't you
> have UCS-2 (UTF-16) favouring the BMP Unicode? Seems reasonable to
> me.

It's not a matter of favouring one, at a cost for the others - it's an
all-or-nothing thing. Either you support other planes or you
don't. Now, you can tell me that there is UTF-16, so you could support
at least some. 

However, I think applications will break if surrogates are ever
used. People have been told for years what advantages wide characters
have when it comes to indexing and determining the string length. All
those assumptions break with UTF-16, which is a MWCS (multi-word
character set :-), so you are back to field one, and could have used
UTF-8 in the first place.

UTF-16 was invented just for Microsoft, who are big supporters of
Unicode, but made the political decision to go for a 16 bit variant,
instead of doing the full job. While I can sympathize with that
decision in the early 1990's, they now have the problem of hacking
support for other planes into all these deployed systems.

Please note that wchar_t is a process character set, and the process
representation is not necessarily the external one. Instead, it seems
likely that UTF-8 will be the external one in many cases, and
converted to 32-bit wchar_t on input.

Regards,
Martin