Unicode support for libstdc++ (was RE: How about basic_string<UTF-8> ?)

[Christophe PIERRET <cpierret@businessobjects.com>]

A rather lengthy reply follows.
On May 19, 1999 7:28 AM, Shiv Shankar Ramakrishnan [SMTP:Shiv@pspl.co.in]
wrote:
> Hi,
> First of thank you everybody for all the insights provided.
> 
> |UTF-8 extensions for basic_string<char> would be incompatible
> |with the design of the library.
> 
> Okay that clears up things wrt the Std.
> 
> |way to handle UTF characters is in basic_string<wchar_t>, or wstring: 
> |a "character" in a wstring _really_is_ a character in UTF.  
> 
> This sounds sensible enough as long as one can convert them back to the
> raw bytes(unsigned char) form to send to code that expects it in that
form.
> 
> |The Standard place for code that understands UTF-8 is the codecvt<> 
> |facet which performs the conversion to and from wide characters.
> 
> Information about locale<> and codevct<> etc is sketchy at best. Where
does
> one find more info about all this? Stroustrup 3e skips over this since its
> beyond the scope of the book. Other than the Std is there any other source
> of info for all this?
> 
> Some of my own thoughts after all the inputs follow -
> 
> 1. Okay so basic_string<UTF8> is a bad idea. So how about some other class
>    like Christophe was suggesting, which would have a lot of
basic_string<>
>    like behaviour.

For the traits, I 've done something like this:

// this way there is no confusion between char* and unsigned char*, 
// and it protects from signed arithmetic in mask and shift operations
typedef unsigned char UTF8; 

class utf8_traits {
public:
	typedef UTF8 char_type;    
	typedef wchar_t int_type;    
	typedef size_t pos_type;

	static size_t length(const UTF8*) const;
	
	// returns length in char for an UTF-8 string of byte_len bytes
	static size_t char_length(const UTF8*, size_t byte_len)  const;
	
	// takes a zero terminated UTF-8 string and returns length in char
	static size_t char_length(const UTF8*)  const;
	
	static size_t copy( UTF8* , const UTF8* src , size_t dest_len);

	// Copies dest_len chars from UTF-8 string src and translate it into
UCS-4 to dest
            // dest must have at least dest_len wchar_t storage space
            // src must contain at least dest_len characters
	static size_t copy( wchar_t* dest , const UTF8* src , size_t
dest_len);

	// determine if one byte is an UTF-8 lead byte
	static bool is_lead_byte(UTF8 byte) {
		return (byte & 0xc0) != 0x80;
	}	

	// return the number of bytes of the UTF-8 character 
	// whose lead byte is 'lead_byte'
	static size_t char_nbytes(UTF8 lead_byte) { 
		return _S_nbytes[lead_byte]; 
	}
	
	// returns the length in bytes of an UCS-4 character 
	// once converted to an UTF-8 character
	static size_t char_nbytes(wchar_t ch) { // wchar_t assumed to be
unsigned !
		size_t bytes;
		if (ch < 0x80) {			bytes = 1;
		} else if (ch < 0x800) {		bytes = 2;
		} else if (ch < 0x10000) {		bytes = 3;
		} else if (ch < 0x200000) {	bytes = 4;
		} else if (ch < 0x4000000) {	bytes = 5;
		} else {				bytes = 6;
		}
		return bytes;
	}
	// And so on ...
private:
	// contains the number of bytes of an UTF-8 char indexed by lead
byte
	static const _S_nbytes[UCHAR_MAX]; 
};

For the string class, I see two approaches :

The simplest (should templatize to add allocator support) :

class utf8_string {
public:
	const UTF8* c_str() const { return m_Impl.c_str(); }
 // ... redefine each member of basic_string<> that makes sense with an
inline redirection to implementation class
 // iterator and reverse_iterator can be left alone, because we sometime
want to work at the byte level
	typedef basic_string<UTF8>::iterator iterator;
	typedef basic_string<UTF8>::reverse_iterator reverse_iterator;
 //  Add 'char_iterator' and 'char_reverse_iterator' that work with
'wchar_t' as their value_type
	class char_iterator {
	public:
		typedef wchar_t value_type;
		// ...
	};
	class char_reverse_iterator {
	public:
		typedef wchar_t value_type;
		// ...
	};
//  Duplicate some members to work on wchar_t for ease of interoperability
	utf8_string( const UTF8* );
	utf8_string( const wchar_t* );
	utf8_string( const UTF8*, size_type byte_len);
	utf8_string( const wchar_t, size_type len);
// And so on ...
private:
	basic_string<UTF8> m_Impl;
};

The other way, you just reimplement all operations you need.
It's the best way to go but also the longest ...

( Sorry, I've already done something like this but due to the (f.....g)
french intellectual property laws, I cannot disclose it. )

> 
> 2. Somebody correct me if I am wrong here. The primary purpose of strings
>    in C++ is to ease writing code which manipulates string like data. Now
>    aren't we missing the point if we don't have a class to handle UTF8
>    strings in C++. Any class will do as long as it is highly usable.
> 
> 3. A lot of problems in UTF8 arises due to the fact that a UTF8 char has
>    a variable length. So how do the other multibyte charsets (like MBCS
>    in Windows) handle this? Well they have functions to do all operations
>    including something like *p++. So can't we encapsulate all that in a
>    class alongwith extra functions to give length and char_length etc.
See the preceding utf8_traits class.
MBCS on windows doesn't use *p++ but the _tcsxxx family of functions and the
"tchar.h" header
_tcsnextc is the function you use to do p++ ...


> 
> 4. Isn't handling wide chars getting too messy with all this? There are so
>    many formats like UTF-8/16,UCS-2/4. Seems to be something very wrong if
>    we can't have classes in C++ that can handle all these and
interconvert.
>    Of course that seems to be like asking for the moon ;-) But I'm sure
>    something can be done.
What can you do ?
You choose one processing format (here UCS-4, 32 bits) and stick to it for
processing.
Then, you'll have to interchange data with other systems or files, you'll
use UTF-8 (no byte endianness problem) or any other format.
UCS-2 (pure 16 bit) is deprecated , so don't use it.
UTF-16 is UCS-2 with surrogates ( a pair of surrogates <=> 2x16 bits <=> one
unicode char)
UTF-16 is the format advocated and supported by the Unicode Consortium.
Since UTF-16 is also a varying length format, the only one suitable for a
basic_string<> is UCS-4 (wchar_t is assumed to be UCS-4)
UCS-4 is defined in "ISO/IEC 10646-1 : Universal Multiple-Octet Coded
Character Set  (UCS)".


> 
> 5. Lastly we *ought* to have a good string class that allows one to write
>    international strings easily and convert them to other formats (at
least
>    UTF8). If this just happens to be wstring with codecvt<>'s then that is
>    good enough.
That's why I have put some new members to convert from and to UTF-8/UCS-4 in
the prototype class.

> 
> I guess asking for a new basic_string<> like class for UTF8 may be too
> much so we should at least have what Nathan is suggesting i.e. codecvt<>
> for UTF8. What do you all think?
> Thanks,
> Shiv
There are two ways to do this:
* define a UTF-8 locale where codecvt<> works on  UTF-8
* specialize codecvt<> for unsigned char and assume unsigned char is UTF-8.
I would personally prefer the later.
What do you think of it ?

Anyway, we will still lack intelligent facets like:
collate<wchar_t>
ctype<wchar_t>

For ctype<>, we could use some existing work as a basis: 
uctype ( http://www.ccil.org/~cowan/uctype-2.0.tar.gz ) from John Cowan
ucdata ( ftp://crl.nmsu.edu/CLR/multiling/unicode/ucdata.tar.gz ) from Mark
Leisher
uctype is more compact ( 6Ko only for tables ) but less adaptable ( cannot
alter the tables without recompiling)

BSD investigates them for inclusion or inspiration. 
BSD Contact: G. Adam Stanislav        mailto:adam@whizkidtech.net  (?)

For collate<>, Unicode Collation Algorithm or ISO 14651 would be nice, but
it's hard work.
I can provide some assistance in this area ( done it also, but cannot
disclose |o( 
There is no free source available in this domain ... perhaps we should ask
Mark Davis (one of the authors of UCA) ?
GNU glibc collation algorithm is not sufficiently adaptable for Unicode ( as
far as I know ) because it uses a hash table for rules ... Another approach
would use trie-like structures to store weights associated with chars as
well as canonical decomposition of chars.
And then apply one of UCA or 14651 to collate.
Using these techniques, we have very good performance, better than Windows
dumb collation (currently Windows collation doesn't handle as much cases as
UCA or 14651).
It is even possible using smart iterators to have the same algorithm work
either on UTF-16, UCS-4 or UTF-8 (done it)


What is UTF-8 ?
===============
   UTF-8 encodes UTF-16 (16 bits Unicode) or UCS-4 (32 bits Unicode)
characters as a varying number of
   octets, where the number of octets, and the value of each, depend on
   the integer value assigned to the character in ISO/IEC 10646 (Unicode).
This
   transformation format has the following characteristics (all values
   are in hexadecimal):

   -  Character values from 0000 0000 to 0000 007F (US-ASCII repertoire)
      correspond to octets 00 to 7F (7 bit US-ASCII values). A direct
      consequence is that a plain ASCII string is also a valid UTF-8
      string.

   -  US-ASCII values do not appear otherwise in a UTF-8 encoded
      character stream.  This provides compatibility with file systems
      or other software (e.g. the printf() function in C libraries) that
      parse based on US-ASCII values but are transparent to other
      values.

   -  Round-trip conversion is easy and efficient between UTF-8 and either
of UCS-4,
      UCS-2.

   -  The first octet of a multi-octet sequence indicates the number of
      octets in the sequence.

   -  Character boundaries are easily found from anywhere in an octet
      stream.

   -  The lexicographic sorting order of UCS-4 strings is preserved.  Of
      course this is of limited interest since the sort order is not
      culturally valid in either case.

   -   string search algorithms ,like Boyer-Moore-Horspool, can be used as
is.

Reference information:
	"UTF-8, a transformation format of ISO 10646", F. Yergeau ( Request
for Comments: 2279, January 1998)
	"ISO/IEC 10646-1 : Universal Multiple-Octet Coded Character Set
(UCS)"



Christophe Pierret