Re: libgcj/9802: Bug in surrogate handling in Unicode to UTF-8 conversion

[James Clark <jjc at jclark dot com>]

Mark Wielaard wrote:
> Thanks for the bug report.
> Your suggested fix seems obviously correct and I verified that making
> sure that avail is always decremented makes String.getBytes("UTF-8")
> work (read not throw an ArrayIndexOutOfBoundException).
>
> But while creating a test case I noticed that for your example we return
> two bytes: {0xf0, 0x90} but other implementations return four bytes
> {0xf0, 0x90, 0x8c, 0x80}. I don't know enough of Unicode and UTF-8
> encoding to know what is correct or why.

Four bytes is correct.

> If someone has a quick reference to the relevant definitions and/or a
> testsuite for these kind of things that would be higly appreciated.

RFC 2279 <http://www.ietf.org/rfc/rfc2279.txt> describes UTF-8
RFC 2781 <http://www.ietf.org/rfc/rfc2781.txt> describes UTF-16

The code is doing a conversion of UTF-16 (represented as chars) to UTF-8 
represented as bytes.

The problem is that String::getBytes assumes that when converter->write 
returns some value N it means that the result of converting N chars has 
been completely written to the output buffer, whereas in fact 
Output_UTF8.write may have some trailing bytes in the representation of 
the last character still to write.   This can happen not just with 
surrogates.  There is probably a similar problem with other multibyte 
encodings (e.g. SJIS).

It's not immediately obvious to me whether the right fix is to

- extend the interface of UnicodeToBytes to indicate whether there are 
still pending bytes to write and change String::getBytes to use this,

- change Output_UTF8 not to convert a char unless there's room for its 
complete representation in the output buffer, or

- change String::getBytes to keep writing even after all characters have 
been converted until no bytes are output.

James

http://gcc.gnu.org/cgi-bin/gnatsweb.pl?cmd=view%20audit-trail&database=gcc&pr=9802