Re: Universal Character Names, v2

[Zack Weinberg <zack at codesourcery dot com>]

martin@v.loewis.de (Martin v. Löwis) writes:

> Zack Weinberg <zack@codesourcery.com> writes:
>
>> ... which I disagree with.  I am rejecting this patch until you
>> implement support for Unicode as she is spoke, which means UAX#15
>> including normalization, not whatever nonsense is in the C and C++
>> standards.
>
> Can you elaborate why you consider this approach technically
> superiour?

So, first off, a meta-issue:  I do not think any support for extended
identifiers should appear in GNU C/C++/ObjC until this issue is
resolved.  Once we have something out there in the wild, users will
come to rely on its behavior, and we don't want to change it from
under them.

Now, let me run down a series of principles.  Hopefully, they are in
increasing order of controversy, which is conveniently also increasing
order of how flexible I'm willing to be about them.

For rationale, I should say that I operate under the assumption
that text editors cannot be trusted to do anything right.  I assume
that we will see source files with unnormalized identifiers; with
inconsistent encoding for the same identifier; with \u escapes used in
one place and raw UTF8 in another place; and so on.  I do not think it
is appropriate for us to punish users for bugs in software they may
not control, especially since the presence of a bug may be debatable.
For instance, I can see someone deliberately designing a text editor
to leave existing \u escapes alone, so as not to introduce unnecessary
deltas in a version-control system, but write out modified lines in
UTF-8.

I know that some people in this discussion are not making the same
assumption, which is why I put it up front.

Thus:

1. Extended identifiers written using \u escapes should be treated
identically to extended identifiers written using a binary encoding of
Unicode (UTF8, UTF16, etc), modulo the fact that we may not support
binary encodings yet.

This means that whatever constraints are applied to the set of
legitimate identifiers writable with \u escapes, should also be
sensible constraints for binary encodings, and vice versa.  In
particular, it means that if we accept the sequence U+0041 U+0308
(that's LATIN CAPITAL LETTER A, COMBINING DIARESIS) in some context
when encoded in UTF-8, then we should also accept A\u0308 in that
context.

2. Visually identical identifiers should be treated as the same
identifier.

To continue the above example, U+00C4 (LATIN CAPITAL LETTER A WITH
DIARESIS) is expected to be displayed using exactly the same visual
representation as U+0041 U+0308.  A user may have no way -- not even
an inconvenient way -- to distinguish them.  Treating them as distinct
will only cause GCC to reject programs that are apparently entirely
correct; or worse, to silently miscompile them.

This is why I insist on normalization.

3. ISO 10646 (Unicode) is updated more frequently than ISO 9899 (C)
and ISO 14882 (C++).  It is reasonable to expect that future revisions
of the latter two standards will augment the lists of acceptable code
points as further identifier characters are added to Unicode.  As a
convenience to our users, we should accept all the plausible
identifier characters in the latest revision of Unicode, not just
whatever revision was current the last time C or C++ was revised.

I'm willing to be flexible on this one.  The ideal situation in my
view would be to ship the current version of UnicodeData.txt with each
GCC release and allow the user to update it later, with consonant
effect on the set of acceptable identifiers, but that may be too much
work.  Hardwiring the codepoints and the normalization map from the
current version of Unicode into each release of GCC would also be
acceptable.

> Apart from the implementation difficulties, I see the following
> problems with this requirement:
>
> 1. It is underspecified, as UAX#15 leaves a number of alternatives for
>    language designers:
>    a) which Unicode version?
>    b) which normalization form?

The most current as of any given release of GCC, and NFC.  This should
naturally be documented.

> 2. It extends the languages, by allowing identifiers which must be
>    rejected in a conforming implementation. Can you propose an
>    implementation strategy that allows proper implementation of the
>    -pedantic option in this case?

Keep the list from C99 around for validation purposes, when
-pedantic.  This would have to happen before normalization.

> 3. It restricts the languages, by disallowing identifiers that are
>    allowed in the language definition.

It shouldn't.  Examples?

> 4. It modifies the languages, by treating identifiers as equal which
>    are not to be treated equal in the language definition.

This is deliberate; see above.  Or are you aware of examples where NFC
merges identifiers that are not visually identical?

Neil Booth <neil@daikokuya.co.uk> writes:

> However, what is your opinion about leading digits and other such
> weirdness?  Do you intend to do any sanity checking?

Extend the notion of a preprocessing number according to the principle
that inserting a blank at any point in an identifier should produce
two valid preprocessing tokens.

Don't extend the notion of a phase-7 numeric constant.  That's too
hairy for me right now.

Identifiers should probably not permit leading 'digit-like' extended
characters.

> It appears that UAX#15 is unsuitable for C and C++ in unmodified
> form. For example, it does not allow underscores (U+005F) as the first
> character of an identifier; I think this restriction of C is not
> acceptable.
>
> What modifications to <identifier_start> and <identifier_extend> do I
> need to make?

UAX15's rules for acceptable identifier characters could apply only to
extended source characters.  U+005F LOW LINE is in the basic source
character set so it continues to be acceptable.

"Joseph S. Myers" <jsm28@cam.ac.uk> writes:

> Again, the C and C++ standards do not use Unicode, they use ISO/IEC
> 10646.

ISO/IEC procedural rules should not stand in the way of doing it right.

zw