Bug 106648 - [C++23] P2071 - Named universal character escapes
Summary: [C++23] P2071 - Named universal character escapes
Alias: None
Product: gcc
Classification: Unclassified
Component: c++ (show other bugs)
Version: unknown
: P3 normal
Target Milestone: ---
Assignee: Jakub Jelinek
Depends on:
Blocks: c++23-core
  Show dependency treegraph
Reported: 2022-08-16 16:57 UTC by Marek Polacek
Modified: 2022-08-26 07:32 UTC (History)
2 users (show)

See Also:
Known to work:
Known to fail:
Last reconfirmed: 2022-08-20 00:00:00

makeuname2c.cc (6.94 KB, text/plain)
2022-08-18 16:41 UTC, Jakub Jelinek
gcc13-pr106648-wip.patch.xz (214.30 KB, application/x-xz)
2022-08-19 16:35 UTC, Jakub Jelinek
gcc13-pr106648.patch.xz (220.82 KB, application/x-xz)
2022-08-20 17:02 UTC, Jakub Jelinek

Note You need to log in before you can comment on or make changes to this bug.
Description Marek Polacek 2022-08-16 16:57:13 UTC
See <https://wg21.link/p2071>.
Comment 1 Jakub Jelinek 2022-08-18 16:41:52 UTC
Created attachment 53471 [details]

I've so far written a generator of a space optimized radix tree for the Unicode name to codepoint mapping (this would be libcpp/makeuname2c.cc),
but will need to write a consumer of those arrays to actually implement the
Unicode name to codepoint mapping.
Comment 2 Jakub Jelinek 2022-08-19 16:35:30 UTC
Created attachment 53478 [details]

Updated WIP patch.  This has fixed bugs in the generator and a routine that implements the mapping (tested on names from glibc UTF-8 and https://eel.is/c++draft/lex.charset table so far), but isn't actually wired up for \N{name}.
It does for now just exact matching, for fixit hints we'll need to do something slightly different.
Comment 3 Jakub Jelinek 2022-08-20 17:02:35 UTC
Created attachment 53483 [details]

So far just lightly tested patch.

This handles did you mean hints using the Unicode UAX44-LM2 algorithm, but doesn't offer fixits fot it (not sure if it is possible in libcpp) and doesn't
use spellcheck* stuff for fallback suggestions (the amount of strings and their sizes are too huge to push them all into vector, but just walking all radix tree nodes, computing current name as we go and at each codepoint (including generated ones) compute Damerau-Levenshtein distance could work.  But spellcheck.{cc,h} are in gcc/ ...
Comment 4 GCC Commits 2022-08-26 07:28:06 UTC
The master branch has been updated by Jakub Jelinek <jakub@gcc.gnu.org>:


commit r13-2212-geb4879ab9053085a59b8d1594ef76487948bba7e
Author: Jakub Jelinek <jakub@redhat.com>
Date:   Fri Aug 26 09:24:56 2022 +0200

    c++: Implement C++23 P2071R2 - Named universal character escapes [PR106648]
    The following patch implements the
    C++23 P2071R2 - Named universal character escapes
    paper to support \N{LATIN SMALL LETTER E} etc.
    I've used Unicode 14.0, there are 144803 character name properties
    (including the ones generated by Unicode NR1 and NR2 rules)
    and correction/control/alternate aliases, together with zero terminators
    that would be 3884745 bytes, which is clearly unacceptable for libcpp.
    This patch instead contains a generator which from the UnicodeData.txt
    and NameAliases.txt files emits a space optimized radix tree (208765
    bytes long for 14.0), a single string literal dictionary (59418 bytes),
    maximum name length (currently 88 chars) and two small helper arrays
    for the NR1/NR2 name generation.
    The radix tree needs 2 to 9 bytes per node, the exact format is
    described in the generator program.  There could be ways to shrink
    the dictionary size somewhat at the expense of slightly slower lookups.
    Currently the patch implements strict matching (that is what is needed
    to actually implement it on valid code) and Unicode UAX44-LM2 algorithm
    loose matching to provide hints (that algorithm essentially ignores
    hyphens in between two alphanumeric characters, spaces and underscores
    (with one exception for hyphen) and does case insensitive matching).
    In the attachment is a WIP patch that shows how to implement also
    spellcheck.{h,cc} style discovery of misspellings, but I'll need to talk
    to David Malcolm about it, as spellcheck.{h,cc} is in gcc/ subdir
    (so the WIP incremental patch instead prints all the names to stderr).
    2022-08-26  Jakub Jelinek  <jakub@redhat.com>
            PR c++/106648
            * charset.cc: Implement C++23 P2071R2 - Named universal character
            escapes.  Include uname2c.h.
            (hangul_syllables, hangul_count): New variables.
            (struct uname2c_data): New type.
            (_cpp_uname2c, _cpp_uname2c_uax44_lm2): New functions.
            (_cpp_valid_ucn): Use them.  Handle named universal character escapes.
            (convert_ucn): Adjust comment.
            (convert_escape): Call convert_ucn even for \N.
            (_cpp_interpret_identifier): Handle named universal character escapes.
            * lex.cc (get_bidi_ucn): Fix up function comment formatting.
            (get_bidi_named): New function.
            (forms_identifier_p, lex_string): Handle named universal character
            * makeuname2c.cc: New file.  Small parts copied from makeucnid.cc.
            * uname2c.h: New generated file.
            * c-cppbuiltin.cc (c_cpp_builtins): Predefine
            __cpp_named_character_escapes to 202207L.
            * c-c++-common/cpp/named-universal-char-escape-1.c: New test.
            * c-c++-common/cpp/named-universal-char-escape-2.c: New test.
            * c-c++-common/cpp/named-universal-char-escape-3.c: New test.
            * c-c++-common/cpp/named-universal-char-escape-4.c: New test.
            * c-c++-common/Wbidi-chars-25.c: New test.
            * gcc.dg/cpp/named-universal-char-escape-1.c: New test.
            * gcc.dg/cpp/named-universal-char-escape-2.c: New test.
            * g++.dg/cpp/named-universal-char-escape-1.C: New test.
            * g++.dg/cpp/named-universal-char-escape-2.C: New test.
            * g++.dg/cpp23/feat-cxx2b.C: Test __cpp_named_character_escapes.
Comment 5 Jakub Jelinek 2022-08-26 07:32:44 UTC
Implemented for GCC 13.