Created attachment 36187 [details] One line patch to add C99 UTF-8 support in identifiers to gcc In response to FAQ > What is the status of adding the UTF-8 support for identifier names in GCC? and the request > Support for actual UTF-8 in identifiers is still pending (please contribute!) my observation is that UTF-8 in identifiers is easy to add to gcc by changing one line in the cpp preprocessor, provided a recent version of iconv is installed on the system. The patch is attached and has been tested for about 6 months. More information about this patch as well as unrelated information about getting cilkrts to work on ARM is available at https://www.raspberrypi.org/forums/viewtopic.php?p=802657
To check the installed version of iconv has C99 support type $ iconv --list | grep "C99" C99 $ which means that iconv is recent enough.
Related to bug 41374.
Have you tried -fextended-identifiers ?
I cannot say anything about the correctness of the patch, but I would expect such a patch to contain many testcases (at least similar to those that test for UCNs see https://gcc.gnu.org/ml/gcc-patches/2014-11/msg00337.html), patches need to be bootstrapped & regression tested and submitted to gcc-patches with a Changelog (https://gcc.gnu.org/wiki/GettingStarted#Basics:_Contributing_to_GCC_in_10_easy_steps). Please CC Joseph Myers when you submit.
There is no "C99" character set in glibc libiconv (after all, it's not a character set at all). Converting extended characters to UCNs like that would in any case be correct for C++ (provided you also convert $ ` @ and control characters other than those in the basic source character set) but not for C - but for C++, it would be necessary to keep track of the conversions to revert them in raw string literals. This requirement to revert such conversions in raw string literals (in C++14, see 2.5 [lex.pptoken] paragraph 3: "Between the initial and final double quote characters of the raw string, any transformations performed in phases 1 and 2 (trigraphs, universal-character-names, and line splicing) are reverted; this reversion shall apply before any d-char, r-char, or delimiting parenthesis is identified.") renders such an approach non-viable (it would break things that currently work); the conversions to UCNs have to take place within cpplib, not through an external iconv conversion. Note that cpplib identifier spelling preservation is now implemented <https://gcc.gnu.org/ml/gcc-patches/2014-11/msg00548.html>, which adds other ways in which it should be visible whether an identifier was represented with UTF-8 or UCNs.
From the webpage (current as of Aug 17, 2015) http://www.gnu.org/software/libiconv/ under *Details* it is described that the library provides support for the following encodings: Full Unicode UTF-8 UCS-2, UCS-2BE, UCS-2LE UCS-4, UCS-4BE, UCS-4LE UTF-16, UTF-16BE, UTF-16LE UTF-32, UTF-32BE, UTF-32LE UTF-7 C99, JAVA Therefore, I don't understand the statement that libiconv doesn't support C99 or that it isn't, somehow, a character set.
Please look at the Raspberry Pi forum post linked in the original report for more information about testing this patch. As the text describes there, the command line options -finput-charset=UTF-8 -fextended-identifiers are both needed in order to compile a UTF-8 input file containing unicode identifiers. I have included a small test program as another attachment. Searching on UTF-8 Identifiers in GCC will turn up a number of people asking for this feature and additional example codes that use UTF-8 identifers. The document "Unicode for the PCC C99 Compiler" available at http://pcc.ludd.ltu.se/documentation/ also contains example UTF-8 C99 input files which can be used to test the compiler. The one-line patch submitted above has also been tested in the sense that the compiler still bootstraps and has no trouble compiling thousands of lines of standard ASCII C input. The patch inserts "C99" in only one place as the uses of SOURCE_CHARSET are conflicted and changing them all to "C99" doesn't yield a working solution. In particular, the "C99" in _cpp_convert_input should not be considered the source character set appearing in the input files but rather an internal character set suitable for later parsing. As iconv is already a well debugged library, it would appear the risks of this patch are minor. Note however, the following problem: "C99" is probably not the correct for EBCDIC hosts. In that case it might be possible to write UCNs using trigraphs of the form ??/uXXXX and ??/UXXXXXXXX, however, as the number of people wanting to compile C source files with identifiers encoded using UTF-EBCDIC is likely zero, the easiest solution going forward is to modify the patch so it only applies to non-EBCDIC hosts. As there are already #ifdef's in the code to check for this, this does not add any new complexity to the code base.
Created attachment 36196 [details] Test program with UTF-8 identifiers... Compile this test program using gcc \ -finput-charset=UTF-8 -fextended-identifiers \ -o circle circle.c to check whether gcc can handle UTF-8 identifiers.
(In reply to Eric from comment #7) > also contains example UTF-8 C99 input files which can be used to test the > compiler. The one-line patch submitted above has also been tested in the > sense that the compiler still bootstraps and has no trouble compiling > thousands of lines of standard ASCII C input. I think what Joseph is saying is that your approach may work for the small examples that you have tested, but it would break things that are working fine right now (in particular raw string literals). Many of those things are not tested by a gcc bootstrap (but some of them should be tested by the regression testsuite, did you run that? Point 4 here: https://gcc.gnu.org/wiki/GettingStarted#Basics:_Contributing_to_GCC_in_10_easy_steps ) I hope Joseph can give you more details so you may try to implement this in the proper way. The only reason why GCC does not have UTF-8 support in identifiers is that no one had time to implement it yet, so your help is appreciated.
(In reply to Eric from comment #7) > command line options > > -finput-charset=UTF-8 -fextended-identifiers > > are both needed in order to compile a UTF-8 input file containing unicode Note also that since GCC 5.1: The option -fextended-identifiers is now enabled by default for C++, and for C99 and later C versions (https://gcc.gnu.org/gcc-5/changes.html) and the default C version is C11, thus it is enabled by default.
Sorry, glibc iconv (not libiconv) doesn't handle "C99". So your patch would not work on any GNU host in normal configurations of GCC (libiconv is a completely separate package and is only likely to be used on non-GNU hosts such as Windows, on GNU hosts iconv from glibc is normally used although it's possible to use libiconv there). You need to test cases such as that if a macro is defined twice, once with a UCN in its expansion and once with the equivalent character written in UTF-8, the difference in the expansion is diagnosed (whichever of all the valid UCNs for that character is the one used). And that the original spelling appears on the right hand side of a definition output with -dD. And that if (in C but not, properly, C++) a string contains a backslash followed by an extended character, this is properly diagnosed as an invalid escape sequence rather than being treated as \\u<something> or \\U<something>. See the tests in my spelling preservation patch <https://gcc.gnu.org/ml/gcc-patches/2014-11/msg00548.html>. (Stringizing isn't necessarily an issue here because of the special C rules about stringizing UCNs together with the C++ rule about converting to UCNs in phase 1 - the effect is that for C it's always OK to stringize as the extended character, though you can't stringize as a UCN if the extended character was originally written, while for C++ you have to stringize as a UCN.) And then you need tests of C++ programs with extended characters inside raw strings (like c-c++-common/raw-string-*.c, but none of those cover extended characters at present). And the patch needs to add all these tests to the testsuite.
I'm glad to know people like Joseph are working on UTF-8 in gcc. Last year I spent a week adding UTF-8 input support to pcc. At that time Microsoft Studio and clang already supported UTF-8 input files and I expected that gcc would do so in the next release. As this didn't happen, a few months ago I looked and developed a one-line patch to add this support to gcc. It appears the C preprocessor falls back to libiconv when it encounters a conversion not supported internally. From what I can tell this is enabled by default, though it is surely possible to disable it. I'm aware that C strings are often used to store 8-bit data, for example, to display various graphics characters from legacy code pages. I will run the regression tests as soon as possible to see what, if anything, has broken by my one-line patch. UCN quoting of UTF-8 input should happen only if the -finput-charset=UTF-8 flag is set and this is worth checking.
(In reply to Eric from comment #12) > I'm glad to know people like Joseph are working on UTF-8 in gcc. I think at the moment, neither Joseph nor anyone else is planning to work on this. There doesn't seem to be sufficient demand for this feature so that companies fund it or volunteers step up to implement it (you are the first one to do an attempt that I am aware of). > I spent a week adding UTF-8 input support to pcc. At that time Microsoft > Studio and clang already supported UTF-8 input files and I expected that gcc > would do so in the next release. Unfortunately, GCC has very few developers compared to Microsoft or Clang. Many things in GCC will never get done if new people do not contribute to its development. This is why if you want to see this feature, you are the best and perhaps the only person to make it happen. The problem is that this cannot be fixed by one-line patch, otherwise it would have been fixed a long time ago. * GCC cannot rely on libiconv being always present. It has to work with glibc's iconv, which is what is used in GNU/Linux. * Even if glibc's supported C99 conversion, this will break other things. * You need to add tests explicitly for various things (see Joseph's comments). The tests will be added to the GCC testsuite to prove that your patch works as it should and to make sure future changes do not break the tests. * At a minimum, look at all the gcc.dg/cpp/ucnid-*.c g++.dg/cpp/ucnid-*.c and see what happens if you replace the \uNNN with actual extended characters. * Joseph thinks that the best approach is to do the conversion from UTF-8 to UCNs "manually" within cpplib, such that you can handle all the corner cases of C/C++ (quoted strings, \µ, macro names,...)
While there may not be current demand for gcc to accept UTF-8 identifiers, the fact that clang and Visual Studio support this C99 feature means source code using Greek and accented characters in variable names is likely to become more prevalent over time. I have done a little testing to check by default whether string literals can contain arbitrary 8-bit data. This is used, for example, in legacy code which directly includes graphics characters from CP437. The original preprocessor specifies "UTF-8" as the default input character set and "UTF-8" as the internal character set. Then, if the internal and working character sets are identical no translation is done and arbitrary 8-bit data is passed through cleanly. A slight modification to my patch needs to be made to retain the same behavior. In particular, the patch now specifies both the internal and default input character sets to be "C99" so no translation is done by default. The improved patch also includes consideration of EBCDIC hosts. As iconv was installed on every GNU/Linux system I've tried, I'm not sure what is wrong with using the C99 mode present in newer releases. This achieves exactly the suggested result of converting all UTF-8 input to UCNs in the preprocessor while directly allowing other potentially useful conversions. Perhaps the configure script should be modified to check for a compatibile version of iconv and if one is not found resort to a manual conversion. Testing is still underway. After the standard regression tests are finished I will create new tests utf8id-.* which will be versions of the uncid-.* tests for native utf-8 files. I will also include a new test for arbitrary 8-bit string literals, to verify further compatibility.
Created attachment 36206 [details] Improved UTF-8 identifier patch Improved patch to support UTF-8 identifiers. This version by default does no translation unless -finput-charset=XXX is specified where XXX is something other than C99 and should not affect EBCDIC hosts.
With my second patch the command line must now include the options -finput-charset=UTF-8 -fextended-identifiers -fexec-charset=UTF-8 or otherwise C99 will also be used for the default execution character set. A better approach to maintain nearly 8-bit clean string literals by default might result from leaving the default input and execution characters sets as UTF-8 and setting the internal character set to C99 only when -fextended-identifiers is selected. Sorry for too many comments. I'll post a new patch when everything is ready and has been tested.
On Tue, 18 Aug 2015, ejolson at unr dot edu wrote: > As iconv was installed on every GNU/Linux system I've tried, I'm not sure what > is wrong with using the C99 mode present in newer releases. This achieves The iconv that is installed is glibc iconv. It has *nothing to do with* libiconv, a completely independent package. iconv --version will report a glibc version and iconv --list will produce a list not mentioning C99, e.g.: $ iconv --version iconv (Ubuntu EGLIBC 2.19-0ubuntu6.6) 2.19 $ iconv --list The following list contains all the coded character sets known. This does not necessarily mean that all combinations of these names can be used for the FROM and TO command line parameters. One coded character set can be listed with several different names (aliases). 437, 500, 500V1, 850, 851, 852, 855, 856, 857, 860, 861, 862, 863, 864, 865, 866, 866NAV, 869, 874, 904, 1026, 1046, 1047, 8859_1, 8859_2, 8859_3, 8859_4, 8859_5, 8859_6, 8859_7, 8859_8, 8859_9, 10646-1:1993, 10646-1:1993/UCS4, ANSI_X3.4-1968, ANSI_X3.4-1986, ANSI_X3.4, ANSI_X3.110-1983, ANSI_X3.110, ARABIC, ARABIC7, ARMSCII-8, ASCII, ASMO-708, ASMO_449, BALTIC, BIG-5, BIG-FIVE, BIG5-HKSCS, BIG5, BIG5HKSCS, BIGFIVE, BRF, BS_4730, CA, CN-BIG5, CN-GB, CN, CP-AR, CP-GR, CP-HU, CP037, CP038, CP273, CP274, CP275, CP278, CP280, CP281, CP282, CP284, CP285, CP290, CP297, CP367, CP420, CP423, CP424, CP437, CP500, CP737, CP770, CP771, CP772, CP773, CP774, CP775, CP803, CP813, CP819, CP850, CP851, CP852, CP855, CP856, CP857, CP860, CP861, CP862, CP863, CP864, CP865, CP866, CP866NAV, CP868, CP869, CP870, CP871, CP874, CP875, CP880, CP891, CP901, CP902, CP903, CP904, CP905, CP912, CP915, CP916, CP918, CP920, CP921, CP922, CP930, CP932, CP933, CP935, CP936, CP937, CP939, CP949, CP950, CP1004, CP1008, CP1025, CP1026, CP1046, CP1047, CP1070, CP1079, CP1081, CP1084, CP1089, CP1097, CP1112, CP1122, CP1123, CP1124, CP1125, CP1129, CP1130, CP1132, CP1133, CP1137, CP1140, CP1141, CP1142, CP1143, CP1144, CP1145, CP1146, CP1147, CP1148, CP1149, CP1153, CP1154, CP1155, CP1156, CP1157, CP1158, CP1160, CP1161, CP1162, CP1163, CP1164, CP1166, CP1167, CP1250, CP1251, CP1252, CP1253, CP1254, CP1255, CP1256, CP1257, CP1258, CP1282, CP1361, CP1364, CP1371, CP1388, CP1390, CP1399, CP4517, CP4899, CP4909, CP4971, CP5347, CP9030, CP9066, CP9448, CP10007, CP12712, CP16804, CPIBM861, CSA7-1, CSA7-2, CSASCII, CSA_T500-1983, CSA_T500, CSA_Z243.4-1985-1, CSA_Z243.4-1985-2, CSA_Z243.419851, CSA_Z243.419852, CSDECMCS, CSEBCDICATDE, CSEBCDICATDEA, CSEBCDICCAFR, CSEBCDICDKNO, CSEBCDICDKNOA, CSEBCDICES, CSEBCDICESA, CSEBCDICESS, CSEBCDICFISE, CSEBCDICFISEA, CSEBCDICFR, CSEBCDICIT, CSEBCDICPT, CSEBCDICUK, CSEBCDICUS, CSEUCKR, CSEUCPKDFMTJAPANESE, CSGB2312, CSHPROMAN8, CSIBM037, CSIBM038, CSIBM273, CSIBM274, CSIBM275, CSIBM277, CSIBM278, CSIBM280, CSIBM281, CSIBM284, CSIBM285, CSIBM290, CSIBM297, CSIBM420, CSIBM423, CSIBM424, CSIBM500, CSIBM803, CSIBM851, CSIBM855, CSIBM856, CSIBM857, CSIBM860, CSIBM863, CSIBM864, CSIBM865, CSIBM866, CSIBM868, CSIBM869, CSIBM870, CSIBM871, CSIBM880, CSIBM891, CSIBM901, CSIBM902, CSIBM903, CSIBM904, CSIBM905, CSIBM918, CSIBM921, CSIBM922, CSIBM930, CSIBM932, CSIBM933, CSIBM935, CSIBM937, CSIBM939, CSIBM943, CSIBM1008, CSIBM1025, CSIBM1026, CSIBM1097, CSIBM1112, CSIBM1122, CSIBM1123, CSIBM1124, CSIBM1129, CSIBM1130, CSIBM1132, CSIBM1133, CSIBM1137, CSIBM1140, CSIBM1141, CSIBM1142, CSIBM1143, CSIBM1144, CSIBM1145, CSIBM1146, CSIBM1147, CSIBM1148, CSIBM1149, CSIBM1153, CSIBM1154, CSIBM1155, CSIBM1156, CSIBM1157, CSIBM1158, CSIBM1160, CSIBM1161, CSIBM1163, CSIBM1164, CSIBM1166, CSIBM1167, CSIBM1364, CSIBM1371, CSIBM1388, CSIBM1390, CSIBM1399, CSIBM4517, CSIBM4899, CSIBM4909, CSIBM4971, CSIBM5347, CSIBM9030, CSIBM9066, CSIBM9448, CSIBM12712, CSIBM16804, CSIBM11621162, CSISO4UNITEDKINGDOM, CSISO10SWEDISH, CSISO11SWEDISHFORNAMES, CSISO14JISC6220RO, CSISO15ITALIAN, CSISO16PORTUGESE, CSISO17SPANISH, CSISO18GREEK7OLD, CSISO19LATINGREEK, CSISO21GERMAN, CSISO25FRENCH, CSISO27LATINGREEK1, CSISO49INIS, CSISO50INIS8, CSISO51INISCYRILLIC, CSISO58GB1988, CSISO60DANISHNORWEGIAN, CSISO60NORWEGIAN1, CSISO61NORWEGIAN2, CSISO69FRENCH, CSISO84PORTUGUESE2, CSISO85SPANISH2, CSISO86HUNGARIAN, CSISO88GREEK7, CSISO89ASMO449, CSISO90, CSISO92JISC62991984B, CSISO99NAPLPS, CSISO103T618BIT, CSISO111ECMACYRILLIC, CSISO121CANADIAN1, CSISO122CANADIAN2, CSISO139CSN369103, CSISO141JUSIB1002, CSISO143IECP271, CSISO150, CSISO150GREEKCCITT, CSISO151CUBA, CSISO153GOST1976874, CSISO646DANISH, CSISO2022CN, CSISO2022JP, CSISO2022JP2, CSISO2022KR, CSISO2033, CSISO5427CYRILLIC, CSISO5427CYRILLIC1981, CSISO5428GREEK, CSISO10367BOX, CSISOLATIN1, CSISOLATIN2, CSISOLATIN3, CSISOLATIN4, CSISOLATIN5, CSISOLATIN6, CSISOLATINARABIC, CSISOLATINCYRILLIC, CSISOLATINGREEK, CSISOLATINHEBREW, CSKOI8R, CSKSC5636, CSMACINTOSH, CSNATSDANO, CSNATSSEFI, CSN_369103, CSPC8CODEPAGE437, CSPC775BALTIC, CSPC850MULTILINGUAL, CSPC862LATINHEBREW, CSPCP852, CSSHIFTJIS, CSUCS4, CSUNICODE, CSWINDOWS31J, CUBA, CWI-2, CWI, CYRILLIC, DE, DEC-MCS, DEC, DECMCS, DIN_66003, DK, DS2089, DS_2089, E13B, EBCDIC-AT-DE-A, EBCDIC-AT-DE, EBCDIC-BE, EBCDIC-BR, EBCDIC-CA-FR, EBCDIC-CP-AR1, EBCDIC-CP-AR2, EBCDIC-CP-BE, EBCDIC-CP-CA, EBCDIC-CP-CH, EBCDIC-CP-DK, EBCDIC-CP-ES, EBCDIC-CP-FI, EBCDIC-CP-FR, EBCDIC-CP-GB, EBCDIC-CP-GR, EBCDIC-CP-HE, EBCDIC-CP-IS, EBCDIC-CP-IT, EBCDIC-CP-NL, EBCDIC-CP-NO, EBCDIC-CP-ROECE, EBCDIC-CP-SE, EBCDIC-CP-TR, EBCDIC-CP-US, EBCDIC-CP-WT, EBCDIC-CP-YU, EBCDIC-CYRILLIC, EBCDIC-DK-NO-A, EBCDIC-DK-NO, EBCDIC-ES-A, EBCDIC-ES-S, EBCDIC-ES, EBCDIC-FI-SE-A, EBCDIC-FI-SE, EBCDIC-FR, EBCDIC-GREEK, EBCDIC-INT, EBCDIC-INT1, EBCDIC-IS-FRISS, EBCDIC-IT, EBCDIC-JP-E, EBCDIC-JP-KANA, EBCDIC-PT, EBCDIC-UK, EBCDIC-US, EBCDICATDE, EBCDICATDEA, EBCDICCAFR, EBCDICDKNO, EBCDICDKNOA, EBCDICES, EBCDICESA, EBCDICESS, EBCDICFISE, EBCDICFISEA, EBCDICFR, EBCDICISFRISS, EBCDICIT, EBCDICPT, EBCDICUK, EBCDICUS, ECMA-114, ECMA-118, ECMA-128, ECMA-CYRILLIC, ECMACYRILLIC, ELOT_928, ES, ES2, EUC-CN, EUC-JISX0213, EUC-JP-MS, EUC-JP, EUC-KR, EUC-TW, EUCCN, EUCJP-MS, EUCJP-OPEN, EUCJP-WIN, EUCJP, EUCKR, EUCTW, FI, FR, GB, GB2312, GB13000, GB18030, GBK, GB_1988-80, GB_198880, GEORGIAN-ACADEMY, GEORGIAN-PS, GOST_19768-74, GOST_19768, GOST_1976874, GREEK-CCITT, GREEK, GREEK7-OLD, GREEK7, GREEK7OLD, GREEK8, GREEKCCITT, HEBREW, HP-GREEK8, HP-ROMAN8, HP-ROMAN9, HP-THAI8, HP-TURKISH8, HPGREEK8, HPROMAN8, HPROMAN9, HPTHAI8, HPTURKISH8, HU, IBM-803, IBM-856, IBM-901, IBM-902, IBM-921, IBM-922, IBM-930, IBM-932, IBM-933, IBM-935, IBM-937, IBM-939, IBM-943, IBM-1008, IBM-1025, IBM-1046, IBM-1047, IBM-1097, IBM-1112, IBM-1122, IBM-1123, IBM-1124, IBM-1129, IBM-1130, IBM-1132, IBM-1133, IBM-1137, IBM-1140, IBM-1141, IBM-1142, IBM-1143, IBM-1144, IBM-1145, IBM-1146, IBM-1147, IBM-1148, IBM-1149, IBM-1153, IBM-1154, IBM-1155, IBM-1156, IBM-1157, IBM-1158, IBM-1160, IBM-1161, IBM-1162, IBM-1163, IBM-1164, IBM-1166, IBM-1167, IBM-1364, IBM-1371, IBM-1388, IBM-1390, IBM-1399, IBM-4517, IBM-4899, IBM-4909, IBM-4971, IBM-5347, IBM-9030, IBM-9066, IBM-9448, IBM-12712, IBM-16804, IBM037, IBM038, IBM256, IBM273, IBM274, IBM275, IBM277, IBM278, IBM280, IBM281, IBM284, IBM285, IBM290, IBM297, IBM367, IBM420, IBM423, IBM424, IBM437, IBM500, IBM775, IBM803, IBM813, IBM819, IBM848, IBM850, IBM851, IBM852, IBM855, IBM856, IBM857, IBM860, IBM861, IBM862, IBM863, IBM864, IBM865, IBM866, IBM866NAV, IBM868, IBM869, IBM870, IBM871, IBM874, IBM875, IBM880, IBM891, IBM901, IBM902, IBM903, IBM904, IBM905, IBM912, IBM915, IBM916, IBM918, IBM920, IBM921, IBM922, IBM930, IBM932, IBM933, IBM935, IBM937, IBM939, IBM943, IBM1004, IBM1008, IBM1025, IBM1026, IBM1046, IBM1047, IBM1089, IBM1097, IBM1112, IBM1122, IBM1123, IBM1124, IBM1129, IBM1130, IBM1132, IBM1133, IBM1137, IBM1140, IBM1141, IBM1142, IBM1143, IBM1144, IBM1145, IBM1146, IBM1147, IBM1148, IBM1149, IBM1153, IBM1154, IBM1155, IBM1156, IBM1157, IBM1158, IBM1160, IBM1161, IBM1162, IBM1163, IBM1164, IBM1166, IBM1167, IBM1364, IBM1371, IBM1388, IBM1390, IBM1399, IBM4517, IBM4899, IBM4909, IBM4971, IBM5347, IBM9030, IBM9066, IBM9448, IBM12712, IBM16804, IEC_P27-1, IEC_P271, INIS-8, INIS-CYRILLIC, INIS, INIS8, INISCYRILLIC, ISIRI-3342, ISIRI3342, ISO-2022-CN-EXT, ISO-2022-CN, ISO-2022-JP-2, ISO-2022-JP-3, ISO-2022-JP, ISO-2022-KR, ISO-8859-1, ISO-8859-2, ISO-8859-3, ISO-8859-4, ISO-8859-5, ISO-8859-6, ISO-8859-7, ISO-8859-8, ISO-8859-9, ISO-8859-9E, ISO-8859-10, ISO-8859-11, ISO-8859-13, ISO-8859-14, ISO-8859-15, ISO-8859-16, ISO-10646, ISO-10646/UCS2, ISO-10646/UCS4, ISO-10646/UTF-8, ISO-10646/UTF8, ISO-CELTIC, ISO-IR-4, ISO-IR-6, ISO-IR-8-1, ISO-IR-9-1, ISO-IR-10, ISO-IR-11, ISO-IR-14, ISO-IR-15, ISO-IR-16, ISO-IR-17, ISO-IR-18, ISO-IR-19, ISO-IR-21, ISO-IR-25, ISO-IR-27, ISO-IR-37, ISO-IR-49, ISO-IR-50, ISO-IR-51, ISO-IR-54, ISO-IR-55, ISO-IR-57, ISO-IR-60, ISO-IR-61, ISO-IR-69, ISO-IR-84, ISO-IR-85, ISO-IR-86, ISO-IR-88, ISO-IR-89, ISO-IR-90, ISO-IR-92, ISO-IR-98, ISO-IR-99, ISO-IR-100, ISO-IR-101, ISO-IR-103, ISO-IR-109, ISO-IR-110, ISO-IR-111, ISO-IR-121, ISO-IR-122, ISO-IR-126, ISO-IR-127, ISO-IR-138, ISO-IR-139, ISO-IR-141, ISO-IR-143, ISO-IR-144, ISO-IR-148, ISO-IR-150, ISO-IR-151, ISO-IR-153, ISO-IR-155, ISO-IR-156, ISO-IR-157, ISO-IR-166, ISO-IR-179, ISO-IR-193, ISO-IR-197, ISO-IR-199, ISO-IR-203, ISO-IR-209, ISO-IR-226, ISO/TR_11548-1, ISO646-CA, ISO646-CA2, ISO646-CN, ISO646-CU, ISO646-DE, ISO646-DK, ISO646-ES, ISO646-ES2, ISO646-FI, ISO646-FR, ISO646-FR1, ISO646-GB, ISO646-HU, ISO646-IT, ISO646-JP-OCR-B, ISO646-JP, ISO646-KR, ISO646-NO, ISO646-NO2, ISO646-PT, ISO646-PT2, ISO646-SE, ISO646-SE2, ISO646-US, ISO646-YU, ISO2022CN, ISO2022CNEXT, ISO2022JP, ISO2022JP2, ISO2022KR, ISO6937, ISO8859-1, ISO8859-2, ISO8859-3, ISO8859-4, ISO8859-5, ISO8859-6, ISO8859-7, ISO8859-8, ISO8859-9, ISO8859-9E, ISO8859-10, ISO8859-11, ISO8859-13, ISO8859-14, ISO8859-15, ISO8859-16, ISO11548-1, ISO88591, ISO88592, ISO88593, ISO88594, ISO88595, ISO88596, ISO88597, ISO88598, ISO88599, ISO88599E, ISO885910, ISO885911, ISO885913, ISO885914, ISO885915, ISO885916, ISO_646.IRV:1991, ISO_2033-1983, ISO_2033, ISO_5427-EXT, ISO_5427, ISO_5427:1981, ISO_5427EXT, ISO_5428, ISO_5428:1980, ISO_6937-2, ISO_6937-2:1983, ISO_6937, ISO_6937:1992, ISO_8859-1, ISO_8859-1:1987, ISO_8859-2, ISO_8859-2:1987, ISO_8859-3, ISO_8859-3:1988, ISO_8859-4, ISO_8859-4:1988, ISO_8859-5, ISO_8859-5:1988, ISO_8859-6, ISO_8859-6:1987, ISO_8859-7, ISO_8859-7:1987, ISO_8859-7:2003, ISO_8859-8, ISO_8859-8:1988, ISO_8859-9, ISO_8859-9:1989, ISO_8859-9E, ISO_8859-10, ISO_8859-10:1992, ISO_8859-14, ISO_8859-14:1998, ISO_8859-15, ISO_8859-15:1998, ISO_8859-16, ISO_8859-16:2001, ISO_9036, ISO_10367-BOX, ISO_10367BOX, ISO_11548-1, ISO_69372, IT, JIS_C6220-1969-RO, JIS_C6229-1984-B, JIS_C62201969RO, JIS_C62291984B, JOHAB, JP-OCR-B, JP, JS, JUS_I.B1.002, KOI-7, KOI-8, KOI8-R, KOI8-RU, KOI8-T, KOI8-U, KOI8, KOI8R, KOI8U, KSC5636, L1, L2, L3, L4, L5, L6, L7, L8, L10, LATIN-9, LATIN-GREEK-1, LATIN-GREEK, LATIN1, LATIN2, LATIN3, LATIN4, LATIN5, LATIN6, LATIN7, LATIN8, LATIN9, LATIN10, LATINGREEK, LATINGREEK1, MAC-CENTRALEUROPE, MAC-CYRILLIC, MAC-IS, MAC-SAMI, MAC-UK, MAC, MACCYRILLIC, MACINTOSH, MACIS, MACUK, MACUKRAINIAN, MIK, MS-ANSI, MS-ARAB, MS-CYRL, MS-EE, MS-GREEK, MS-HEBR, MS-MAC-CYRILLIC, MS-TURK, MS932, MS936, MSCP949, MSCP1361, MSMACCYRILLIC, MSZ_7795.3, MS_KANJI, NAPLPS, NATS-DANO, NATS-SEFI, NATSDANO, NATSSEFI, NC_NC0010, NC_NC00-10, NC_NC00-10:81, NF_Z_62-010, NF_Z_62-010_(1973), NF_Z_62-010_1973, NF_Z_62010, NF_Z_62010_1973, NO, NO2, NS_4551-1, NS_4551-2, NS_45511, NS_45512, OS2LATIN1, OSF00010001, OSF00010002, OSF00010003, OSF00010004, OSF00010005, OSF00010006, OSF00010007, OSF00010008, OSF00010009, OSF0001000A, OSF00010020, OSF00010100, OSF00010101, OSF00010102, OSF00010104, OSF00010105, OSF00010106, OSF00030010, OSF0004000A, OSF0005000A, OSF05010001, OSF100201A4, OSF100201A8, OSF100201B5, OSF100201F4, OSF100203B5, OSF1002011C, OSF1002011D, OSF1002035D, OSF1002035E, OSF1002035F, OSF1002036B, OSF1002037B, OSF10010001, OSF10010004, OSF10010006, OSF10020025, OSF10020111, OSF10020115, OSF10020116, OSF10020118, OSF10020122, OSF10020129, OSF10020352, OSF10020354, OSF10020357, OSF10020359, OSF10020360, OSF10020364, OSF10020365, OSF10020366, OSF10020367, OSF10020370, OSF10020387, OSF10020388, OSF10020396, OSF10020402, OSF10020417, PT, PT2, PT154, R8, R9, RK1048, ROMAN8, ROMAN9, RUSCII, SE, SE2, SEN_850200_B, SEN_850200_C, SHIFT-JIS, SHIFT_JIS, SHIFT_JISX0213, SJIS-OPEN, SJIS-WIN, SJIS, SS636127, STRK1048-2002, ST_SEV_358-88, T.61-8BIT, T.61, T.618BIT, TCVN-5712, TCVN, TCVN5712-1, TCVN5712-1:1993, THAI8, TIS-620, TIS620-0, TIS620.2529-1, TIS620.2533-0, TIS620, TS-5881, TSCII, TURKISH8, UCS-2, UCS-2BE, UCS-2LE, UCS-4, UCS-4BE, UCS-4LE, UCS2, UCS4, UHC, UJIS, UK, UNICODE, UNICODEBIG, UNICODELITTLE, US-ASCII, US, UTF-7, UTF-8, UTF-16, UTF-16BE, UTF-16LE, UTF-32, UTF-32BE, UTF-32LE, UTF7, UTF8, UTF16, UTF16BE, UTF16LE, UTF32, UTF32BE, UTF32LE, VISCII, WCHAR_T, WIN-SAMI-2, WINBALTRIM, WINDOWS-31J, WINDOWS-874, WINDOWS-936, WINDOWS-1250, WINDOWS-1251, WINDOWS-1252, WINDOWS-1253, WINDOWS-1254, WINDOWS-1255, WINDOWS-1256, WINDOWS-1257, WINDOWS-1258, WINSAMI2, WS2, YU > exactly the suggested result of converting all UTF-8 input to UCNs in the Which, as I have explained, is fundamentally incompatible with C++ requirements on raw string literals (namely that UTF-8 within such a string appears as UTF-8 bytes in the resulting object, while a \u or \U sequence appears as such in the resulting object). Proper C++ semantics require a conversion that is aware of the lexical context and only converts to UCNs in certain contexts. The string literal R"(\u00C0)" must contain the six bytes \u00C0 plus the trailing null byte. The string literal R"(À)" (given UTF-8 as the multibyte encoding of the execution character set) must contain the two bytes of that character's UTF-8 encoding plus the trailing null byte. See how lex_raw_string deals with reverting trigraph and line splicing transformations; conversions to UCNs would need similarly reverting if done at all (it's probably better to do the conversions later, only in the corner cases where it's actually visible whether such a conversion was done, lexing as UTF-8 as far as possible).
Thanks Joseph for the clarification about the two different versions of iconv. I was admittedly confused about this until moments ago. Anyway, I just discovered that libiconv doesn't support conversions to and from the IBM1047 EBCDIC character set and this causes some of the regression tests to fail. Coupled with the fact that C99 isn't supported in the glibc version of iconv this creates a little problem with my patch. You mention a bigger problem which I had not thought about: the C++ semantics of raw strings. Processing UCNs in C++ code apparently requires surprisingly deep syntactic analysis. Raw literals seem to appear in the gnu99 and gnu11 extensions to C as well. Amusingly, if I understand the C++ specifications www.open-std.org/jtc1/sc22/wg21/docs/papers/2012/n3337.pdf trigraphs are supposed to be interpreted before any other processing takes place. However, the simple code #include <stdio.h> int main(){ char p1[]="??/u00E4"; char p2[]=R"(??/u00E4)"; char p3[]=R"(\u00E4)"; printf("%s or %s or %s\n",p1,p2,p3); return 0; } compiled with $ g++ -std=c++11 pp.c produces output ä or ??/u00E4 or \u00E4 which illustrates that g++ does not process trigraphs inside raw string literals. Admittedly I'm looking at the draft standard, but I don't think this is something which changed suddenly in the final draft. Clearly, my patch makes a further mess of raw string literals in gcc. My first reaction is that raw string literals were not well thought out, but I suppose bad standards are sometimes better than no standards. At anyrate, there appears no easy way of supporting both UTF-8 identifiers and raw literal strings. My plan for now is to take a break and keep my UTF-8 identifier support as a one-line patch reliant on libiconv which breaks EBCDIC encodings and raw string literals.
On Tue, 18 Aug 2015, ejolson at unr dot edu wrote: > which illustrates that g++ does not process trigraphs inside raw string > literals. Admittedly I'm looking at the draft standard, but I don't think this As stated in [lex.pptoken] in both C++11 and C++14: "Between the initial and final double quote characters of the raw string, any transformations performed in phases 1 and 2 (trigraphs, universal-character-names, and line splicing) are reverted; this reversion shall apply before any d-char, r-char, or delimiting parenthesis is identified.". Yes, the positioning of this in the standard may be confusing.... That is, the effect is more or less as if trigraphs weren't processed inside raw strings (but the implementation involves undoing trigraph substitutions, as described in the standard). I think the right way to implement UTF-8 in identifiers involves making lex_identifier handle UTF-8 (when extended identifiers are enabled), and making _cpp_lex_direct handle bytes with the high bit set as potentially[*] starting identifiers (requiring the same handling of normalization state as for the other cases of characters starting identifiers, of course). If you do that, then raw strings and all the corner cases of spelling preservation fall out naturally (though they still need testcases added to the testsuite). [*] I think the right rule for C is that UTF-8 for a character not allowed in identifiers should produce a preprocessing token on its own rather than an error for an invalid character in an identifier (and similarly, such a character after the start of the identifier should terminate the identifier and produce such a preprocessing token). Unless and until someone implements the C++ phase 1 conversion to UCNs, it would seem reasonable to follow this rule for C++ as well.
I've been looking at the code in lex_identifier as well as what goes on in forms_identifier_p and so forth. As some point each identifier needs to be stored in the symbol table using ht_lookup_with_hash. Proper functioning requires that UTF-8 and UCN representations of the same unicode characters are treated as the same symbol. Thus, there needs to be some point at which the identifiers are regularized to be either all UTF-8 or all UCN escaped ASCII. As gcc is working with UCNs right now, the obvious implementation allocates temporary memory to hold the UCN escaped ASCII version of an UTF-8 identifier and then frees it again after calling ht_lookup. Any comments would be appreciated.
_cpp_interpret_identifier converts UCNs to UTF-8 which is the canonical internal form for identifiers - for UTF-8 in identifiers, you just need to pass in straight through unmodified there. (cpplib takes care to store the original spelling of the identifier as well for purposes for which that matters, but that's simply a matter of lex_identifier calling cpp_lookup on the original spelling as well as using _cpp_interpret_identifier to get the canonical version.) So you never need to convert UTF-8 to UCNs in order to handle UTF-8 in identifiers (cpplib has logic to do so when needed for output, but you don't need to add anything new in that regard). You do need to decode UTF-8 into character values for the code that checks normalization, which characters are allowed at the start of identifiers, etc., just as the existing code decodes UCNs into such values. (But as I noted, a UCN not allowed in identifiers is lexed as part of an identifier, which is then considered invalid, whereas a UTF-8 character not allowed in identifiers should be lexed as a separate pp-token. However, UTF-8 for a character allowed in identifiers but not at the start of an identifier should, I think, be lexed as an identifier character even at the start of an identifier, and then give an error for an invalid identifier if it appears at the start of an identifier. That's my reading of the syntax productions in the C standard.) You can ignore anything claiming to handle UTF-EBCDIC.
Has there been any progress on this since 2015? I'm maintaining a project that uses the International Phonetic Alphabet (IPA) internally. My life would be much easier if I could use identifiers like aʊ or dʒ. Both are valid C++ identifiers supported by Clang, Xcode and Visual Studio, but not supported by GCC. My knowledge of compilers is very limited, so I'm afraid I can't be of practical help. But I'd like to point out that there is indeed demand for this feature -- see for example this StackOverflow question: <http://stackoverflow.com/questions/12692067/and-other-unicode-characters-in-identifiers-not-allowed-by-g#>
An important patch. Is there a similar patch for versions later than 5.2.0 of gcc? I'm looking for gcc-7.2.1-2 patch for unicode idenfifiers.
(In reply to spoa@eircom.net from comment #23) > An important patch. Is there a similar patch for versions later than 5.2.0 > of gcc? I'm looking for gcc-7.2.1-2 patch for unicode idenfifiers. The patch above is not recommended due to the problems mentioned above. The recommended work-around is given here: https://gcc.gnu.org/wiki/FAQ#utf8_identifiers Guidelines for a proper implementation are given in comment #21.
Many thanks Manu. The to_UCN.sh script works well. The only trouble was that my include file also contain unusual characters with diacritic marks and the script changes these file names to UCN also. So compiler cant find them. I had to re-edit the .cpp file manually after conversion to UCN to change the include file names back. But in spite of that, it is useful and enables coding with much greater choice of words for identifiers. Much easier for me to read my code. Thanks again.
Created attachment 46618 [details] Patch with test cases that implements extended chars in identifiers Hi All- I am interested in helping out with this if there is still interest to support this feature. (Full disclosure, I don't have any experience with the gcc codebase, but I do have a lot of experience developing with gcc.) I took a crack at implementing it based on Joseph's outline in Comment #21 and the rest of the discussion in this thread. The patch is attached, including test cases. (I more or less took all the existing ucnid* test cases and adapted them for this, plus added a couple extra ones.) It seems to work fine, as far as interpreting the identifiers and bootstrapping clean, and test cases also pass except for one that I'll mention below, but I have many comments + questions as well: 1. The number of changes to libcpp is actually pretty small. All the work to recognize UTF-8 happens in forms_identifier_p(), so that the existing fast paths for regular characters are not affected, and that extended chars end up getting treated just like UCNs for the most part. forms_identifier_p() makes use of a new utility _cpp_valid_utf8_in_identifier() in charset.c that is similar to the existing _cpp_valid_ucn() and handles the UTF8 details. 2. otherwise _cpp_interpret_identifier() and lex_identifier() didn't need any changes. The former could be optimized a bit, it always allocates a temporary buffer, even though the buffer is only needed if UCNs appear. (This is the case already in the case of dollar signs that end up in this code path too.) Probably it's not a big deal though. 3. Invalid UTF-8 is left alone and parsed as stray tokens, the same as now. 4. Regarding the case of codepoints not allowed to appear in an identifier. In C, I did as Joseph suggested, or at least I tried to. The grammar specifies that an identifier ends once an illegal character is encountered, so this is how it works now, and then the disallowed UTF-8 forms a stray token next. It was not clear to me though whether this stray token should consist of just the next 1 byte of the input, or the entire disallowed UTF-8 character. Currently it's just the next byte because that's how things worked out of the box. Changing it wouldn't be too hard, just means the default case of _cpp_lex_direct()'s main switch statement would need to try to read a UTF char rather than a byte. In C++, I think UCNs or UTF-8 in identifiers should be treated identically in all respects, unless I misunderstand things (because technically the UTF-8 was supposed to be converted to UCNs in translation phase 1), so in that case a disallowed codepoint does not end the token but rather triggers an invalid character error. 5. There is a problem with diagnostics output when the source contains UTF-8 characters. The locator caret ends up in the wrong place, I assume just because this code is not aware of the multibyte encoding. That much is not specific to my patch, it exists already now e.g. with: $ cat t.cpp const char* x = "ππππππππππππππ"; int y = z; $ g++ -c t.cpp t.cpp:1:57: error: ‘z’ was not declared in this scope const char* x = "ππππππππππππππ"; int y = z; ^ The bigger problem though is in layout::print_source_line() which colorizes the source lines. It seems to end up attempting to colorize just the first byte, even for UTF-8, which makes the output no longer valid. I tried to look into it but I wasn't sure what are the implications, e.g. would it require some much larger overhaul of diagnostics infrastructure anyway to get this right, and would it perhaps be better just to disable colorization in the presence of UTF-8 input or something like this, for the meantime. As an example of what I mean, from preprocessing this (in c99 mode): -------- int ٩; -------- I get: t3.c:1:5: error: universal character ٩ is not valid at the start of an identifier 1 | int ٩; But if color is enabled, the output gets corrupted because ANSI escapes are inserted between the two bytes of the multibyte character on the 2nd line of diagnostics. 6. There is also a problem with formatting the output of some diagnostics, e.g. when I compile this: --------------- int π = 3; int x = π2; --------------- I get: t2.cpp:2:9: error: ‘π2’ was not declared in this scope; did you mean ‘\xcf\x80’? This is also not specific to this patch and occurs the same if UCN is used: $ cat t.cpp int \u03C0 = 3; int x = \u03C02; $ g++ -c t.cpp t.cpp:2:9: error: ‘π2’ was not declared in this scope int x = \u03C02; ^~~~~~~ t.cpp:2:9: note: suggested alternative: ‘\xcf\x80’ int x = \u03C02; ^~~~~~~ 7. What is the expected output from gcc -E of this code? ------- int π; -------- Currently it outputs: int \U000003c0; So curiously, it's as if C++ required translation of extended chars to UCNs is happening, so I think this output is actually potentially correct in C++ mode? But it is also this way in C mode which I think is probably not expected. It seems to come from cpp_output_token() which does not make use of the "original spelling" data structures. I am not sure about this one but probably the right solution is not much work, if someone knows what that might be? This is also the reason that one of the new testcases (gcc/testsuite/gcc.dg/cpp/ucnid-13-utf8.c) fails, this: #define Á 1 also preprocesses (in -E -dD) to include UCNs. I am not sure what is expected here. 8. There are tests (e.g. gcc/testsuite/gcc.dg/ucnid-10.c) which verify that when the locale is not utf8, diagnostics use UCNs instead of raw UTF8. I am not sure if this still makes sense when the files themselves contain UTF8, but that was the behavior that came out so I maintained these tests as well. Thanks for taking a look at this, and for all your work on gcc. I am happy to work on this if someone has any feedback on what I tried so far. -lewis
Created attachment 46620 [details] second attempt at posting the patch Sorry, the previous patch I sent doesn't seem to show correctly in Bugzilla. I think probably because the test cases include invalid UTF-8 and also latin1-encoded characters, it may have caused the whole file to be misinterpreted as the wrong encoding? It only affects the test cases but I am sending it here as binary and hoping that it is easier to read then. Thanks!
On Mon, 22 Jul 2019, lhyatt at gmail dot com wrote: > I am interested in helping out with this if there is still interest to support > this feature. (Full disclosure, I don't have any experience with the gcc > codebase, but I do have a lot of experience developing with gcc.) Thanks for working on this! I encourage sending this to gcc-patches once a few fixes have been made and you've done the legal paperwork, see <https://gcc.gnu.org/contribute.html>. I'm wary of the MIN use in _cpp_lex_direct, as this is performance-critical code so it's not clear an extra operation should be added for every token. I'd rather put the check for UTF-8 in the default case (a case that should in practice be rare), with a goto from there to the case of identifiers. As a coding style matter, note that in various places sentences in comments should start with a capital letter. > 5. There is a problem with diagnostics output when the source contains UTF-8 > characters. The locator caret ends up in the wrong place, I assume just because > this code is not aware of the multibyte encoding. That much is not specific to > my patch, it exists already now e.g. with: This seems like it should have a separate bug filed for it (I don't see any currently open bugs for this issue). > The bigger problem though is in layout::print_source_line() which colorizes the > source lines. It seems to end up attempting to colorize just the first byte, > even for UTF-8, which makes the output no longer valid. I tried to look into it > but I wasn't sure what are the implications, e.g. would it require some much > larger overhaul of diagnostics infrastructure anyway to get this right, and > would it perhaps be better just to disable colorization in the presence of > UTF-8 input or something like this, for the meantime. And this should probably also have a separate bug filed (whether or not it can occur without this patch applied). > This is also not specific to this patch and occurs the same if UCN > is used: This also seems like a matter for filing a separate bug. Or maybe two separate bugs, one for C and one for C++, since the fixes might be different. For C, the suggestion of \xcf\x80 looks like a missing call to identifier_to_locale when printing an identifier using %qs - but the C++ code is using %qE, which should use identifier_to_locale automatically, so I'm not sure what's wrong there. > 7. What is the expected output from gcc -E of this code? > > ------- > int π; > -------- > > Currently it outputs: > int \U000003c0; > > So curiously, it's as if C++ required translation of extended chars to UCNs is > happening, so I think this output is actually potentially correct in C++ mode? > But it is also this way in C mode which I think is probably not expected. It > seems to come from cpp_output_token() which does not make use of the "original > spelling" data structures. I am not sure about this one but probably the right > solution is not much work, if someone knows what that might be? I don't think the -E output matters much here; it's not specified by the standard. The results of stringizing *are* more precisely defined (the relevant tests stringize twice to verify those results). Strictly, for C++ stringizing twice (for extended characters including $ @ `) should make the conversion of such characters to UCNs visible (in strings, not just in identifiers), because, unlike C, C++ does not have the special rule making it implementation-defined whether the \ of a UCN in a string literal is doubled when stringizing. I don't think that's something you need to fix, however, since there's no attempt to implement that conversion for C++ at present, but it does make a couple of the C++ tests in your patch strictly invalid. > This is also the reason that one of the new testcases > (gcc/testsuite/gcc.dg/cpp/ucnid-13-utf8.c) fails, this: > > #define Á 1 > > also preprocesses (in -E -dD) to include UCNs. I am not sure what is expected > here. There is definitely no need to preserve spelling there (it's not even possible in general, since the same macro name can be spelt differently in otherwise identical definitions of the same macro; it's only a constraint violation if either macro argument names or the RHS are different, not if the name of the macro itself is spelt differently). So the right thing is to test that the output in that case uses a UCN. > 8. There are tests (e.g. gcc/testsuite/gcc.dg/ucnid-10.c) which verify that > when the locale is not utf8, diagnostics use UCNs instead of raw UTF8. I am not > sure if this still makes sense when the files themselves contain UTF8, but that > was the behavior that came out so I maintained these tests as well. Yes, I think that's correct.
(In reply to joseph@codesourcery.com from comment #28) > Thanks for working on this! I encourage sending this to gcc-patches once > a few fixes have been made and you've done the legal paperwork, see > <https://gcc.gnu.org/contribute.html>. > Thank you very much for taking a look and for the feedback. I will incorporate all this and send to gcc-patches. Regarding the copyright assignment, I couldn't quite discern from this link what I need to do next... it seems like I need someone to email me the necessary form, is that correct? I will also file additional bug reports for the diagnostics-related stuff; I believe I can construct test cases that do not depend on this patch for those.
https://git.savannah.gnu.org/cgit/gnulib.git/plain/doc/Copyright/request-assign.future is the form to complete and send to assign@gnu.org (to do an assignment covering past and future changes to GCC, which is usually the best one to use).
Author: jsm28 Date: Thu Sep 19 19:56:11 2019 New Revision: 275979 URL: https://gcc.gnu.org/viewcvs?rev=275979&root=gcc&view=rev Log: Support extended characters in C/C++ identifiers (PR c/67224) libcpp/ChangeLog 2019-09-19 Lewis Hyatt <lhyatt@gmail.com> PR c/67224 * charset.c (_cpp_valid_utf8): New function to help lex UTF-8 tokens. * internal.h (_cpp_valid_utf8): Declare. * lex.c (forms_identifier_p): Use it to recognize UTF-8 identifiers. (_cpp_lex_direct): Handle UTF-8 in identifiers and CPP_OTHER tokens. Do all work in "default" case to avoid slowing down typical code paths. Also handle $ and UCN in the default case for consistency. gcc/Changelog 2019-09-19 Lewis Hyatt <lhyatt@gmail.com> PR c/67224 * doc/cpp.texi: Document support for extended characters in identifiers. * doc/cppopts.texi: Likewise. gcc/testsuite/ChangeLog 2019-09-19 Lewis Hyatt <lhyatt@gmail.com> PR c/67224 * c-c++-common/cpp/ucnid-2011-1-utf8.c: New test. * g++.dg/cpp/ucnid-1-utf8.C: New test. * g++.dg/cpp/ucnid-2-utf8.C: New test. * g++.dg/cpp/ucnid-3-utf8.C: New test. * g++.dg/cpp/ucnid-4-utf8.C: New test. * g++.dg/other/ucnid-1-utf8.C: New test. * gcc.dg/cpp/ucnid-1-utf8.c: New test. * gcc.dg/cpp/ucnid-10-utf8.c: New test. * gcc.dg/cpp/ucnid-11-utf8.c: New test. * gcc.dg/cpp/ucnid-12-utf8.c: New test. * gcc.dg/cpp/ucnid-13-utf8.c: New test. * gcc.dg/cpp/ucnid-14-utf8.c: New test. * gcc.dg/cpp/ucnid-15-utf8.c: New test. * gcc.dg/cpp/ucnid-2-utf8.c: New test. * gcc.dg/cpp/ucnid-3-utf8.c: New test. * gcc.dg/cpp/ucnid-4-utf8.c: New test. * gcc.dg/cpp/ucnid-6-utf8.c: New test. * gcc.dg/cpp/ucnid-7-utf8.c: New test. * gcc.dg/cpp/ucnid-9-utf8.c: New test. * gcc.dg/ucnid-1-utf8.c: New test. * gcc.dg/ucnid-10-utf8.c: New test. * gcc.dg/ucnid-11-utf8.c: New test. * gcc.dg/ucnid-12-utf8.c: New test. * gcc.dg/ucnid-13-utf8.c: New test. * gcc.dg/ucnid-14-utf8.c: New test. * gcc.dg/ucnid-15-utf8.c: New test. * gcc.dg/ucnid-16-utf8.c: New test. * gcc.dg/ucnid-2-utf8.c: New test. * gcc.dg/ucnid-3-utf8.c: New test. * gcc.dg/ucnid-4-utf8.c: New test. * gcc.dg/ucnid-5-utf8.c: New test. * gcc.dg/ucnid-6-utf8.c: New test. * gcc.dg/ucnid-7-utf8.c: New test. * gcc.dg/ucnid-8-utf8.c: New test. * gcc.dg/ucnid-9-utf8.c: New test. Added: trunk/gcc/testsuite/c-c++-common/cpp/ucnid-2011-1-utf8.c trunk/gcc/testsuite/g++.dg/cpp/ucnid-1-utf8.C trunk/gcc/testsuite/g++.dg/cpp/ucnid-2-utf8.C trunk/gcc/testsuite/g++.dg/cpp/ucnid-3-utf8.C trunk/gcc/testsuite/g++.dg/cpp/ucnid-4-utf8.C trunk/gcc/testsuite/g++.dg/other/ucnid-1-utf8.C trunk/gcc/testsuite/gcc.dg/cpp/ucnid-1-utf8.c trunk/gcc/testsuite/gcc.dg/cpp/ucnid-10-utf8.c trunk/gcc/testsuite/gcc.dg/cpp/ucnid-11-utf8.c trunk/gcc/testsuite/gcc.dg/cpp/ucnid-12-utf8.c trunk/gcc/testsuite/gcc.dg/cpp/ucnid-13-utf8.c trunk/gcc/testsuite/gcc.dg/cpp/ucnid-14-utf8.c trunk/gcc/testsuite/gcc.dg/cpp/ucnid-15-utf8.c trunk/gcc/testsuite/gcc.dg/cpp/ucnid-2-utf8.c trunk/gcc/testsuite/gcc.dg/cpp/ucnid-3-utf8.c trunk/gcc/testsuite/gcc.dg/cpp/ucnid-4-utf8.c trunk/gcc/testsuite/gcc.dg/cpp/ucnid-6-utf8.c trunk/gcc/testsuite/gcc.dg/cpp/ucnid-7-utf8.c trunk/gcc/testsuite/gcc.dg/cpp/ucnid-9-utf8.c trunk/gcc/testsuite/gcc.dg/ucnid-1-utf8.c trunk/gcc/testsuite/gcc.dg/ucnid-10-utf8.c trunk/gcc/testsuite/gcc.dg/ucnid-11-utf8.c trunk/gcc/testsuite/gcc.dg/ucnid-12-utf8.c trunk/gcc/testsuite/gcc.dg/ucnid-13-utf8.c trunk/gcc/testsuite/gcc.dg/ucnid-14-utf8.c trunk/gcc/testsuite/gcc.dg/ucnid-15-utf8.c trunk/gcc/testsuite/gcc.dg/ucnid-16-utf8.c trunk/gcc/testsuite/gcc.dg/ucnid-2-utf8.c trunk/gcc/testsuite/gcc.dg/ucnid-3-utf8.c trunk/gcc/testsuite/gcc.dg/ucnid-4-utf8.c trunk/gcc/testsuite/gcc.dg/ucnid-5-utf8.c trunk/gcc/testsuite/gcc.dg/ucnid-6-utf8.c trunk/gcc/testsuite/gcc.dg/ucnid-7-utf8.c trunk/gcc/testsuite/gcc.dg/ucnid-8-utf8.c trunk/gcc/testsuite/gcc.dg/ucnid-9-utf8.c Modified: trunk/gcc/ChangeLog trunk/gcc/doc/cpp.texi trunk/gcc/doc/cppopts.texi trunk/gcc/testsuite/ChangeLog trunk/libcpp/ChangeLog trunk/libcpp/charset.c trunk/libcpp/internal.h trunk/libcpp/lex.c
Implemented for GCC 10.
This is a big enough feature that it should probably get an entry in gcc-10/changes.html
(In reply to Eric Gallager from comment #33) > This is a big enough feature that it should probably get an entry in > gcc-10/changes.html I emailed a suggested patch to that effect here: https://gcc.gnu.org/ml/gcc-patches/2020-01/msg01667.html. I can commit if it looks OK. Thanks!
(In reply to Lewis Hyatt from comment #34) > (In reply to Eric Gallager from comment #33) > > This is a big enough feature that it should probably get an entry in > > gcc-10/changes.html > > I emailed a suggested patch to that effect here: > https://gcc.gnu.org/ml/gcc-patches/2020-01/msg01667.html. I can commit if it > looks OK. Thanks! With GCC 10 release imminent, would anyone be able to confirm it's OK to push this to changes.html please? Thanks so much. https://gcc.gnu.org/pipermail/gcc-patches/2020-April/544343.html
If the patch is in, it should be ok. Or ask in gcc-patches for someone to commit on your behalf. Gerald is very helpful. Just make sure the subject of the email is very clear. On Fri, 1 May 2020, 16:12 lhyatt at gmail dot com, <gcc-bugzilla@gcc.gnu.org> wrote: > https://gcc.gnu.org/bugzilla/show_bug.cgi?id=67224 > > --- Comment #35 from Lewis Hyatt <lhyatt at gmail dot com> --- > (In reply to Lewis Hyatt from comment #34) > > (In reply to Eric Gallager from comment #33) > > > This is a big enough feature that it should probably get an entry in > > > gcc-10/changes.html > > > > I emailed a suggested patch to that effect here: > > https://gcc.gnu.org/ml/gcc-patches/2020-01/msg01667.html. I can commit > if it > > looks OK. Thanks! > > With GCC 10 release imminent, would anyone be able to confirm it's OK to > push > this to changes.html please? Thanks so much. > https://gcc.gnu.org/pipermail/gcc-patches/2020-April/544343.html > > -- > You are receiving this mail because: > You are on the CC list for the bug.
Redoing a few CCs that got removed without being marked as removed in the bug history; presumably from the server migration