Re: Windows Unicode and GCC

[Zack Weinberg <zackw at panix dot com>]

> I think that CPP should try to determine the encoding for each file
> and not use a single encoding for every file.  It should look for
> a unicode header when it opens a file (original c source or any
> include), and if it doesn't find one, use the default: -finput-charset,
> LC_CTYPE, UTF-8, until it's done processing that file.  Note that
> vim is reads files saved with unicode headers without problem.

This is a desired feature, but one that no one has ever had time
to implement.  If you implement it, I can critique it until it is
ready for inclusion.  [Editors that put a BOM on files in UTF-8
are in error, but it is a common error so it should be accepted
gracefully.  And, of course, it is supposed to be there on a UTF-16
or UTF-32 file.]

Note that GCC should not be limited to looking for the Unicode
"byte order mark".  It should recognize and handle all other
reasonable in-band annotations of the file encoding.  Examples are
Emacs' -*- marker in a comment on the first line and (rather more
complicated) "Local Variables:" marker near the end of the file;
other editors have similar, but of course incompatible, conventions
(I know Vim has one but I don't know what it looks like).  It would
also be good to take advantage of the fact that 95+% of C source
files start with "/*", "//", "#i", or "#d" to distinguish ASCII
from EBCDIC.  (This is in fact necessary in order to have any hope
of detecting and processing an editor's code page marker in an EBCDIC
source file.)

You should have read and fully understood the long comment near the top
of libcpp/charset.c, and the sections of the C standard that it refers
to, before you attempt to code this.

It may be necessary to import GNU iconv to the source tree in order to
gain reliable handling of non-Unicode encodings.  This should not be
hard but has to be run by the steering committee.

zw