Re: Input charsets - What's going on?

[Zack Weinberg <zack at codesourcery dot com>]

Paolo Bonzini <bonzini@gnu.org> writes:

> cppcharset.c's _cpp_default_encoding use nl_langinfo (CHARSET) to find
> the default input charset.  This usage is guarded by a configure
> script symbol, HAVE_LANGINFO_CODESET.  Too bad, in the current GCC
> sources, HAVE_LANGINFO_CODESET is never defined and is not even in the
> auto-host.h template.  This means that GCC never attempts any input
> charset conversion except if -finput-charset is given.

Oops.  I had thought that was there, because Java uses the same
construct...

> While moving libcpp to the toplevel, I put the test correctly in the
> libcpp configure script so that nl_langinfo (CODESET) is now used to
> find out the default input charset.  Problem is, this fails
> miserably. fold-const.c and java/typeck.c had two non-breaking spaces
> (Unicode 160) which I have already committed fixes for, and
> libgfortran's files have Tobi's name in it which sports an umlaut-u
> character (Unicode 252): both of these break the default codeset on my
> machine, which is ANSI_X3.4-1968, and cause the bootstrap to error
> because a conversion failure.
>
> Is the feature broken by design?

The feature is indeed broken by design (for precisely this reason);
however, it is in SUS so we're stuck with it.

> How should I proceed in the standalone libcpp?

Disable the feature for now - change #if HAVE_LANGINFO_CODESET to #if 0,
with a comment explaining the problem.

If you feel ambitious, you can look into recognizing in-band markers
of file encoding, which was my plan for dealing with the above
headache.  There are several different possible markers:

- Unicode byte-order mark (FE FF) in any of UCS(2,4)(LE,BE) and UTF8

- to distinguish ASCII-superset encodings from EBCDIC, the following
  heuristic: 90+% of C family source files have "#i", "#d", "/*", or
  "//" as their first two characters.  Have a table with the ASCII and
  EBCDIC values of those patterns.  That will at least get one as far
  as being able to look for a more specific tag.

- An explicit GCC-specific marker, #pragma GCC encoding <xyz> on the
  first line.

- Common conventions for marking this in-band and visibly: the Emacs
  and VIM mode line tags (the VIM convention puts it on the last line,
  which may be a problem - ditto if we want to support the more
  elaborate "Local variables:" Emacs convention)

- There may be another common convention over in Java land, which
  would be friendly to support.

I'll get back to you on your cpplib patch later today.  Thanks for
doing this!

zw