This is the mail archive of the mailing list for the GCC project.

Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]
Other format: [Raw text]

Re: gcc compile-time (multibyte issue)

In a discussion offlist with davem, he claimed that multibyte support would
significantly slow down lexical analysis (and preprocessing in particular)

He claimed that it meant that you would have to have a procedure call per
character. I disagreed, and he suggested I post my thoughts.

First let me give the scenario I have in mind, since perhaps it may be
different from what other people are thinking.

We are faced with source files that contain extended characters represented
in some manner. In the general case we must know the representation being
used, since the same text can mean different things in different 
represaentations (e.g. Shift-JIS and IEC). These representations typically
fall into three patterns:

1. The use of uniform 2 or 4 byte characters. Apart from having to read a bit
more data, this obviously has no impact on performance.

2. The use of escape sequences, which typically comes down to two subcases

   2a. The escape is triggered by the use of an upper half character (this is
       a standard Chinese representation for example)

   2b. The escape is triggered by a specific character as in Shift-JIS.

A compiler needs to support a variety of encoding methods. You often find that
the official standard methods are in fact not the ones used in practice (in
Japan for example, few people use UTF).

What GNAT does (it supports half a dozen different forms in the 2a and 2b
category) is to quickly scan past blanks, and then do a case statement on
the first character (this is the only reasonable way to write a fast lexer
in any case). Then the handling of escape sequences happens only if they are
encountered, and there is no distributed overhead.

Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]