Re: __enc_traits and Unicode strings and related stuff

[Benjamin Kosnik <bkoz at redhat dot com>]

> Good guess about the Unicode.

figured -- standard libraries  really need a solution for this.

;)

> And yes, I've studied your __enc_traits stuff pretty closely. I like your
> approach, but because your implementation is specific to the GNU libstdc++,
> and because my employer is in the business of building cross-platform
> products (not to mention a stdlib), I needed to recreate a version that
> would sit on top of any compliant stdlib implementation.

Interesting. Looks like we've both come to a similar conclusion.

I'd thought about using libiconv for non-GNU systems but the need for 
that hasn't materialized yet.

> One of the main differences between your solution and mine is that I've been
> building my stuff on top of the IBM ICU library instead of iconv. I too use
> my own class as the _stateT argument, and instances of this class are used
> to specify what type of conversion is to be performed.

At some point I'd like to have an --enable-clocale=ICU option. That would 
be useful for many other things, not just codecvt, as I'm sure you've 
already noticed. It would mean that aspects of named locales could be 
implemented well on non-GNU systems, for instance.

That's something that I'd love to work on but again, the need/funding 
isn't really there. (Besides, I have to finish the proof-of-concept with 
glibc first anyway.)

> Many of my classes are actually specializations of existing stdlib templates
> that I place in the std namespace. This is explicitly allowed by the
> standard so as long as your full or partial specialization references types
> that are defined outside of the std namespace (that way your specialization
> can't conflict with ones allowed by the base library).

Yeah. I haven't been especially good about namespace issues.

Are you putting your __enc_traits-ish class (_StateT partial 
specialization type) in a namespace like ext:: ? Or are you doing a 
vendor-specific namespace? 

> The biggest problem that I ran into is that the design specified by the
> standard assumes that the type of conversion (and therefore the source and
> destination encoding) can be controlled by the types used as the template
> arguments. While codecvt_byname can be used to identify a specific external
> encoding (using the [lang[_terr[.encoding]]] locale id), that information is
> only known within the codecvt instance. This presents a problem if the
> actual conversion is controlled by the state object passed in via the in(),
> out(), and length() methods (as is the case in my implementation).

Hmm. Yeah, codecvt_byname doesn't really fit in, as there is no "unicode" 
or "UCS2" locale in POSIX/C99/Single Unix. Not to mention 
encoding-specific locale  names, although you are correct, some locales 
do have a preferred encoding... Doesn't really mesh well with the codecvt interface. Thus the 
need for a specialized state type.

(This is all noted in the issues section of the link I posted earlier.)

> To work around this problem, I had to make my codecvt specialization act as
> an implicit state object factory. Whenever a state object is passed in, the
> codecvt instance will check to make sure that the state object is
> representative of the type of conversion that is to be performed.

Thanks for sharing. 

I'll post my slides from LWE/NY which touched a bit on this.

There's got to be a better way to do a unicode string <-> string 
conversion than encapsulating a basic_string<unicode_type> in a new 
unicode_string class and making conversion constructors/assignment 
operators. That's a lot of code duplication.

-benjamin