Re: is the concept of codecvt_t::partial broken ?

[Oncaphillis <oncaphillis at snafu dot de>]

Thanks for the fast reply
On 09/26/2012 12:30 PM, Paolo Carlini wrote:
> Hi,
>
> On 09/26/2012 12:18 PM, Oncaphillis wrote:
> > Hi,
> > I'm writing yet another char encoding translator with the help of 
> > codecvt< >
> > and stumbled across the following problem. When I try to interprese a 
> > byte sequence
> > as ShiftJis I never get the result code codcvt_t::partial even if I 
> > feed a single
> > 0xe0 as the only input which the iconv should interprete as an 
> > incomplete two byte
> > sequence. The same seems to hold true with other multi char encodings 
> > as well
> now I don't have the time to analyze the details of this (and it would 
> be very difficult without a complete self-contained testcase) but in 
> general the current codecvt code works pretty well for us. I would 
> suggest: 1- Have a look to the existing testcases in our testsuite, 
> which all pass, and see if you can learn something from those about 
> your specific case; 2- We may have issues with encoding -1 locales, 
> thus I would at first try >= 0 locales. ja_JP.sjis is -1, right?
 Please find attached code which hopefully compiles cleanly on any system.
It tries to transform a given argument string into a std::wstring assuming
an encoding given by the optinal second arg which defaults to en_US.utf8.
The transcoding is done one input char after the other and the current input
and output is printed out.Since it's a profane utf8->wchar_t transcoding
I'm pretty sure it's me who doesn't understand the concept.

On my system with utf8 locale

./a.out 'Hellö Wörld' en_US.utf8

gives me:

<0x48> [0x0048]  // 'H'
<0x65> [0x0065]  // 'e'
<0x6c> [0x006c]  // 'l'
<0x6c> [0x006c]  // 'l'
<0xc3> []        // start of the german umlaut ö-- here I would expect 
to get a codecvt::partial result
<0xb6> [0x00f6]  // second part of the german umlaut ö
<0x20> [0x0020]  // etc....
<0x57> [0x0057]
<0xc3> []
<0xb6> [0x00f6]
<0x72> [0x0072]
<0x6c> [0x006c]
<0x64> [0x0064]


Thanks
Sebastian

> Thanks,
> Paolo.

#include <string>
#include <vector>
#include <string.h>
#include <errno.h>
#include <locale>
#include <iostream>
#include <stdexcept>
#include <iomanip>

bool  ToWString(const std::string & str,std::wstring & ws,const std::string & rEnc="en_US.utf8")
{
    typedef std::codecvt< wchar_t, char, std::mbstate_t > codecvt_t;
    
    std::locale loc(rEnc.c_str());  
    
    const codecvt_t & fc = std::use_facet<codecvt_t>( loc );
    
    ws.clear();
    codecvt_t::result r=codecvt_t::ok;
    
    std::mbstate_t st=mbstate_t();
    
    const char *ca = str.c_str();
    const char *co = ca+str.length();
    const char *cn = ca;
    
    while(co!=cn)
    {
        static const int l = 10;
        static wchar_t wo[l];
        wchar_t *wn=NULL;
        
        {
            std::string str(ca,1);
            std::cerr << "<";
            for(int i=0;i<str.length();i++)
            {
                std::cerr << "0x" << std::hex << std::setw(2) << std::setfill('0') << (unsigned int)(str[i] & 0xff) << std::dec;
            }
            std::cerr << "> ";
        }
        
        if( (r=fc.in(st,ca,ca+1,cn,wo,wo+l,wn))==codecvt_t::error)
        {
            break;
        }
        
        {
            std::wstring ws(wo,wn-wo);
            std::cerr << "[";
            for(int i=0;i<ws.length();i++)
            {
                std::cerr << "0x" << std::hex << std::setw(4) << std::setfill('0') << (unsigned int)ws[i] << std::dec;
            }
            std::cerr << "]";
        }
        
        if(r==codecvt_t::partial)
        {
            std::cerr << " PARTIAL";
        }
        std::cerr << std::endl;
        ws+=std::wstring(wo,wn-wo);
        ca=cn;
    }
    return (r==codecvt_t::ok || r==codecvt_t::noconv);
}


int main(int argc,char ** argv)
{
    if(argc>1)
    {
        try
        {
            std::wstring ws;
            std::string enc=(argc==3 ? argv[2] : "en_US.utf8");
            std::cerr << ToWString(argv[1],ws,enc) << std::endl;
        }
        catch(std::exception & ex)
        {
            std::cerr << "caught:" << ex.what() << std::endl;
        }
    }
}