Imbuing streams to auto-codecvt [was Re: _M_underflow_common missing]

[Brad Spencer <spencer at infointeractive dot com>]

On Fri, Aug 16, 2002 at 10:11:12AM -0700, Benjamin Kosnik wrote:
> 
> Yo.
> 
> std::basic_filebuf<unsigned char, std::char_traits<unsigned
> char>>::_M_underflow_common(bool)

I came across this same problem from a different angle while trying to
write code that did implicit ShiftJIS-to-UCS4 conversion as part of an
ifstream.  I have no trouble reading 'char' directly and throwing it
through codecvt::in(), but that's way less cool than just imbuing the
stream with a locale that knows how to do it on the fly (i.e. properly).

> Notice that this is not a required instantiation
> (basic_filebuf<unsigned char>). It's required to compile, but not
> necessarily link.

Hehehe.. ok :)

> There is a list of member functions, locale facets, etc that must be
> provided for fully-formatted IO to work with non-required types. I'll
> add documenting this to the TODO list.

Does this include private, implementation-specific member functions of
classes?  Perhaps I misunderstand what you mean here.

> In the meantime, I suggest you hack up a version of
> _M_underflow_common (found in src/fstream.cc) for unsigned char and
> link it in...

I'm trying to do that as well, as chance happens, but I'm not meeting
with a lot of success so far.  I've been re-re-reading Langer and
Kreft, and digging through all the testsuite examples and
documentation I can find, but nobody ever actually tries to do what
I'm doing ;) 

Can someone point me towards the right way to make
something along these lines "work"?  
I've pasted a summary of my "best effort" so far below.  Where am I
going wrong?  I've tried using ext/enc_filebuf, but it just lets me
get the linking problems quicker and easier, as I expected :)  

The best I've got so far is a bad_cast being thrown or in always being
the empty string and then hitting end of file.  It can't be this hard.

// Fix the __enc_traits to a partiular pair of codesets.  Benjamin's
// alternative method in enc_filebuf seems to be to do this at filebuf
// construction time, but that seems harder to use to me.
class ucs4_shiftjis_state : public std::__enc_traits
{
  typedef std::__enc_traits BaseClass;
public:
  ucs4_shiftjis_state()
    : BaseClass("UCS-4LE", "SHIFT_JIS")
  {
    // Finish initialization
    BaseClass::_M_init();
  }
};

// We need our own traits type for basic_ifstream et al
struct ucs4_shiftjis_traits : public std::char_traits<wchar_t>
{
  typedef ucs4_shiftjis_state state_type;
};

// We need to do this, too, or all of the do_* members are not
// linkable.
class my_codecvt : public std::codecvt<wchar_t, char,
std::__enc_traits>
{
  typedef std::codecvt<wchar_t, char, std::__enc_traits> BaseClass;

public:
  // Nothing special
  explicit 
  my_codecvt(size_t __refs = 0)
    : BaseClass(__refs)
  {}
};

// What are we using?
typedef ucs4_shiftjis_traits my_char_traits;

// Exact copy of fstream.cc's wchar_t _M_underflow_common with a new name
template<> inline
basic_filebuf<wchar_t, my_char_traits>::int_type 
basic_filebuf<wchar_t, my_char_traits>::_M_underflow_common(bool)
{
  // ...
}

int
main()
{
  // Make a locale for that  
  std::locale loc(std::locale::classic(), new my_codecvt);

  std::basic_ifstream<wchar_t, my_char_traits> file("somefile");
  file.rdbuf()->pubimbue(loc);

  std::basic_string<wchar_t, my_char_traits> in;
  file >> in;
}

-- 
-----------------------------------------------------------------
Brad Spencer - spencer@infointeractive.com - "It's quite nice..."
  Systems Architect | InfoInterActive Corp. | An AOL Company