Ambiguous interpretation of -1 among char_traits::eof, char_traits::to_int_type and streambuf::underflow

[Jonathan Wakely]

On 10 February 2014 16:26, Brent Lewis <coder0xff@gmail.com> wrote:
> In certain circumstances, a 0xFF read from a stream buffer may be
> erroneously interpreted as the EOF value.
>
> Example from boost iostreams (
> iostreams\detail\streambuf\indirect_streambuf.hpp ) :
>
> template<typename T, typename Tr, typename Alloc, typename Mode>
> typename indirect_streambuf<T, Tr, Alloc, Mode>::int_type
> indirect_streambuf<T, Tr, Alloc, Mode>::underflow()
> {
>     //... code removed for brevity
>     return chars != 0 ?
>     traits_type::to_int_type(*gptr()) :
>     traits_type::eof();
> }
>
> In the event that the parameter to to_int_type is a signed char (which
> is the case with istream)

No, std::istream is std::basic_istream<char> not
std::basic_istream<signed char>. The difference is significant.

> with the value -1, sign extension will cause
> it to return (int_type) -1. Thusly, when underflow() is called by
>
> std::basic_streambuf<signed char, std::char_traits<signed char> >::uflow()
> {
>     //...
>     int_type __ret = traits_type::eof();
>     const bool __testeof = traits_type::eq_int_type(this->underflow(), __ret);
>     //...
> }
>
> It erroneously interprets this -1 as eof.

It shouldn't do, because underflow() is supposed to use
char_traits<C>::to_int_type(const C&) to disambiguate char(0xff) and
char_traits<C>::eof().

I had to read your mail a couple of times to notice you're using
'signed char' above.

If you're using streams of 'signed char' then you need to use a custom
traits class because std::char_traits<signed char> is not specialized
and doesn't necessarily do the right thing.


> Though not optimal, a possible workaround is to ensure EOF is a value
> that is not ambiguous. In char_traits.h:
>
> template<typename _CharT>
> struct char_traits {
>     //...
>     static _GLIBCXX_CONSTEXPR int_type
>     eof() { //if char_type is signed, to_int_type may return -1, so -1
> is NOT a suitable value for EOF
> static_assert(sizeof(char_type) < sizeof(int_type), "Please choose a
> char_type with a size that is smaller than int_type.");
>         return static_cast<int_type>(std::numeric_limits<int_type>::max());
>     }
> }
>
> and whatever other corresponding changes might need to be made.
> Perhaps instead, changing the EOF define would be enough. Another
> solution might be to make char_traits::to_int_type circumvent sign
> extension, so (char)-1 becomes (int)255, though once again I don't
> know what implications this might have elsewhere.

I don't think we want to change char_traits<_CharT>::eof().

The problem shouldn't exist assuming a correct traits type and if
underflow() correctly uses to_int_type() to ensure that when a
char(0xff) is read it is returned as a value distinct from
char_traits<char>::eof().

> In my case (on Android using boost) 4kb are read for each underflow,
> so the 0xFF has to fall exactly on one of these boundaries for the
> buggy behavior to be occur.

Do you have a testcase reproducing the problem?