Re: Localization: example from document page doesn't work for latin characters ?

[Jan Pfeifer <pfjan at yahoo dot com dot br>]

Silly of me, if the input is UTF-8, then latin characters are multibyte,
and tolower/toupper will never have the chance to see the whole
character (it only sees one byte at a time).

I guess this is something worth mentioning in the documentation page:

http://gcc.gnu.org/onlinedocs/libstdc++/22_locale/howto.html#7


So I converted the whole program to wchar_t/wstring/wcout. But I get
garbage on the output. Any ideas why ? I suspect that wcout is not
converting the wchar_t* to the terminal's utf-8 encoding. But I don't
know how the dynamics are there.


$ locale
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

$ g++ -o test test.cc && ./test
C++ version:
?rf?o
?RF?O
?rf?o

C version:
?rf?o
?RF?O
?rf?o

The test.cc is attached.


thx!!

- jan




Paolo Carlini wrote:

>Jan Pfeifer wrote:
>
>  
>
>>Thanks for the answer :). I tried the C version as well, and it displays
>>the same problem. I didn't know that the stdc++ used the libc
>>implementation for these localization issues.
>>
>>    
>>
>Yes, I suspected that... ;)
>
>If you have available the sources of the C library (for example, glibc),
>you may want to look in the directory localedata, where all the locales
>are described in "plain" ascii files. There are, for example, tables,
>describing what is considered an uppercase char and what not and
>references to the Unicode standard. I think for some reason those chars
>are sometimes not considered uppercase / lowercase, respectively, but at
>the moment I don't know more, sorry.
>
>I'll follow the replies you will get ;)
>
>Paolo.
>  
>

#include <iostream>
#include <iterator>    // for back_inserter
#include <locale>
#include <string>
#include <algorithm>
#include <cctype>      // old <ctype.h>

using namespace std;

struct MyToUpper
{
	MyToUpper(std::locale const& l) : loc(l) {;}
	wchar_t operator() (wchar_t c) const  { return std::toupper(c,loc); }
	private:
		std::locale const& loc;
};
   
struct MyToLower
{
	MyToLower(std::locale const& l) : loc(l) {;}
	wchar_t operator() (wchar_t c) const  { return std::tolower(c,loc); }
	private:
		std::locale const& loc;
};


int main(int argc, wchar_t** argv)
{
	// locale loc( "" );
	// locale loc( "en_US.UTF-8" );
	locale loc( "pt_BR.UTF-8" );
	MyToUpper up( loc );
	MyToLower down( loc );

	wcout << L"C++ version: " << endl;
	const wchar_t *reference = L"�rfão";
	wstring normal = reference; 
	wcout << normal << endl;
	transform( normal.begin(), normal.end(), normal.begin(), up );
	wcout << normal << endl;
	transform( normal.begin(), normal.end(), normal.begin(), down );
	wcout << normal << endl;


	// C version
	//setlocale(LC_ALL, "");
	//setlocale(LC_ALL, "pt_BR.UTF-8");
	setlocale(LC_ALL, "en_US.UTF-8");
	wcout << endl << L"C version: " << endl;
	wchar_t buffer[256];
	wcscpy( buffer, reference );
	wchar_t *buffer_end = buffer + wcslen(buffer);
	wcout << buffer << endl;
	transform( buffer, buffer_end, buffer, ::toupper );
	wcout << buffer << endl;
	transform( buffer, buffer_end, buffer, ::tolower );
	wcout << buffer << endl;

   return 0;
}