Problems with garbage characters in Chinese Win7 under VS2008.

Post by David
I have a C++ app which when built with VS2008 displays '?' for
characters on an English Win7 system with Hong Kong language pack but
displays proper characters on a native Hong Kong Win7 system. When
built with VS6, it displays properly on both systems. Any insights
would be appreciated.
I can provide a tiny sample program.

I think you'll need to show a code snippet to illustrate what you're
doing.

Also the microsoft.public.win32.programmer.international might have
been a more appropriate group for your question.

Dave Lowndes

Mihai N.

2010-02-04 09:09:38 UTC

Check the project settings.
IT is most likely that the VS2008 project does not build a Unicode version.
See here: http://www.mihai-nita.net/article.php?artID=20060723a

--
Mihai Nita [Microsoft MVP, Visual C++]
http://www.mihai-nita.net
------------------------------------------
Replace _year_ with _ to get the real email

David

2010-02-04 14:37:17 UTC

Check the project settings.
IT is most likely that the VS2008 project does not build a Unicode version.
See here:http://www.mihai-nita.net/article.php?artID=20060723a
--
Mihai Nita [Microsoft MVP, Visual C++]http://www.mihai-nita.net
------------------------------------------
Replace _year_ with _ to get the real email

Neither the VS6 or VS2008 projects build Unicode versions. This is by
design. The resource file is a Big5-encoded TrChinese string. Looks
fine under both VS6 and 2008 on full Hong Kong Win7, looks okay only
under VS6 on English Win7 with HK language pack. The code is simply
the following:

HINSTANCE hInst = LoadLibrary("resources.dll");
if (hInst != NULL)
{
AfxSetResourceHandle(hInst);
MessageBox("Language File Loaded", "", MB_OK);
}
else
{
MessageBox("Language File NOT Loaded", "ERROR", MB_ICONSTOP);
EndDialog(-1);
return FALSE;
}

// Now read a single string from the resource file and put
// its text onto the text string.
CString textToSet;
#define IDS_ACCEPT_LICENSE 233
textToSet.LoadString(IDS_ACCEPT_LICENSE);
((CStatic *)GetDlgItem(IDC_TEXT))->SetWindowText(textToSet);

Alexander Grigoriev

2010-02-04 14:56:00 UTC

The best advice is: switch to UNICODE. That'll solve many problems.

David Lowndes

2010-02-04 15:18:27 UTC

Post by David
Neither the VS6 or VS2008 projects build Unicode versions. This is by
design. The resource file is a Big5-encoded TrChinese string. Looks
fine under both VS6 and 2008 on full Hong Kong Win7, looks okay only
under VS6 on English Win7 with HK language pack. The code is simply
...

Can you be more specific about what characters you should be getting
and where precisely they're becoming "?" characters. Presumably latin
characters in the string aren't exhibiting the problem?

As Alexander says, a move to a Unicode build is probably the best
solution.

Dave

David

2010-02-04 16:18:37 UTC

Post by David Lowndes

Can you be more specific about what characters you should be getting
and where precisely they're becoming "?" characters. Presumably latin
characters in the string aren't exhibiting the problem?
As Alexander says, a move to a Unicode build is probably the best
solution.
Dave

Moving to Unicode is out of the question. This program has shipped
many thousands of copies and runs on WinXP, Vista and Windows 7. It is
a mature product and has language files for about 15 languages, all
produced in UTF-8 format with the appropriate code pages being called
out. A future major rework would probably be done in Unicode but that
is not in the short-term view.

Thanks for the assistance.

Ivo Beltchev

2010-02-04 17:23:52 UTC

I think you are confusing things here (or maybe I am?). But as far as I
know the resources are ALWAYS built as Unicode, even for ANSI
applications. It doesn't matter what's the encoding of your resource
file. So you can convert your .rc file to Unicode (UTF-8, UTF-16,
whatever) and still have ANSI program.

Also what you are saying "UTF-8 format with appropriate code pages"
makes no sense. If the .rc file is Unicode (UTF-8) then you shouldn't
specify code pages. Code pages are if you have ANSI .rc file. If you
have both bad things can happen. Can that be your problem? Some time ago
(either VS 2003 or VS 2005) they added support for Unicode .rc files.
Maybe VC6 treated the file as ANSI, ignoring the fact it is UTF-8. The
new Visual Studio recognizes the BOM and treats it (correctly) as
Unicode, but later gets confused by the code page directives.

If that's the problem I would advise to convert your .rc file to
Unicode. This will simplify the handling of all languages. You will be
able to see all languages at the same time without having to switch the
code page in your text editor. I want to stress that this won't make
your application Unicode. All you need to do is reformat the .rc file.
No code change is needed.

Ivo

Post by David Lowndes

Can you be more specific about what characters you should be getting
and where precisely they're becoming "?" characters. Presumably latin
characters in the string aren't exhibiting the problem?
As Alexander says, a move to a Unicode build is probably the best
solution.
Dave

David

2010-02-04 21:22:07 UTC

Not to beat a dead horse, but the current application (as shipping)
works on all of the target languages including Win7 Hong Kong edition.
The problem is that it does not work on Win7 English edition with Hong
Kong language pack. It also works on both of those when built with
VS6.

The fundamental question is: what is different between the native Hong
Kong Win7 and Win7 with HK language pack systems? Also, what is
different between the VS6 and VS2008 programs? (The same resource
file, built with VS2008, works on both versions of the code.)

Ivo Beltchev

2010-02-05 04:05:20 UTC

???? usually indicates that some text is being converted from Unicode to
ANSI, and the characters can't be represented in the current code page.
This can happen inside a system call like LoadStringA/DialogBoxA, or in
your user code (I'm lumping the MFC sources in this category).

So try this - call WideCharToMultiByte(CP_ACP) with the same Hong
Kongian text on both systems. This is what functions like LoadStringA
use internally. If you get correct result on one system and wrong on the
other you can remove the resource system as a suspect. If both are
correct, then most likely the failure is in your code or MFC.

Post by David
Not to beat a dead horse, but the current application (as shipping)
works on all of the target languages including Win7 Hong Kong edition.
The problem is that it does not work on Win7 English edition with Hong
Kong language pack. It also works on both of those when built with
VS6.
The fundamental question is: what is different between the native Hong
Kong Win7 and Win7 with HK language pack systems? Also, what is
different between the VS6 and VS2008 programs? (The same resource
file, built with VS2008, works on both versions of the code.)

Mihai N.

2010-02-05 10:20:29 UTC

Post by David
The problem is that it does not work on Win7 English edition with Hong
Kong language pack.

Must be Unicode for that.

Post by David
It also works on both of those when built with VS6.

Sorry, I don't believe that.

Post by David
The fundamental question is: what is different between the native Hong
Kong Win7 and Win7 with HK language pack systems?

Language pack has nothing to do with it.
What matters is the system locale, which determines the system code page.
See here: http://www.mihai-nita.net/article.php?artID=20050611a
Set the system locale (or "Language for non-Unicode Programs") to
Chinese Hong Kong, reboot and it will work.

The other thing that might matter is on what system are you building the
thing (not the compiler version).

If the resources are encoded in big-5 (no UTF-8, the resource compiler
does not support utf-8) then you have to compile them on a Traditional
Chinese system, or have the proper #pragma code_pag

So, to recap:
- How do you compile the resources
- The application is Unicode or not
- If not, then the system code page

As Ivo already explained, ?? means that characters were converted to a code
page that cannot represent them.
That can be when you compile the resources (case 1) or at runtime
(because the app is not Unicode and the system code page is not big-5)

--
Mihai Nita [Microsoft MVP, Visual C++]
http://www.mihai-nita.net
------------------------------------------
Replace _year_ with _ to get the real email

David

2010-02-06 13:58:16 UTC

Post by David
The problem is that it does not work on Win7 English edition with Hong
Kong language pack.

Must be Unicode for that.

Not true. The resources are built with 8-bit ANSI code-page specific
characters. I have solved the problem and gotten this to work
properly.

Post by David
It also works on both of those when built with VS6.

Sorry, I don't believe that.

Why don't you believe that? Why would you think I would lie about how
the program works - would doing so help me get a valid answer?

Post by David
The fundamental question is: what is different between the native Hong
Kong Win7 and Win7 with HK language pack systems?

Language pack has nothing to do with it.
What matters is the system locale, which determines the system code page.
See here:http://www.mihai-nita.net/article.php?artID=20050611a
Set the system locale (or "Language for non-Unicode Programs") to
Chinese Hong Kong, reboot and it will work.
The other thing that might matter is on what system are you building the
thing (not the compiler version).
If the resources are encoded in big-5 (no UTF-8, the resource compiler
does not support utf-8) then you have to compile them on a Traditional
Chinese system, or have the proper #pragma code_pag
- How do you compile the resources
- The application is Unicode or not
- If not, then the system code page
As Ivo already explained, ?? means that characters were converted to a code
page that cannot represent them.
That can be when you compile the resources (case 1) or at runtime
(because the app is not Unicode and the system code page is not big-5)
--
Mihai Nita [Microsoft MVP, Visual C++]http://www.mihai-nita.net
------------------------------------------
Replace _year_ with _ to get the real email

To sum up the solution:
1) The difference between the native Hong Kong Win7 system and an
English system with HK language pack appears to be that the locale
used for non-Unicode programs on the latter system is the English-
language locale, while the HK-native version of course used the HK
locale. Hence, the English-language version doesn't use the 8-bit
characters against the correct code page (950) and doesn't have the
correct characters.
2) VS6 apparently apparently defaults to the USER's default locale
(and code page) as specified in the .rc file while VS2008 doesn't do
so.
3) The solution is to add a call to
SetThreadLocale(LOCALE_USER_DEFAULT) to the program. Then all works
properly.

Thanks to all who attempted to understand and solve this problem.

Mihai N.

2010-02-07 08:29:29 UTC

Post by David
1) The difference between the native Hong Kong Win7 system and an
English system with HK language pack appears to be that the locale
used for non-Unicode programs on the latter system is the English-
language locale, while the HK-native version of course used the HK
locale. Hence, the English-language version doesn't use the 8-bit
characters against the correct code page (950) and doesn't have the
correct characters.

That's what I have explained.

Post by David
2) VS6 apparently apparently defaults to the USER's default locale
(and code page) as specified in the .rc file while VS2008 doesn't do
so.

The user's default locale has nothing to do with this.
(that locale is only used to format dates/times/numeber/etc.)

Post by David
3) The solution is to add a call to
SetThreadLocale(LOCALE_USER_DEFAULT) to the program. Then all works
properly.

That is not a solution. You will have other problems later on, you will see.
And again, LOCALE_USER_DEFAULT is the locale used for formatting,
has nothing to do with the UI locale, or code pages.

You have to set the Locale for non-Unicode programs to Traditional
Chinese in the English system. That is the relevant difference between
the real HK system and the English one.
And has nothing to do with Visual Studio (although it might be something
deep in MFC, if your application is MFC).

--
Mihai Nita [Microsoft MVP, Visual C++]
http://www.mihai-nita.net
------------------------------------------
Replace _year_ with _ to get the real email

Liviu

2010-02-07 17:58:26 UTC

Post by David
The problem is that it does not work on Win7 English edition
with Hong Kong language pack.

[...]
1) The difference between the native Hong Kong Win7 system and an
English system with HK language pack appears to be that the locale
used for non-Unicode programs on the latter system is the English-
language locale, while the HK-native version of course used the HK
locale. Hence, the English-language version doesn't use the 8-bit
characters against the correct code page (950) and doesn't have the
correct characters.
2) VS6 apparently apparently defaults to the USER's default locale
(and code page) as specified in the .rc file while VS2008 doesn't do
so.
3) The solution is to add a call to
SetThreadLocale(LOCALE_USER_DEFAULT) to the program.
Then all works properly.

If the above did indeed fix the problem, then you may have been bit
by http://msdn.microsoft.com/en-us/library/w1sc4t4k.aspx

"In versions of ATL up to and including ATL 3.0 in Visual C++ 6.0,
string conversions using the macros in atlconv.h were always performed
using the ANSI code page of the system (CP_ACP).
Starting with ATL 7.0 in Visual C++ .NET, string conversions are
performed using the default ANSI code page of the current thread"

(and the same applies to _wcstombsz, _mbstowcsz).

However, in that case, there must be more to your code than the
(over)simplified LoadString + SetWindowText you quoted earlier.

Liviu

David

2010-02-09 04:05:51 UTC

Post by Liviu

Post by David
The problem is that it does not work on Win7 English edition
with Hong Kong language pack.

If the above did indeed fix the problem, then you may have been bit
byhttp://msdn.microsoft.com/en-us/library/w1sc4t4k.aspx
"In versions of ATL up to and including ATL 3.0 in Visual C++ 6.0,
string conversions using the macros in atlconv.h were always performed
using the ANSI code page of the system (CP_ACP).
Starting with ATL 7.0 in Visual C++ .NET, string conversions are
performed using the default ANSI code page of the current thread"
(and the same applies to _wcstombsz, _mbstowcsz).
However, in that case, there must be more to your code than the
(over)simplified LoadString + SetWindowText you quoted earlier.
Liviu- Hide quoted text -
- Show quoted text -

As I've said in other posts, this problem has been solved and fixed.

As for the comment "there must be more to your code...", I showed all
of the code in the test program. It was simplified from the production
application but the code provided was sufficient to show the problem.

Liviu

2010-02-09 05:24:30 UTC

Post by David
The problem is that it does not work on Win7 English edition
with Hong Kong language pack.

If the above did indeed fix the problem, then you may have been bit
by http://msdn.microsoft.com/en-us/library/w1sc4t4k.aspx
"In versions of ATL up to and including ATL 3.0 in Visual C++ 6.0,
string conversions using the macros in atlconv.h were always
performed using the ANSI code page of the system (CP_ACP).
Starting with ATL 7.0 in Visual C++ .NET, string conversions are
performed using the default ANSI code page of the current thread"
(and the same applies to _wcstombsz, _mbstowcsz).
However, in that case, there must be more to your code than the
(over)simplified LoadString + SetWindowText you quoted earlier.

As I've said in other posts, this problem has been solved and fixed.

Maybe "fixed" or patched, rather than "solved". I agree with others'
advice that the only actual "solution" is moving to Unicode.

The SetThreadLocale documentation
http://msdn.microsoft.com/en-us/library/dd374051(VS.85).aspx
does mention a number of caveats, to the effect of "don't use it".

Post by David
As for the comment "there must be more to your code...", I showed all
of the code in the test program. It was simplified from the production
application but the code provided was sufficient to show the problem.

I won't really argue this point, since I don't have an HK-native test
machine handy to verify myself. However, I am still curious as to why
your app's locale was other than the user's default one, and needed to
be explicitly forced back to LOCALE_USER_DEFAULT.

Liviu

Mihai N.

2010-02-08 09:32:04 UTC

Post by Mihai N.
Must be Unicode for that.

Not true. The resources are built with 8-bit ANSI code-page specific
characters. I have solved the problem and gotten this to work
properly.

The .rc files get compiled to .res, and the .res files are always
Unicode, it does not matter if the .rc files are Unicode or not.
The code page uset to convert to Unicode is specified by
#pragma code_page or the /c switch in the resource compiler.
(the pragma takes priority)

At runtime the Unicode resources get loaded and used as is
if the application is Unicode, or converted to the ANSI code
page for non-Unicode applications.
Even you the resource loading starts using the thread locale
instead of the system locale, you will get big-5 strings in
an ANSI application running on a system with 1252 system code
page. So using the big-5 strings will lead to bugs down the line.
Loading them from resources is only half of the problem.
So you will see other bugs down the line.

The only way to safely run a Traditional Chinese application is
to set the system code page to 950 (by setting the language
for non-Unicode applications) or by making your application
Unicode.

Any other "solution" is unsupported, and will give you all
kind of problems.

--
Mihai Nita [Microsoft MVP, Visual C++]
http://www.mihai-nita.net
------------------------------------------
Replace _year_ with _ to get the real email

David

2010-02-09 04:12:04 UTC

Post by Mihai N.
Must be Unicode for that.

Not true. The resources are built with 8-bit ANSI code-page specific
characters. I have solved the problem and gotten this to work
properly.

The .rc files specify the code page with directives within the file.

Post by Mihai N.
At runtime the Unicode resources get loaded and used as is
if the application is Unicode, or converted to the ANSI code
page for non-Unicode applications.
Even you the resource loading starts using the thread locale
instead of the system locale, you will get big-5 strings in
an ANSI application running on a system with 1252 system code
page. So using the big-5 strings will lead to bugs down the line.
Loading them from resources is only half of the problem.
So you will see other bugs down the line.

The resource files contain 8 bit characters (not unicode as I haven't
built with that flag set).
I know that big-5 strings are present and require code page 950. The
only time I use the Traditional Chinese version of the strings (from a
DLL I load based on current language), is when the user's specified
language/locale is 0404 or 0c04 (and hence, code page 950 is the
active CP). The problem is that even in that situation, the system is
using the system's native (English) code page. By explicitly setting
the locale (SetThreadLocale(LOCALE_USER_DEFAULT);) the correct code
page (950) is used.

I've built the application and verified that it works properly so
whatever others in this thread may think, the analysis and solution
are valid.

Post by Mihai N.
The only way to safely run a Traditional Chinese application is
to set the system code page to 950 (by setting the language
for non-Unicode applications) or by making your application
Unicode.
Any other "solution" is unsupported, and will give you all
kind of problems.
--
Mihai Nita [Microsoft MVP, Visual C++]http://www.mihai-nita.net
------------------------------------------
Replace _year_ with _ to get the real email

Regards, David

Leslie Milburn

2010-02-09 08:29:19 UTC

Post by David
The resource files contain 8 bit characters (not unicode as I haven't
built with that flag set).

David,

I think you misunderstand this point. Regardless of your #defines and other
settings at compile time, the rc and dlg files *are always* double byte
characters - I know for a fact this is the case for Visual C/C++ 6 as I am
using it. So even if the application is non-unicode the dialog templates are
still stored with double byte characters.

Leslie.

Alexander Grigoriev

2010-02-09 15:34:48 UTC

No. Only the _compiled_ resources are always double-byte (more precisely,
UTF-16). RC files can be single-byte, UTF-8, or MBCS, or UTF-16.

Post by Leslie Milburn

Post by David
The resource files contain 8 bit characters (not unicode as I haven't
built with that flag set).

David,
I think you misunderstand this point. Regardless of your #defines and
other settings at compile time, the rc and dlg files *are always* double
byte characters - I know for a fact this is the case for Visual C/C++ 6 as
I am using it. So even if the application is non-unicode the dialog
templates are still stored with double byte characters.
Leslie.

Leslie Milburn

2010-02-10 03:42:58 UTC

Post by Alexander Grigoriev
No. Only the _compiled_ resources are always double-byte (more precisely,
UTF-16). RC files can be single-byte, UTF-8, or MBCS, or UTF-16.

Come ON! That is just common sense *of course* we are talking about the
delivered binary !!!!! Who cares what the format of the readable source is -
you are just being pedantic

Mihai N.

2010-02-10 08:25:46 UTC