Unicode issue: Not able to serialize properly some character like "式，进"

[afrozm]

When I build a json object containing UTF8 encoded value of string "式，进"
serialize this json object to string
parse the string again into another json object and read the value
Result: string is lost
here is sample pseudo code

Json::Value json;
json["abc"] = UnicodeToUTF8(L"式，进"); // wrapper over MultiByteToWideChar API
// serialize json to string
std::string jsonString = json.toStyledString();
// now parse again
Json::Value json2 = parse(jsonString); // parse using Json::CharReader
std::wstring unicodeString = UTF8ToUnicode(json2["abc"]); // wrapper over MultiByteToWideChar API
assert(unicodeString == L"式，进"); // fails

I checked UnicodeToUTF8 and UTF8ToUnicode works fine.
The only problem I see in toStyledString() API which make character encoding weird in output

[cdunn2001]

Well, try to debug it. What does the result of toStyledString() look like? I think it should be "\u5f0f\uff0c\u8fdb".

If you are able to fix the bug and submit a PR, we'd be happy to merge it.

[BillyDonahue]

This is an incomplete repro and it's hard to help you.

"I checked UnicodeToUTF8 and UTF8ToUnicode works fine." How do you know that? What do they produce? Why not include them? Repro should start and end with the utf8, so we aren't relying on functions we don't have access to.

You also just said that your assert fails, but not how. That doesn't really help.

Where's the body of 'parse'?

We need exact stimulus and response involving only the jsoncpp library if possible.
There are too many moving parts in your example.

I tried to reproduce this and got the styledString:

styled: {
	"abc" : "\u5f0f\uff0c\u8fdb"
}

If I specify the unicode directly in the C++, with either:

const std::string uni = "\xe5\xbc\x8f" "\xef\xbc\x8c" "\xe8\xbf\x9b";
const std::string uni = u8"式，进";

The problem MUST be in the UnicodeToUTF8 and UTF8ToUnicode helpers that were omitted from the bug report. I think we have to close this. I am adding a test that I believe shows this not to be a problem, please modify the test if you review it and find that I've missed something.

[SuperBlc]

In the main.cpp, about Line1651
I change

const std::string uni =  u8"式，进"; // "\u5f0f\uff0c\u8fdb"

to

const std::string uni = "\u5f0f\u77db\u839e"; // "式茅莞"

and it works.