JSON fixes

[CyberShadow]

Please see the individual commits for details.

[dlang-bot]

Thanks for your pull request, @CyberShadow!

Bugzilla references

Fix	Bugzilla	Description
✓	5904	std.json parseString doesn't handle chars outside the BMP
✓	17553	std.json should not do UTF decoding when encoding JSON
✓	17555	[REG2.070.0] Control characters in JSON data are invalid and should cause an exception
✓	17556	std.json encodes non-BMP characters incorrectly with JSONOptions.escapeNonAsciiChars
✓	17557	std.json should not do UTF decoding when parsing

[CyberShadow]

@wilzbach Is CircleCI broken on stable?

[wilzbach]

Is CircleCI broken on stable?

Looks like it:

make: *** No rule to make target `../generated/linux/release/64/VERSION', needed by `../generated/linux/release/64/dmd'.  Stop.
make: *** Waiting for unfinished jobs....

[aG0aep6G]

Last commit (fix for issue 17557) looks bad. Rest looks good.

[aG0aep6G]

Could specify the type of the expected exception.

[CyberShadow]

Done.

[aG0aep6G]

This breaks stuff:

parseJSON("\"\U0001D11E\"");
    /* std.json.JSONException@std/json.d(1374): Illegal control character. (Line 1:3) */

The thing is that the rest of the code assumes dchar. While char converts to dchar, it doesn't always keep the same meaning. So stuff gets misinterpreted.

In this specific case, isControl is given a char that becomes a control character when converted to dchar. So it rings the alarm even though the encoded code point in the string is not a control character. Could also go the other way: When a control character is encoded with chars that don't look like control characters, it slips through (haven't checked if this can actually happen).

[CyberShadow]

Good catch, fixed.

[CyberShadow]

Could also go the other way: When a control character is encoded with chars that don't look like control characters, it slips through (haven't checked if this can actually happen).

Hmm, it's one thing to force decoding a D string just to avoid putting Unicode control characters in the JSON text (as that would result in non-conforming JSON)... but to force decoding JSON just so we detect and throw on Unicode control characters?

[CyberShadow]

Done.

[CyberShadow]

Done.

Said done thing (posted it on the wrong PR):

---

OK, well, I checked the latest JSON RFC (7159) and it explicitly states that we don't care about the Unicode control characters:

and the control characters (U+0000 through U+001F).

so, I'm going to amend this and look into disabling auto-decoding for encoding JSON as well.

[aG0aep6G]

The isControl thing is fixed. As far as I see, there are no other such issues. So, LGTM.

But it's rather fragile. Future code might wrongly assume that it's always dealing with dchars.

[aG0aep6G]

I'd like to assert here that escapeNonAsciiChars is not set together with Char = char. If that could happen, we'd output a \u sequence for each char in a multibyte sequence (which would be completely wrong).

[CyberShadow]

Done. Also discovered and fixed another issue (see last commit).

[wilzbach]

Hmm, then why is #5511 failing?

Bad karma? No it's due to dlang/dmd#6935

Fix: dlang/dmd#6941