Encode with ensure_ascii=True and then fix

[richvdh]

This turns out to be way quicker. It also allows us to fix #2.

(includes #7 and #3)

Fixes #2.

[erikjohnston]

I'm assuming the handling of the surrogate pairs are correct. It might be nice to have some tests for those cases too, potentially testing _unascii directly.

[richvdh]

I'm assuming the handling of the surrogate pairs are correct. It might be nice to have some tests for those cases too, potentially testing _unascii directly.

Well, we do test the surrogate pairs, via the encoding of the smiley poo (https://github.com/matrix-org/python-canonicaljson/pull/9/files#diff-a9559bb86637bd1d66d1a37f0b48a732R35).

I feel like this is sufficient?

[erikjohnston]

Ah, my bad, yup that's fine

[richvdh]

I really wish I'd kept a record of why I thought that this was faster than just using ensure_ascii=False. The simplejson bug affecting \u2028 and \u2029 is fixed as of 3.14.0, so I'm wondering if we can just back this out.

(I also don't remember how I benchmarked it)

[richvdh]

ohhh, I found https://github.com/richvdh/json_benchmark.

Benchmarks against simplejson 3.17.0:

(seconds per loop: smaller is better)

                       dumps (large obj)   dumps (small objs)
py2.7, ensure_ascii=True           0.035                0.041
py2.7, ensure_ascii=False          0.066                0.096
py3.6.9, ensure_ascii=True         0.035                0.048
py3.6.9, ensure_ascii=False        0.045                0.068
py3.7.5, ensure_ascii=True         0.029                0.038
py3.7.5, ensure_ascii=False        0.052                0.079

conclusion: this approach is still substantially faster, although the difference is less marked than it used to be under python 2.7.

The main difference (I think) is that simplejson has an optimised implementation of encode_basestring_ascii (which is used with ensure_ascii=True), but not for encode_basestring.

(the simplejson folks have in the past said they'd more than happy to accept new optimisations, so we could certainly send them a PR for an optimised version of encode_basestring.)

[richvdh]

Having said all that, repeating the py3.7.5 benchmarks with stdlib json instead of simplejson:

                                  dumps (large obj)   dumps (small objs)
py3.7.5, stdlib json, ensure_ascii=True       0.016                0.019
py3.7.5, stdlib json, ensure_ascii=False      0.018                0.018

so yeah. #23.