Ensure JSON representation is compact.

[BERRADA-Omar]

Discussed in #3204

^{Originally posted by alleml May 17, 2024}
httpx.Request blindly dumps provided json to ascii characters by invoking json.dumps without any parameters. Currently there is no way to provide ensure_ascii=False argument to the Request instance (or even better to the client).

It leads to the following scenario:

>>> httpx.Request(method="POST", url="http://localhost:8000/test", json={"field": "cześć"}).content
b'{"field": "cze\\u015b\\u0107"}'

when I need properly encoded unicode bytes: b"cze\xc5\x9b\xc4\x87"

[BERRADA-Omar]

This bug have a huge impact on the openai package (1.52.0).
Openai package uses httpx to make it requests, and it turns out that when we make call in French, the GPT models receives text in the wrong encoding. The encoding should be UTF-8 !

For example :

• input text in French : "salut ça va ? très bien ou pas bien ?"
• GPT receives in payload : "salut \u00e7a va ? tr\u00e8s bien ou pas bien ?"

This could have huge impact on text understanding by LLMs, and also could generate more costs for intesive llm users.

Here's a comparison in the official openai tokenizer to prove the point :
UTF-8 correct encoding ==> 11 tokens for GPT-4o

ASCII encoding ==> 21 tokens !!!

Please have this issue fixed.

[BERRADA-Omar]

For those who are looking for a workaround, you can override the method that causes the issue in the entry point of your app (main.py for example).

The patch would be :

import httpx._content
from httpx import ByteStream
from json import dumps as json_dumps

def custom_httpx_encode_json(json: Any) -> tuple[dict[str, str], ByteStream]:
    # disable ascii for json_dumps 
    body = json_dumps(json, ensure_ascii=False).encode("utf-8") # <------------------------------------------------
    content_length = str(len(body))
    content_type = "application/json"
    headers = {"Content-Length": content_length, "Content-Type": content_type}
    return headers, ByteStream(body)

# fix encoding utf-8 bug
httpx._content.encode_json = custom_httpx_encode_json

[tomchristie]

Openai package uses httpx to make it requests, and it turns out that when we make call in French, the GPT models receives text in the wrong encoding.

We're compliant with the JSON spec here, and parsers will be decoding text escape sequences into the correct unicode text. Tho you are correct that a more compact format would be sensible here. Would you like to submit a pull request resolving this?

The following related issues could all be resolved together...

• We should use ensure_ascii=False for more compact text representations.
• We should use separators = (',', ':') for more compact list and object representations.
• We should use allow_nan=False to disallow invalid Infinity and NaN representations.

Useful review points are the UNICODE_JSON, COMPACT_JSON and STRICT_JSON settings in REST framework where we previously worked through these design issues.

[BERRADA-Omar]

Openai package uses httpx to make it requests, and it turns out that when we make call in French, the GPT models receives text in the wrong encoding.

We're compliant with the JSON spec here, and parsers will be decoding text escape sequences into the correct unicode text. Tho you are correct that a more compact format would be sensible here. Would you like to submit a pull request resolving this?

The following related issues could all be resolved together...

• We should use ensure_ascii=False for more compact text representations.
• We should use separators = (',', ':') for more compact list and object representations.
• We should use allow_nan=False to disallow invalid Infinity and NaN representations.

Useful review points are the UNICODE_JSON, COMPACT_JSON and STRICT_JSON settings in REST framework where we previously worked through these design issues.

Thank you for your reply.

I will submit a pull request to resolve the issue.
Regards

[tomchristie]

Thanks @BERRADA-Omar