Making charset auto-detection strictly opt-in.

[tomchristie]

So, I've been thinking for a while about making our charset auto-detection strictly opt-in.

Our charset auto-detection is used for cases where response.text is accessed, but no charset is present in the response Content-Type.

Right now we'll fallback to using charset_normalizer in that case in order to auto-detect an encoding. Which is a bit of a mixed bag. It's a bit of a fuzzy approach, and I'm not overly keen on it. We've tried utf-8 only as the fallback in the past, which

So, here's an alternative.

Rather than having an apparent_encoding on the Request class, which is used as the fallback, I'd suggest the following for charset control...

• charset_fallback="utf-8" # Default to utf-8 as the fallback.
• charset_errors="replace" # Default to the lenient "replace" for decoding failures.

We'd have those on the Request() model, and on the Client() model, so for instance...

# If no charset is specified, use `shift_jis`.
client = httpx.Client(charset_fallback="shift_jis")

Or...

# If no charset is specified, use `utf-8`.
# If text decoding fails, raise errors.
client = httpx.Client(charset_fallback="utf-8", charset_errors="strict")

However we would still like to support auto-detection for the fallback, but make it strictly opt-in.

We'd do that by having charset_normalizer as a regular installable codec...

import codecs
import httpx

codecs.register(httpx.chardet_codec)
codecs.register(httpx.charset_normalizer_codec)

# We can now use `"chardet"` or `"charset_normalizer"` as a codec name...
assert b"abc".decode("chardet") == "abc"

Which then allows...

client = httpx.Client(charset_fallback="charset_normalizer")

Or...

client = httpx.Client(charset_fallback="chardet")

Related: #1018, #1269, #1657, #1791

[simonw]

Looks like this recently landed (in May 2022):

• Make charset auto-detection optional. #2165