Use charset from Content-Type header

[jaymode]

In #22691 (comment), I added a comment which points out that our code currently ignores the charset parameter of the Content-Type header and that this is something we should look into. Looking at the javadocs of JsonFactory to see how different charsets are handled:

Encoding is auto-detected from contents according to JSON
specification recommended mechanism. Json specification
supports only UTF-8, UTF-16 and UTF-32 as valid encodings,
so auto-detection implemented only for this charsets.
For other charsets use {@link #createParser(java.io.Reader)}.

Unfortunately not all clients adhere to the unicode only encodings as I have seen some send data as ISO-8859-1. I think we should consider parsing the charset from the content-type when available and handling appropriately (failing if we cannot support, convert, create parser differently etc.).

[jaymode]

Discussed in Fix it Friday, the plan forward is to:

1. Add deprecation logging for 5.x with non unicode encoding with a strict mode defaulted to off
2. Enable a strict mode in 6.0 by default and deprecate non-strict mode
3. Strict only for 7.0

Ultimately this plan allows us to provide notice to users, hopefully learn the encodings that are being used other than unicode, and allow users a way to continue working while we decide if there is anything we can do about the other encodings.

[Pyppe]

Not only it ignores the charset, but if we e.g. use header of Content-Type: application/x-ndjson; charset=UTF-8 for the bulk-API we also get a misleading deprecation warning:

[WARN ][o.e.d.r.RestController   ] Content type detection for rest requests is deprecated. Specify the content type using the [Content-Type] header.

It took me a while to realize the warning was caused by the additional ; charset=UTF-8 suffix...

[nik9000]

@Pyppe can you open an issue for the deprecation warning with the ; charset UTF-8 thing? That seems like a thing we should discuss and I think it deserves its own issue.

[pgomulka]

we might want to combine this with #72969
So the idea would be that before the request routing in RestController we would validate the parameters

We already have a way to declare allowed parameters for given media type, but nothing is validated.

elasticsearch/libs/x-content/src/main/java/org/elasticsearch/xcontent/XContentType.java

Line 154 in 20c9f75

Map.of(COMPATIBLE_WITH_PARAMETER_NAME, VERSION_PATTERN)),

[elasticsearchmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[ldematte]

#64406 (which went into v8.0.0) seems to have added the ability to parse a charset, but accepts only utf-8.
Unclear what is still valid and what we want to do next (support utf-16 or other charsets? Error if the charset is not utf-8?), so I'm labelling this for team discuss.