[Filebeat][HTTP Endpoint] Add a functionality to decode gzip content and process the ndjson data

[darshan-elastic]

This is a request for the following feature to be added to the http_endpoint input.

1. Decode the gzip content when the Content-Encoding header presents with gzip value.
2. Ability to process the ndjson data.

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[andrewkroh]

Regarding gzip, what I would expect is that incoming POST requests that contain Content-Encoding: gzip have the request bodies automatically decoded.

And similarly for requests with Content-Type: application/ndjson or Content-Type: application/x-ndjson that each JSON document generates a separate event.

[andrewkroh]

The two features are essentially independent and it would be easier to review if the changes were contributed as two separate PRs.

[jamiehynds]

@andrewkroh after discussing with the field team and Zscaler, we have a few users keen to get CloudNSS up and running, but we're blocked by the gzip support. Do you think someone could work on this sooner rather than later?

[jamiehynds]

Cloudflare Logpush is one of the integration we have planned, and will leverage the http_endpoint input. Looks like like they require gzip support for their events too: https://developers.cloudflare.com/logs/about/

[andrewkroh]

Do you think someone could work on this sooner rather than later?

Yes. Here's a PR to add it. #31061

No changes were required to support NDJSON. That already works today with Content-Type: application/json. It doesn't matter if they are newline separated or not. It just consumes all the JSON objects that it reads and creates one event for each.

[efd6]

I was thinking about handling of ndjson and it is more fault tolerant than using the encoding/json decoder because a corrupted message will not break the entire stream following the corrupted message, while in the case of the stream decode, it will (and depending on the approach taken to error handling may lose all messages for one corrupted message). To make use of this observation, when working on ndjson you first need to split the input stream on newlines and then work on each message individually, potentially filling the event with a message.error and event.original when the unmarshalling fails.