Proposal: Support nested structures JSON formatted in access_log

[Pchelolo]

Background
The format_json support was implemented in #2692 but it only allows flat single-level objects as JSON output. As I understand from the discussion on the issue, nesting was left out because there was no clear need for it at the time, but the API was designed to be future proof to support it.

Rational
At Wikimedia, we ingest all the access logs into Kafka, funneling them into Hadoop for analytics purposes. All of our logs have a schema, and unluckily, our schema has nested objects. We would like to have envoy access logs conform to the common schema patterns we have.

There are endless possibilities on how to convert flat JSON structure into a nested one when upon ingestion, but by far the cleanest option is to support nested structures in envoy natively to emit the logs in a correct format right away. In the end, I do not see any downsides in supporting nested objects, so the rational could be best described as "why not?".

Proposal
Support nested JSON in format_json access log.

If we provide the implementation/testing/documentation, would this feature be accepted?
cc @aa-stripe as an author of the original patch

[aa-stripe]

I agree, nested objects are totally reasonable, and if you need that support, we should add it natively.

Imo we should definitely avoid any kind of "flatten when emitted, but re-nest upon ingestion type of solution".

I'd be curious of what your specific use case is - what nesting would you like to introduce?

[Pchelolo]

I'd be curious of what your specific use case is - what nesting would you like to introduce?

Here is an example of the schema we are using for some other access logs, the one we're planning to use for envoy would be very similar. So eventually the config in our case should look somewhat like this

[mattklein123]

+1 this sounds like a good thing to add.

[aa-stripe]

[jnwright]

@Pchelolo Hi! I'm a product designer working at HashiCorp, and I'm researching how we can best show envoy access logs in Consul's UI. I saw your comment above and thought you might be a good person to reach out to. If you aren't interested in them being shown in the UI, though, please feel free to ignore this!

If you are, can you tell me what information from the logs would be most important for you to see in the UI? Also, are these logs something you'd like to see on a per-service basis, or for a whole namespace? Or both?