[Reporting] Check Elasticsearch http.max_content_length against Kibana xpack.reporting.csv.maxSizeBytes

[tsullivan]

If xpack.reporting.csv.maxSizeBytes (default 10mb) in kibana.yml is greater than http.max_content_length in Elasticsearch (default 100mb), that would be an invalid stack configuration.

We can probably detect this automatically, and log a warning if needed, by checking the max_content_length in ES with: http://localhost:9200/_cluster/settings?include_defaults=true but first let's verify that changing the value in elasticsearch.yml will reflect in the API response. I'm not sure what happens if that value is set differently across multiple ES nodes.

[elasticmachine]

Pinging @elastic/kibana-app

[joelgriffith]

Taking a look now, whilst #26307 is landing

[joelgriffith]

Initial pass has landed here: #26482. Regarding:

let's verify that changing the value in elasticsearch.yml will reflect in the API response. I'm not sure what happens if that value is set differently across multiple ES nodes.

The answer here is that changing the value is reflected in the cluster API's response. However, different nodes can have differing settings for this value. So there isn't a bullet-proof way to guarantee that each node will react similarly. Also, we aren't checking if a node gets restarted with a different value, which can break any safeguards we build in on start-up.

Another alternative here is doing a pre-flight check on save to see if the node reporting persists to will be able to handle the body-size. I'm not sure how load-balancing happens in ES when there's multiples nodes in a cluster... so even that check can hit a different node than the actual document persistance call does.

[tsullivan]

It's not only this ES setting that can potentially break functionality, but a reverse-proxy in front of ES. Therefore it doesn't pay off to get too into safety checking for them. I think the best we can aim to do is 2 things:

1. Make an effort to catch config problems early, so at Kibana start time we can call the cluster settings API and log a warning that we found it's possible things might not work: Reporting can generate a report that is larger in bytes than what ES will accept. Include in the message the warning will go away if they change their ES setting.
2. If storing a generated report fails for any reason, we should log as much as we can in the server error logs showing the response we got. If it's some kind of failure, it should be a 4xx and it will be self-explanatory. Ideally the error would also be preserved in the job history data, the .reporting* indices. I think that means set the status to failure and the job "output" as the error message. [*]

We can also document this as a problem scenario in the troubleshooting guide, and the documentation could explain how to look in the ES settings and logs (we'll have to check if ES would actually log it) to see if it's an ES problem, or a problem with a reverse-proxy.

[*] preserving the error messages with that field feels wrong to me, but that's a different conversation

[joelgriffith]

#26482 closed!