Poor error handling in Elasticsearch exporter of incorrect response parsing

[npepinpe]

Describe the bug

If you configure Elastic behind a reverse proxy, or in any way which would cause your request to be redirected, you may some time send a request in the exporter (whether exporting a bulk, or putting an index template) and receive HTML.

This occurred in a support case.

If you see such an error when parsing a response, e.g.

Caused by: com.fasterxml.jackson.core.JsonParseException: Unexpected character ('<' (code 60)): expected a valid value (JSON String, Number, Array, Object or token 'null', 'true' or 'false')

This is highly likely because the server returned HTML (or possible XML, but less likely).

In this case, as there was no request error, it means the server returned something between 200 and 399. The root cause is not yet clear.

To Reproduce

Still unsure, I will try to get this from the support case. However, I can say that it is connecting Zeebe to an external, self-managed Elasticsearch (so not deployed via the Helm charts). The version however should be compatible, 7.13.x.

Expected behavior

If it happens to be a redirect, we should handle redirects properly. If the response is truly invalid JSON, then we should print out the raw response in the exception so we can diagnose what's happening.

Log/Stacktrace

Full Stacktrace

2022-11-08 06:16:52.573 [Broker-0-Exporter-1] [Broker-0-zb-fs-workers-0] WARN 
      io.camunda.zeebe.broker.exporter.elasticsearch - Error on exporting record with key 2251799813685249
io.camunda.zeebe.exporter.ElasticsearchExporterException: Failed to put component template
	at io.camunda.zeebe.exporter.ElasticsearchClient.putComponentTemplate(ElasticsearchClient.java:215) ~[zeebe-elasticsearch-exporter-8.1.2.jar:8.1.2]
	at io.camunda.zeebe.exporter.ElasticsearchClient.putComponentTemplate(ElasticsearchClient.java:145) ~[zeebe-elasticsearch-exporter-8.1.2.jar:8.1.2]
	at io.camunda.zeebe.exporter.ElasticsearchExporter.createComponentTemplate(ElasticsearchExporter.java:209) ~[zeebe-elasticsearch-exporter-8.1.2.jar:8.1.2]
	at io.camunda.zeebe.exporter.ElasticsearchExporter.createIndexTemplates(ElasticsearchExporter.java:147) ~[zeebe-elasticsearch-exporter-8.1.2.jar:8.1.2]
	at io.camunda.zeebe.exporter.ElasticsearchExporter.export(ElasticsearchExporter.java:79) ~[zeebe-elasticsearch-exporter-8.1.2.jar:8.1.2]
	at io.camunda.zeebe.broker.exporter.stream.ExporterContainer.lambda$export$3(ExporterContainer.java:156) ~[zeebe-broker-8.1.2.jar:8.1.2]
	at io.camunda.zeebe.util.jar.ThreadContextUtil.runCheckedWithClassLoader(ThreadContextUtil.java:58) ~[zeebe-util-8.1.2.jar:8.1.2]
	at io.camunda.zeebe.util.jar.ThreadContextUtil.runWithClassLoader(ThreadContextUtil.java:34) ~[zeebe-util-8.1.2.jar:8.1.2]
	at io.camunda.zeebe.broker.exporter.stream.ExporterContainer.export(ExporterContainer.java:155) ~[zeebe-broker-8.1.2.jar:8.1.2]
	at io.camunda.zeebe.broker.exporter.stream.ExporterContainer.exportRecord(ExporterContainer.java:142) ~[zeebe-broker-8.1.2.jar:8.1.2]
	at io.camunda.zeebe.broker.exporter.stream.ExporterDirector$RecordExporter.export(ExporterDirector.java:523) ~[zeebe-broker-8.1.2.jar:8.1.2]
	at io.camunda.zeebe.scheduler.retry.BackOffRetryStrategy.run(BackOffRetryStrategy.java:51) ~[zeebe-scheduler-8.1.2.jar:8.1.2]
	at io.camunda.zeebe.scheduler.ActorJob.invoke(ActorJob.java:94) ~[zeebe-scheduler-8.1.2.jar:8.1.2]
	at io.camunda.zeebe.scheduler.ActorJob.execute(ActorJob.java:45) ~[zeebe-scheduler-8.1.2.jar:8.1.2]
	at io.camunda.zeebe.scheduler.ActorTask.execute(ActorTask.java:119) ~[zeebe-scheduler-8.1.2.jar:8.1.2]
	at io.camunda.zeebe.scheduler.ActorThread.executeCurrentTask(ActorThread.java:106) ~[zeebe-scheduler-8.1.2.jar:8.1.2]
	at io.camunda.zeebe.scheduler.ActorThread.doWork(ActorThread.java:87) ~[zeebe-scheduler-8.1.2.jar:8.1.2]
	at io.camunda.zeebe.scheduler.ActorThread.run(ActorThread.java:198) ~[zeebe-scheduler-8.1.2.jar:8.1.2]
Caused by: com.fasterxml.jackson.core.JsonParseException: Unexpected character ('<' (code 60)): expected a valid value (JSON String, Number, Array, Object or token 'null', 'true' or 'false')
 at [Source: (org.apache.http.nio.entity.ContentInputStream); line: 1, column: 2]
	at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:2391) ~[jackson-core-2.13.4.jar:2.13.4]
	at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:735) ~[jackson-core-2.13.4.jar:2.13.4]
	at com.fasterxml.jackson.core.base.ParserMinimalBase._reportUnexpectedChar(ParserMinimalBase.java:659) ~[jackson-core-2.13.4.jar:2.13.4]
	at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._handleUnexpectedValue(UTF8StreamJsonParser.java:2737) ~[jackson-core-2.13.4.jar:2.13.4]
	at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._nextTokenNotInObject(UTF8StreamJsonParser.java:902) ~[jackson-core-2.13.4.jar:2.13.4]

Environment:

• OS: Ubuntu Focal (from image)
• Zeebe Version: 8.1.2
• Configuration: self-managed, external Elasticsearch (self deployed)

[npepinpe]

My proposal would be to read in the whole body completely, then parse it. When you do so, Jackson will print out the raw message in the exception, something it cannot do if it's parsing the stream. It's obviously less efficient as we have to copy the whole body in memory, but I think it's acceptable for this case, as it's definitely not the hot path.

[npepinpe]

I did find the root cause, for the record: the configured URL was pointing to a domain, but the user had omitted the port, and as such it was accessing "https://mydomain.com/" instead of "https://mydomain.com:9200", resulting in receiving whatever was returned by that. That said, having the actual HTML in the error message would have made things much faster.