Time series database TSDB causing Logstash to crash

[mbudge]

Elasticsearch Version

8.11.4

Installed Plugins

No response

Java Version

bundled

OS Version

Na

Problem Description

Hi

If something goes wrong like a Logstash server goes down and it goes un-noticed (the Logstash server is still running, it’s just hanging), and the Fleet managed metrics TSDB index rolls over to Frozen (read only) after 20 days a downstream Logstash server can get stuck trying to write to the Frozen index. Elastic returns a 403 Forbidden which Logstash retries indefinitely until the Elasticagent Logstash pipeline gets stuck which leads to data loss. We work in a regulated environment so can’t loose data due to TSDS, which has happened recently and I need to provide screenshots to auditors which show how long we retain data, however there’s a gap because this happened again.

elastic/logstash#15832

Can TSDB return a 400 status code to prevent Logstash getting stuck OR talk to the Logstash team about a Max retries setting or way to stop retrying 403 Forbidden responses.

We don’t want to use DLQ as it means other status codes aren’t logged in the logstash log. We use these logs to fix other issues.

thanks

Steps to Reproduce

Write to a TSDB in frozen

Logs (if relevant)

Na

[DaveCTurner]

Hi @mbudge,

Can TSDB return a 400 status code to prevent Logstash getting stuck OR talk to the Logstash team about a Max retries setting or way to stop retrying 403 Forbidden responses.

Changing the response status code in this situation would count as a breaking change so I'm pretty sure we won't be doing that. We'll use elastic/logstash#15832 and the associated support case to work out what needs to change in Logstash to avoid this situation. If we identify any changes on the Elasticsearch side needed to support that work then we'll open issues or PRs here, but for now it'd be best to avoid fragmenting the discussion any further. I'm going to close this as there's no specific action for the Elasticsearch dev team to take yet.

[mbudge]

A better option might be to add a step in the metrics ingest pipelines to drop late metrics data.

Drop metrics data which is arriving 24 hours or 7 days late.

This would massively reduce the risk of logstash getting into a muddle.

The late metrics data would be dropped instead of elastic returning a 409 forbidden error.