help request: Fluentbit intermittent 404 issues on /api/v2/metrics/prometheus endpoint

[mshivanna]

Describe the issue

We have installed 3.2.0 version of fluent operator helm chart and have enabled fluentbit to ship logs from k8s containers to opensearch in aws. When we turn on servicemonitor we see 404 on /api/v2/metrics/prometheus on different fluentbit pods at different times.
at t1:

at t2:

How did you install fluent operator?

helm chart version 3.2.0

Additional context

No response

[cw-Guo]

this looks more like a fluent-bit issue. Please raise the issue there. https://github.com/fluent/fluent-bit/issues

It's possible that your fluent-bit pods are not running healthy at that time.

[mshivanna]

ok Thanks. The metrics buffer is not getting populated and the server sends a 404. https://github.com/fluent/fluent-bit/blob/master/src/http_server/api/v2/metrics.c#L181

[mshivanna]

similar issue while using prometheus exporter fluent/fluent-bit#9366