-
Notifications
You must be signed in to change notification settings - Fork 6
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Entry out of order for stream #16
Comments
No, it isn't. Because Go loki client library makes chunks which are grouped events and it doesn't provide failure record confirmation interface or function. |
@cosmo0920 I think the error can be explained by this: grafana/loki#168 (comment) I am running multiple instances of fluent-bit but they are given the same labels as seen in the logs above - I guess this plugin should add some unique source label as per the comment, although an instance label seems more sensible IMO to be closer to what Prometheus is doing. |
Just seen that you already support extra labels - I gave my instances unique ones and there are still the same errors on startup (about 10) but after that there are no more (so far at least) |
Hmm..., it seems that this issue is caused by Loki itself.... |
Potentially related: fluent/fluent-bit#1746 |
This repository is deprecated. |
Problem
Fluent-bit appears to be sending log messages out of order and Loki is rejecting them - finding it hard to narrow down the source of the issue. There are lots of rejections when fluent-bit starts up and they continue but settle down quickly. I built the fluent-bit-go-loki library from master which could have an impact - see commit hash in td-agent-bit output below.
Maybe related:
Steps to replicate
Fluent-bit conf
Loki conf
Expected Behavior or What you need to ask
No out of order messages - is it possible to see which logs are being rejected? I change log level to trace and there was no obvious extra relevant info.
Using Fluentd and loki plugin version
fluent-bit --version
ortd-agent-bin --version
6.2.5
, Loki0.2.0
The text was updated successfully, but these errors were encountered: