server returned HTTP status 400 Bad Request (400): entry out of order for stream

[bvandewalle]

I'm getting those type of errors nonstop on my cluster's Loki:

level=error ts=2019-03-29T02:06:49.147408285Z caller=client.go:185 msg="final error sending batch" status=400 error="server returned HTTP status 400 Bad Request (400): entry out of order for stream: {__filename__=\"/var/log/pods/33cce0fa-4fe7-11e9-a0ac-0a75695ff060/calico-node/0.log\", container_name=\"calico-node\", controller_revision_hash=\"4160928241\", instance=\"calico-node-gd7fx\", job=\"kube-system/\", k8s_app=\"calico-node\", namespace=\"kube-system\", pod_template_generation=\"1\", role_kubernetes_io_networking=\"1\", stream=\"stdout\"}"

I deployed the upstream Helm chart, the only modification I did was to delete all the drop statements from the config file (with those drop statements I would miss most of my pods for some reason).

[daixiang0]

Without delete all drop rule it would be ok i think.
The reason you mean is tracked in #432.

[daixiang0]

I test at local, the whole loki service still work, while those logs exist.

[stale]

This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[olegderid]

Please, reopen this issue it is reproduced consistently.

[ofiry-tm]

I have this issue also. it come and goes and i have no idea why