Out of order logs from internal_logs to loki (?)

[abh]

Running nightly-2021-12-28-debian I am getting errors from the loki sink that the messages were sent out of order.

2022-01-18T18:02:29.289711Z  WARN sink{component_kind="sink" component_id=loki-vector component_type=loki component_name=loki-vector}: vector::internal_events::loki: Received out-of-order event; dropping event. internal_log_rate_secs=30
2022-01-18T18:02:29.289682Z  WARN sink{component_kind="sink" component_id=loki-vector component_type=loki component_name=loki-vector}: vector::internal_events::loki: Internal log [Received out-of-order event; dropping event.] has been rate limited 1 times.
2022-01-18T18:01:06.166170Z  WARN sink{component_kind="sink" component_id=loki-vector component_type=loki component_name=loki-vector}: vector::internal_events::loki: Received out-of-order event; dropping event. internal_log_rate_secs=30

  sources:
    vector-logs:
      type: "internal_logs"

sinks:
    loki-vector:
      type: loki
      inputs:
      - vector-logs
      endpoint: "http://loki-gateway.ntpsystem/"
      tenant_id: "ntpservers"
      healthcheck: false
      encoding:
        codec: "json"
      labels:
        type: "vector"
        host: "${HOSTNAME}"
      buffer:
        type: "disk"
        max_size: 100000000  # ~100mi

I verified that the HOSTNAME environment variable is set on each vector instance and are each unique (there are four running in a kubernetes statefulset).

[jszwedko]

Technically I think this could happen because Vector is mult-threaded and may have different threads writing logs at the same time where they end up out of order. Given that Loki 2.4 relaxes the ordering constraints, I'm inclined to close this in favor of #11131

[jszwedko]

Closing in-lieu of #11131 . If you are running Loki >= 2.4 you can set out_of_order_action = "allow" in Vector 0.21.

[pgassmann]

out_of_order_action = "accept" not allow
https://vector.dev/docs/reference/configuration/sinks/loki/#out_of_order_action