docs: Document timestamp ordering rules (#1521)

Co-authored-by: Ed Welch <ed@oqqer.com>

docs/api.md

@@ -484,9 +484,12 @@ JSON post body can be sent in the following format:
 ```
 
 > **NOTE**: logs sent to Loki for every stream must be in timestamp-ascending
-> order, meaning each log line must be more recent than the one last received.
-> If logs do not follow this order, Loki will reject the log with an out of
-> order error.
+> order; logs with identical timestamps are only allowed if their content
+> differs. If a log line is received with a timestamp older than the most
+> recent received log, it is rejected with an out of order error. If a log
+> is received with the same timestamp and content as the most recent log, it is
+> silently ignored. For more details on the ordering rules, refer to the
+> [Loki Overview docs](./overview/README.md#timestamp-ordering).
 
 In microservices mode, `/loki/api/v1/push` is exposed by the distributor.
 

docs/architecture.md

@@ -150,6 +150,11 @@ if any write failed to one of the replicas, multiple differing chunk objects
 will be created in the backing store. See [Querier](#querier) for how data is
 deduplicated.
 
+The ingesters validate timestamps for each log line received maintains a
+strict ordering. See the [Loki
+Overview](./overview/README.md#timestamp-ordering) for detailed documentation on
+the rules of timestamp order.
+
 #### Handoff
 
 By default, when an ingester is shutting down and tries to leave the hash ring,

docs/clients/promtail/troubleshooting.md

@@ -98,11 +98,11 @@ from there. This means that if new log entries have been read and pushed to the
 ingester between the last sync period and the crash, these log entries will be
 sent again to the ingester on `promtail` restart.
 
-However, for each log stream (set of unique labels) the Loki ingester skips all
-log entries received out of timestamp order. For this reason, even if duplicated
-logs may be sent from `promtail` to the ingester, entries whose timestamp is
-older than the latest received will be discarded to avoid having duplicated
-logs. To leverage this, it's important that your `pipeline_stages` include
-the `timestamp` stage, parsing the log entry timestamp from the log line instead
-of relying on the default behaviour of setting the timestamp as the point in
-time when the line is read by `promtail`.
+However, it's important to note that Loki will reject all log lines received in
+what it perceives is [out of
+order](../../overview/README.md#timestamp-ordering). If `promtail` happens to
+crash, it may re-send log lines that were sent prior to the crash. The default
+behavior of Promtail is to assign a timestamp to logs at the time it read the
+entry from the tailed file. This would result in duplicate log lines being sent
+to Loki; to avoid this issue, if your tailed file has a timestamp embedded in
+the log lines, a `timestamp` stage should be added to your pipeline.

docs/overview/README.md

@@ -68,6 +68,11 @@ the sample to before responding to the user.
 The **ingester** service is responsible for writing log data to long-term
 storage backends (DynamoDB, S3, Cassandra, etc.).
 
+The ingester validates that ingested log lines are not out of order. When an
+ingester receives a log line that doesn't follow the expected order, the line
+is rejected and an error is returned to the user. See the section on [Timestamp
+ordering](#timestamp-ordering) for more information.
+
 The ingester validates that ingested log lines are received in
 timestamp-ascending order (i.e., each log has a timestamp that occurs at a later
 time than the log before it). When the ingester receives a log that does not
@@ -80,6 +85,21 @@ If an ingester process crashes or exits abruptly, all the data that has not yet
 been flushed will be lost. Loki is usually configured to replicate multiple
 replicas (usually 3) of each log to mitigate this risk.
 
+#### Timestamp Ordering
+
+In general, all lines pushed to Loki for a given stream (unique combination of
+labels) must have a newer timestamp than the line received before it. There are,
+however, two cases for handling logs for the same stream with identical
+nanosecond timestamps:
+
+1. If the incoming line exactly matches the previously received line (matching
+   both the previous timestamp and log text), the incoming line will be treated
+   as an exact duplicate and ignored.
+
+2. If the incoming line has the same timestamp as the previous line but
+   different content, the log line is accepted. This means it is possible to
+   have two different log lines for the same timestamp.
+
 #### Handoff
 
 By default, when an ingester is shutting down and tries to leave the hash ring,