[BUG] Segment replication lag metric seems to be incorrect

[varunbharadwaj]

Describe the bug

Segment replication lag metric seems to be incorrect. On testing with both push and pull based indexing, the observation is the replication lag (segments.segment_replication.max_replication_lag from node stats API) catches up soon after indexing stops. But the replication lag value does not seem to match bytes behind and other metrics, and is very high.

We convert the replication lag in milliseconds and is seen in days in the following graph.

Verify/confirm if this is a bug.

Related component

Indexing:Replication

To Reproduce

1. Setup a cluster running on segment replication with remote store (GCS). OS 3.x is used (latest main branch - 3.1.0 unreleased)
2. Note down segment replication metrics by calling node stats API (segments.segment_replication.max_replication_lag metric)
3. Verify the metric is correct and matching other segrep metrics (like segments.segment_replication.max_bytes_behind)

Expected behavior

Replication lag metric is similar to bytes behind and other segrep metrics

Additional Details

[varunbharadwaj]

@mch2 This is the potential bug we were discussing few weeks ago. Creating an issue to continue the discussion.

[mch2]

@varunbharadwaj Thanks for raising this. I think its about time that we fix our segrep metrics for good, the current implementation where primary accounts for each replica's checkpoint states in its replicationTracker is flaky at best.

I suggest we adopt the strategy we use for search replicas with r/w split, where we compute best effort lag/bytes behind metrics only on the replicas themselves based on latest fetched checkpoint. We can compute lag to a particular checkpoint as queued time + ongoing sync time.

[mch2]

Ok I was wrong here about tracking at primary, that no longer occurs outside of cases for backpressure. There are a couple of issues with our computation, it looks like when we updated this to compute at replicas we did not update to use time computations referenced from epoch for across jvm durations, leading to unpredictable times. The timestamp on checkpoint was intended for creation & duration computations only at the primary. Further we are not pruning the highest checkpoint from tracking after a replica finishes syncing to that checkpoint. This means that the next time we receive a checkpoint the replica will compute lag against. Example:

Checkpoint 1 - time 1 bytes 200
Replica finishes to Checkpoint 1.
Checkpoint 2 - time 100, bytes 500
Metrics here computes time as now - 1 instead of now-100, and bytes behind as 300 (diff between current cp and new incoming).

[varunbharadwaj]

Good find! Thanks for the quick debugging and fix @mch2 !