[Segment replication] Point in time search comaptibility

[bharath-techie]

Currently segment replication will be a breaking change for point in time search. We need to refactor point in time flow to support segment replication.

Reference comment - #2212 (comment)

[mch2]

I've been trying to write an integ test here that proves scroll/PIT fails with SR, but I am unable to. Even after confirming segments are not present locally my requests are succeeding. Test I am trying - https://github.com/mch2/OpenSearch/blob/9dddfab35437b776916d8769d95b3684e81b742b/server/src/internalClusterTest/java/org/opensearch/search/pit/PitMultiNodeIT.java#L179

[mch2]

@nknize / @bharath-techie Curious of your thoughts on #6644. This will solve the issue of a PIT/scroll request created on a replica by preserving segments until a reader context is closed.

We still face an issue with PIT when the context is created on a primary, but I believe PIT queries are currently sticky and return to the same shard? We currently send only the primary's current infos to replicas, and the replica purges files that are not contained in those infos & the latest commit point. We will need to send additional filenames to the replica to ensure it preserves the same set of segments if queries are not sticky.

[mch2]

Realized there is another gap here with SR. Any open pit reqs on an active replica will be broken once promoted to primary. We'll need to re-acquire the reader references or somehow hand off the incref'd files when we transition from NRT to Internal Engine and incref them on the indexWriter.