Add support to skip pinned timestamp in remote translog garbage collector

[sachinpkale]

Description

• As part of shallow snapshot optimisations, we are introducing timestamp based implicit locking support. This feature will be called Timestamp Pinning in remote backed storage. When a timestamp is pinned, remote store garbage collectors will skip deleting data corresponding to the timestamp.
• Changes to segment store garbage collection are made as part of PR: Add support to skip pinned timestamp in remote segment garbage collector #15017
• In this PR, we are making following changes to translog garbage collection:
  • Fetch metadata files in reverse-chronological order
  • Filter out metadata files that match with pinned timestamp and metadata files that are created post successful fetch of pinned timestamp.
  • From the remaining metadata files, read min and max generation and filter out generations by pinned timestamp metadata files.

Related Issues

• [Timestamp Pinning in Remote Store] Remote translog garbage collection changes #15064

Check List

• Functionality includes testing.
• API changes companion pull request created, if applicable.
• Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

❌ Gradle check result for 56d7626: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 82091a2: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 1a280db: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 8c32686: null

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for a69a975: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 770a791: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 770a791: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❕ Gradle check result for 770a791: UNSTABLE

• TEST FAILURES:

      1 org.opensearch.smoketest.SmokeTestMultiNodeClientYamlTestSuiteIT.test {yaml=search.aggregation/360_date_histogram/Date histogram aggregation w/ shared field range test}
      1 org.opensearch.action.admin.cluster.node.tasks.ResourceAwareTasksTests.testTaskResourceTrackingDuringTaskCancellation

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.