[Segment Replication] Add SegmentReplicationTargetService to orchestrate replication events.

[mch2]

Signed-off-by: Marc Handalian handalm@amazon.com

Description

This PR introduces components that orchestrate the flow for Segment Replication events. They are not currently wired up as part of this PR. This will happen in following PRs.

• SegmentReplicationTargetService - orchestration component for replication events. Registers a request handler to handle file chunk requests for segrep.
• SegmentReplicationSource - Interface that will provide checkpoint data & copy segment files. This PR only introduces the interface, the next PR will include an implementation where the primary shard is the source of replication.
• Request/response objects from SegmentReplicationSource
• SegmentReplicationSourceFactory - A factory class that determines the source to use for a particular shard.

Along with some stubbed components that will be implemented in upcoming PRs:
SegmentReplicationTarget - ReplicationTarget implementation for segrep, will handle invoking methods on a provided source.
SegmentReplicationState State object to track replication events.

It also includes two refactors of peer recovery components for reuse.

• Rename RecoveryFileChunkRequest to FileChunkRequest and extract code to handle throttling into
  ReplicationTarget.
• Extracts a component to execute retryable requests over the transport layer.

Issues Resolved

closes #3112
relates #2355

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[opensearch-ci-bot]

❌   Gradle Check failure 16cafd1
Log 5588

Reports 5588

[mch2]

start gradle check

[opensearch-ci-bot]

✅   Gradle Check success 16cafd1
Log 5589

Reports 5589

[kartg]

Only nitpicks, so approved

[kartg]

Doesn't seem like this would ever be null ? If so, you can remove all of the null checks from the numerous times PeerRecoveryTargetService invokes this method.

[mch2]

markRequestReceivedAndCreateListener delegates to ReplicationRequestTracker.markReceivedAndCreateListener which can return null if the unique request id has already been processed. In this case it completes the voidListener and returns null. I'll add nullable all the way down so we are aware of why the checks are necessary.

[opensearch-ci-bot]

✅   Gradle Check success 4c3e74d
Log 5617

Reports 5617

[nknize]

I think this looks good as an incremental step. It does take a lot of domain knowledge to see how the refactor is being done but the PR explanation helps a lot here. I like the intentional avoidance of the monolithic class implementations like RecoverySourceHandler. The SegmentReplicationTargetServiceTests help follow the placeholders.

And given that this is a large development effort of a feature that is already feature flagged, we should be okay w/ incremental TODOs; this is good progress over perfection.

One suggestion, lets design to keep the public facing class footprint minimal and make all classes final that we don't want extended by a plugin (especially if they have to be public facing in order to be used across internal packages).

[nknize]

We should be able to make this final so it can't be extended? I'm also assuming it has to remain public so the Node object can bind it and it can used by IndicesService and IndexShard?

Suggested change

	public class SegmentReplicationTargetService implements IndexEventListener {
	public final class SegmentReplicationTargetService implements IndexEventListener {

[opensearch-ci-bot]

❌   Gradle Check failure b39c2664bd8fb8d86d46864e661dfe5ac24a82b3
Log 5635

Reports 5635

[opensearch-ci-bot]

✅   Gradle Check success d07ee74
Log 5636

Reports 5636

[opensearch-trigger-bot]

The backport to 2.2 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-2.2 2.2
# Navigate to the new working tree
cd .worktrees/backport-2.2
# Create a new branch
git switch --create backport/backport-3439-to-2.2
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 e9b19a095c2cf90274c41aa701432a2748c03468
# Push it to GitHub
git push --set-upstream origin backport/backport-3439-to-2.2
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-2.2

Then, create a pull request where the base branch is 2.2 and the compare/head branch is backport/backport-3439-to-2.2.

[kartg]

2.2 backport is not required since this change was already in 2.x when 2.2 was cut from it - 343045f