Add Changes in Snapshot Clone Flow for remote store interoperability. #7496

kasundra07 · 2023-05-10T00:27:50Z

Description

Changes in Clone Snapshot API required for Snapshot interoperability with Remote Store.

Related Issues

This change is dependent on Lock Manager changes and Create Snapshot API changes.

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2023-05-10T00:34:32Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/15160/
CommitID: f673eb2
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-05-17T20:33:09Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/15626/
CommitID: 085c788
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-05-17T20:45:22Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/15629/
CommitID: 4537753
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-05-17T21:03:17Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/15633/
CommitID: e7759fa
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

sohami

Please add tests for successful and failure cases.

cloning with combination of source and target snapshot being shallow vs full snapshot, each snapshot with mix of regular + remote store index.
Failure cases like after taking lock there is failure to write metadata file, or even failure after writing both the files, etc

server/src/main/java/org/opensearch/index/store/lockmanager/RemoteStoreMetadataLockManager.java

server/src/main/java/org/opensearch/repositories/blobstore/BlobStoreRepository.java

server/src/main/java/org/opensearch/snapshots/SnapshotsService.java

server/src/main/java/org/opensearch/index/store/lockmanager/RemoteStoreLockManager.java

server/src/main/java/org/opensearch/repositories/Repository.java

Bukhtawar

Do we need thread safety for the LockManager interface, espl when we do cloning how do we know the lock is not released while getting cloned. I would recommend implement locks using a Releasable interface and add thread safety to the implementation

Signed-off-by: Bansi Kasundra <kasundra@amazon.com>

github-actions · 2023-07-06T19:27:09Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/19312/
CommitID: 6b8a5d1
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-07-06T20:08:21Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/19316/
CommitID: fd15428
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Signed-off-by: Bansi Kasundra <kasundra@amazon.com>

github-actions · 2023-07-06T23:58:49Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.remotestore.RemoteStoreRefreshListenerIT.testRemoteRefreshRetryOnFailure
      1 org.opensearch.remotestore.RemoteStoreIT.testStaleCommitDeletionWithoutInvokeFlush

URL: https://build.ci.opensearch.org/job/gradle-check/19341/
CommitID: 5a7b3ee
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

…#7496) Signed-off-by: Bansi Kasundra <kasundra@amazon.com> (cherry picked from commit 0e037a2) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…#7496) (#8515) (cherry picked from commit 0e037a2) Signed-off-by: Bansi Kasundra <kasundra@amazon.com>

…opensearch-project#7496) Signed-off-by: Bansi Kasundra <kasundra@amazon.com>

…opensearch-project#7496) Signed-off-by: Bansi Kasundra <kasundra@amazon.com> Signed-off-by: sahil buddharaju <sahilbud@amazon.com>

…opensearch-project#7496) Signed-off-by: Bansi Kasundra <kasundra@amazon.com>

…opensearch-project#7496) Signed-off-by: Bansi Kasundra <kasundra@amazon.com> Signed-off-by: Shivansh Arora <hishiv@amazon.com>

kasundra07 requested review from reta, anasalkouz, andrross, Bukhtawar, CEHENKLE, dblock, gbbafna, setiah, kartg, kotwanikunal, mch2, nknize, owaiskazi19, Rishikesh1159, ryanbogan, saratvemulapalli, shwetathareja, dreamer-89, tlfeng and VachaShah as code owners May 10, 2023 00:27

kasundra07 force-pushed the cloneSnapshot branch from f673eb2 to 085c788 Compare May 17, 2023 20:24

kasundra07 force-pushed the cloneSnapshot branch from 085c788 to 4537753 Compare May 17, 2023 20:36

sohami requested changes May 18, 2023

View reviewed changes

Bukhtawar reviewed May 18, 2023

View reviewed changes

server/src/main/java/org/opensearch/index/store/lockmanager/RemoteStoreLockManager.java Outdated Show resolved Hide resolved

Bukhtawar reviewed May 18, 2023

View reviewed changes

server/src/main/java/org/opensearch/repositories/Repository.java Outdated Show resolved Hide resolved

Bukhtawar reviewed May 18, 2023

View reviewed changes

gbbafna added the skip-changelog label Jul 6, 2023

kasundra07 added 8 commits July 6, 2023 12:15

Add Changes in Snapshot Clone Flow for remote store interoperability.

8041083

Signed-off-by: Bansi Kasundra <kasundra@amazon.com>

Addressed PR comments

656c40c

Signed-off-by: Bansi Kasundra <kasundra@amazon.com>

Minor Clone Snapshot changes related to parameter

90ffb17

Signed-off-by: Bansi Kasundra <kasundra@amazon.com>

Added Tests for Clone Snapshot API

2b5945a

Signed-off-by: Bansi Kasundra <kasundra@amazon.com>

Updated changelog and added default implementation

a1750e3

Signed-off-by: Bansi Kasundra <kasundra@amazon.com>

Addressed PR comments

7645c83

Signed-off-by: Bansi Kasundra <kasundra@amazon.com>

Addressed PR comments

fef5b5c

Signed-off-by: Bansi Kasundra <kasundra@amazon.com>

Addressed PR comment

5cc46cb

Signed-off-by: Bansi Kasundra <kasundra@amazon.com>

kasundra07 force-pushed the cloneSnapshot branch from c61df0d to 6b8a5d1 Compare July 6, 2023 19:17

kasundra07 force-pushed the cloneSnapshot branch from 6b8a5d1 to fd15428 Compare July 6, 2023 19:45

Addressed PR comments

5a7b3ee

Signed-off-by: Bansi Kasundra <kasundra@amazon.com>

kasundra07 force-pushed the cloneSnapshot branch from fd15428 to 5a7b3ee Compare July 6, 2023 23:26

gbbafna approved these changes Jul 7, 2023

View reviewed changes

harishbhakuni approved these changes Jul 7, 2023

View reviewed changes

gbbafna merged commit 0e037a2 into opensearch-project:main Jul 7, 2023

gbbafna added the backport 2.x Backport to 2.x branch label Jul 7, 2023

opensearch-trigger-bot bot mentioned this pull request Jul 7, 2023

[Backport 2.x] Add Changes in Snapshot Clone Flow for remote store interoperability. #8515

Merged

gbbafna pushed a commit that referenced this pull request Jul 7, 2023

Add Changes in Snapshot Clone Flow for remote store interoperability. (…

94b1d48

…#7496) (#8515) (cherry picked from commit 0e037a2) Signed-off-by: Bansi Kasundra <kasundra@amazon.com>

vikasvb90 pushed a commit to raghuvanshraj/OpenSearch that referenced this pull request Jul 12, 2023

Add Changes in Snapshot Clone Flow for remote store interoperability. (…

8bac2c0

…opensearch-project#7496) Signed-off-by: Bansi Kasundra <kasundra@amazon.com>

raghuvanshraj pushed a commit to raghuvanshraj/OpenSearch that referenced this pull request Jul 12, 2023

Add Changes in Snapshot Clone Flow for remote store interoperability. (…

1373a76

…opensearch-project#7496) Signed-off-by: Bansi Kasundra <kasundra@amazon.com>

dzane17 pushed a commit to dzane17/OpenSearch that referenced this pull request Jul 12, 2023

Add Changes in Snapshot Clone Flow for remote store interoperability. (…

52ed230

…opensearch-project#7496) Signed-off-by: Bansi Kasundra <kasundra@amazon.com>

baba-devv pushed a commit to baba-devv/OpenSearch that referenced this pull request Jul 29, 2023

Add Changes in Snapshot Clone Flow for remote store interoperability. (…

a84b673

…opensearch-project#7496) Signed-off-by: Bansi Kasundra <kasundra@amazon.com>

ashking94 mentioned this pull request Oct 12, 2024

[Feature Request] Optimise clone operation for incremental full cluster snapshots #16295

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Changes in Snapshot Clone Flow for remote store interoperability. #7496

Add Changes in Snapshot Clone Flow for remote store interoperability. #7496

kasundra07 commented May 10, 2023 •

edited

Loading

github-actions bot commented May 10, 2023

github-actions bot commented May 17, 2023

github-actions bot commented May 17, 2023

github-actions bot commented May 17, 2023

sohami left a comment •

edited

Loading

Bukhtawar left a comment

github-actions bot commented Jul 6, 2023

github-actions bot commented Jul 6, 2023

github-actions bot commented Jul 6, 2023

Add Changes in Snapshot Clone Flow for remote store interoperability. #7496

Add Changes in Snapshot Clone Flow for remote store interoperability. #7496

Conversation

kasundra07 commented May 10, 2023 • edited Loading

Description

Related Issues

Check List

github-actions bot commented May 10, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented May 17, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented May 17, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented May 17, 2023

Gradle Check (Jenkins) Run Completed with:

sohami left a comment • edited Loading

Choose a reason for hiding this comment

Bukhtawar left a comment

Choose a reason for hiding this comment

github-actions bot commented Jul 6, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jul 6, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jul 6, 2023

Gradle Check (Jenkins) Run Completed with:

kasundra07 commented May 10, 2023 •

edited

Loading

sohami left a comment •

edited

Loading