[Remote Store] Add Lock Manager in Remote Segment Store to persist data #6787

harishbhakuni · 2023-03-22T02:31:39Z

… for longer duration.

Description

Changes for Lock Manager in Remote Segment Store to keep data for longer duration and prevent their deletion from Garbage Collection.
Manually Tested Feature End to end on an opensearch cluster.

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

Issues Resolved

#6691
#6688

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2023-03-22T02:35:23Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/12765/
CommitID: a814037
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

server/src/main/java/org/opensearch/index/store/lockmanager/RemoteStoreMDLockManager.java

server/src/main/java/org/opensearch/index/store/RemoteSegmentStoreDirectoryFactory.java

server/src/main/java/org/opensearch/index/store/lockmanager/RemoteLockIndexOutput.java

server/src/main/java/org/opensearch/index/store/lockmanager/RemoteStoreMDLockManager.java

server/src/main/java/org/opensearch/index/store/lockmanager/RemoteLockDirectory.java

server/src/main/java/org/opensearch/index/store/RemoteSegmentStoreDirectory.java

server/src/test/java/org/opensearch/index/store/RemoteSegmentStoreDirectoryTests.java

server/src/main/java/org/opensearch/index/store/lockmanager/RemoteLockDirectory.java

server/src/main/java/org/opensearch/index/store/lockmanager/RemoteLockIndexOutput.java

server/src/main/java/org/opensearch/index/store/RemoteSegmentStoreDirectory.java

github-actions · 2023-04-11T22:57:52Z

Gradle Check (Jenkins) Run Completed with:

RESULT: null ❌
URL: null
CommitID: b4e361c
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-05-09T08:12:42Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/15095/
CommitID: 47cf901
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-05-09T08:20:15Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.indices.replication.SegmentReplicationIT.testReplicationPostDeleteAndForceMerge

URL: https://build.ci.opensearch.org/job/gradle-check/15094/
CommitID: dd59d70
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

github-actions · 2023-05-09T08:43:51Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/15096/
CommitID: a690e95

sohami

Few minor comments, otherwise LGTM. Can you also please create the tracking issues for:
a) race condition discussed here
b) monitor to catch any leaked lock files for which metadata file is not present

server/src/main/java/org/opensearch/index/store/RemoteSegmentStoreDirectory.java

server/src/main/java/org/opensearch/index/store/lockmanager/FileLockInfo.java

harishbhakuni · 2023-05-10T02:47:06Z

Few minor comments, otherwise LGTM. Can you also please create the tracking issues for:
a) race condition discussed #6787 (comment)
b) monitor to catch any leaked lock files for which metadata file is not present

Created Issues to track: #7500 #7501

github-actions · 2023-05-10T03:02:01Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/15165/
CommitID: 8d102f0

CHANGELOG.md

server/src/main/java/org/opensearch/index/store/lockmanager/FileLockInfo.java

server/src/main/java/org/opensearch/index/store/lockmanager/RemoteStoreMetadataLockManager.java

server/src/main/java/org/opensearch/index/store/RemoteBufferedIndexOutput.java

.../src/main/java/org/opensearch/index/store/lockmanager/RemoteStoreCommitLevelLockManager.java

Signed-off-by: Harish Bhakuni <hbhakuni@amazon.com>

github-actions · 2023-05-11T04:37:02Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.remotestore.SegmentReplicationRemoteStoreIT.testScrollWithOngoingSegmentReplication
      1 org.opensearch.cluster.allocation.AwarenessAllocationIT.testThreeZoneOneReplicaWithForceZoneValueAndLoadAwareness

URL: https://build.ci.opensearch.org/job/gradle-check/15216/
CommitID: fa75afd
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

…ta (#6787) Signed-off-by: Harish Bhakuni <hbhakuni@amazon.com> (cherry picked from commit 20ad1d8) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…ta (#6787) (#7517) (cherry picked from commit 20ad1d8) Signed-off-by: Harish Bhakuni <hbhakuni@amazon.com>

Bukhtawar · 2023-05-16T09:45:54Z

server/src/main/java/org/opensearch/index/store/RemoteBufferedIndexOutput.java

+    public void close() throws IOException {
+
+        try (final BytesStreamOutput outStream = out; InputStream stream = out.bytes().streamInput()) {
+            indexOutputBuffer.close();


Should indexOutputBuffer.close(); be in the finally block?

…ta (opensearch-project#6787) Signed-off-by: Harish Bhakuni <hbhakuni@amazon.com> Signed-off-by: Shivansh Arora <hishiv@amazon.com>

github-actions bot added the first-time-contributor label Mar 22, 2023

sohami requested changes Mar 29, 2023

View reviewed changes

gbbafna reviewed Mar 29, 2023

View reviewed changes

harishbhakuni closed this Apr 11, 2023

harishbhakuni force-pushed the lockManager branch from a814037 to b4e361c Compare April 11, 2023 22:57

harishbhakuni mentioned this pull request Apr 11, 2023

[Remote Store] Add Lock Manager in Remote Segment Store to persist da… #7116

Closed

harishbhakuni reopened this Apr 12, 2023

harishbhakuni marked this pull request as ready for review April 12, 2023 00:42

harishbhakuni requested review from reta, anasalkouz, andrross, Bukhtawar, CEHENKLE, dblock, setiah, kartg, kotwanikunal, mch2, nknize, owaiskazi19, Rishikesh1159, ryanbogan, saratvemulapalli, shwetathareja, dreamer-89, tlfeng, VachaShah and xuezhou25 as code owners April 12, 2023 00:42

This was referenced May 10, 2023

[Snapshot Interop] Add Changes in Snapshot Status Flow for remote sto… #7495

Merged

Add Changes in Snapshot Clone Flow for remote store interoperability. #7496

Merged

Add Changes in Snapshot Delete Flow for remote store interoperability. #7497

Merged

sohami reviewed May 10, 2023

View reviewed changes

harishbhakuni force-pushed the lockManager branch from a690e95 to 8d102f0 Compare May 10, 2023 02:30

harishbhakuni mentioned this pull request May 10, 2023

[Snapshot Interop] Fix Race Condition during acquiring lock on remote store metadata #7500

Open

sohami approved these changes May 10, 2023

View reviewed changes

gbbafna reviewed May 11, 2023

View reviewed changes

sachinpkale approved these changes May 11, 2023

View reviewed changes

gbbafna added the skip-changelog label May 11, 2023

Harish Bhakuni added 5 commits May 10, 2023 21:04

[Remote Store] Add Lock Manager Changes in Remote Segment Store.

25c9c24

Signed-off-by: Harish Bhakuni <hbhakuni@amazon.com>

Addressed PR Comments

26016c7

Signed-off-by: Harish Bhakuni <hbhakuni@amazon.com>

Fix Spotless Issue

713ee76

Signed-off-by: Harish Bhakuni <hbhakuni@amazon.com>

Addressed PR Comments

0f60fee

Signed-off-by: Harish Bhakuni <hbhakuni@amazon.com>

Addressed PR Comments

fa75afd

Signed-off-by: Harish Bhakuni <hbhakuni@amazon.com>

harishbhakuni force-pushed the lockManager branch from 8d102f0 to fa75afd Compare May 11, 2023 04:05

gbbafna approved these changes May 11, 2023

View reviewed changes

gbbafna merged commit 20ad1d8 into opensearch-project:main May 11, 2023

gbbafna added the backport 2.x Backport to 2.x branch label May 11, 2023

opensearch-trigger-bot bot mentioned this pull request May 11, 2023

[Backport 2.x] [Remote Store] Add Lock Manager in Remote Segment Store to persist data #7517

Merged

gbbafna pushed a commit that referenced this pull request May 15, 2023

[Remote Store] Add Lock Manager in Remote Segment Store to persist da…

75dbce2

…ta (#6787) (#7517) (cherry picked from commit 20ad1d8) Signed-off-by: Harish Bhakuni <hbhakuni@amazon.com>

Bukhtawar reviewed May 18, 2023

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Remote Store] Add Lock Manager in Remote Segment Store to persist data #6787

[Remote Store] Add Lock Manager in Remote Segment Store to persist data #6787

harishbhakuni commented Mar 22, 2023 •

edited

Loading

github-actions bot commented Mar 22, 2023

github-actions bot commented Apr 11, 2023

github-actions bot commented May 9, 2023

github-actions bot commented May 9, 2023

github-actions bot commented May 9, 2023

sohami left a comment

harishbhakuni commented May 10, 2023

github-actions bot commented May 10, 2023

github-actions bot commented May 11, 2023

Bukhtawar May 16, 2023

[Remote Store] Add Lock Manager in Remote Segment Store to persist data #6787

[Remote Store] Add Lock Manager in Remote Segment Store to persist data #6787

Conversation

harishbhakuni commented Mar 22, 2023 • edited Loading

Description

Check List

Issues Resolved

github-actions bot commented Mar 22, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Apr 11, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented May 9, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented May 9, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented May 9, 2023

Gradle Check (Jenkins) Run Completed with:

sohami left a comment

Choose a reason for hiding this comment

harishbhakuni commented May 10, 2023

github-actions bot commented May 10, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented May 11, 2023

Gradle Check (Jenkins) Run Completed with:

Bukhtawar May 16, 2023

Choose a reason for hiding this comment

harishbhakuni commented Mar 22, 2023 •

edited

Loading