Add allocation tests for searchable snapshot remote shards #5048

kotwanikunal · 2022-11-02T19:51:50Z

Signed-off-by: Kunal Kotwani kkotwani@amazon.com

Description

Add allocation tests for searchable snapshot remote shards

Issues Resolved

[Searchable Snapshot] Add integration test for failure scenarios #4962

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2022-11-02T19:55:59Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/6256/
CommitID: db495f5
Please examine the workflow log, locate, and copy-paste the failure below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2022-11-02T20:05:17Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/6257/
CommitID: 1817e04
Please examine the workflow log, locate, and copy-paste the failure below, then iterate to green.
Is the failure a flaky test unrelated to your change?

andrross · 2022-11-02T20:03:25Z

server/src/internalClusterTest/java/org/opensearch/snapshots/SearchableSnapshotIT.java

+            .setRenamePattern("(.+)")
+            .setRenameReplacement("$1-copy")
+            .setStorageType(RestoreSnapshotRequest.StorageType.REMOTE_SNAPSHOT)
+            .setWaitForCompletion(true)


I'd be inclined to not set "wait for completion" because you assert the relevant state below and it is usually best to avoid making tests wait on timeouts like this if possible

Got rid of the synchronous timeout based call.

andrross · 2022-11-02T20:03:58Z

server/src/internalClusterTest/java/org/opensearch/snapshots/SearchableSnapshotIT.java

@@ -118,6 +125,206 @@ public void testCreateSearchableSnapshot() throws Exception {
        assertIndexDirectoryDoesNotExist("test-idx-1-copy", "test-idx-2-copy");
    }

+    /**
+     * Tests the functionality of {@link org.opensearch.cluster.routing.allocation.allocator.RemoteShardsBalancer} to


Nitpick but these types of documentation links tend to go stale with refactorings and it is probably not necessary to link to such low-level code in high-level integration tests like this.

Added general statements instead of specific implementation details.

server/src/internalClusterTest/java/org/opensearch/snapshots/SearchableSnapshotIT.java

github-actions · 2022-11-02T20:45:57Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/6258/
CommitID: 2b26d2a

github-actions · 2022-11-02T22:11:28Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/6263/
CommitID: e8857ea

andrross · 2022-11-02T23:26:21Z

server/src/internalClusterTest/java/org/opensearch/snapshots/SearchableSnapshotIT.java

+     * nodes with search capabilities are added back to the cluster.
+     */
+    public void testSearchableSnapshotAllocationForFailoverAndRecovery() throws Exception {
+        final int numReplicasIndex = randomIntBetween(1, 4);


What do you think about taking the randomness away here and setting number of replicas to 1. Then you could run the following scenario:

Ensure green after initial setup

Stop one search node and ensure yellow

Stop another search node and ensure red

Add 3 new search nodes and ensure green

Stop one search node and ensure green

Then I think you could get rid of the "testSearchableSnapshotAllocationWithNoSearchNodes" test case since that is covered here. It would also cover the "automatically recovery from red" scenario that currently isn't being tested.

This led to a discovery of an issue when the primary remote shards go unassigned. I'll add in the fix for it as well.
Thanks for the suggestion 🙂

github-actions · 2022-11-03T18:05:16Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/6316/
CommitID: cb8af5a
Please examine the workflow log, locate, and copy-paste the failure below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2022-11-03T20:00:55Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/6321/
CommitID: 3741502
Please examine the workflow log, locate, and copy-paste the failure below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2022-11-04T01:34:05Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/6331/
CommitID: 3741502
Please examine the workflow log, locate, and copy-paste the failure below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2022-11-04T04:34:30Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/6339/
CommitID: 3741502
Please examine the workflow log, locate, and copy-paste the failure below, then iterate to green.
Is the failure a flaky test unrelated to your change?

kotwanikunal · 2022-11-04T17:39:10Z

This will cause a merge conflict due to #5055. Updating the PR.

github-actions · 2022-11-04T17:40:07Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/6376/
CommitID: 8f2a00f
Please examine the workflow log, locate, and copy-paste the failure below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Signed-off-by: Kunal Kotwani <kkotwani@amazon.com>

github-actions · 2022-11-04T18:19:35Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/6381/
CommitID: 118cca1

Signed-off-by: Kunal Kotwani <kkotwani@amazon.com> Signed-off-by: Kunal Kotwani <kkotwani@amazon.com> (cherry picked from commit 082f059) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

Signed-off-by: Kunal Kotwani <kkotwani@amazon.com> Signed-off-by: Kunal Kotwani <kkotwani@amazon.com> (cherry picked from commit 082f059) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Signed-off-by: Kunal Kotwani <kkotwani@amazon.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

kotwanikunal requested review from a team and reta as code owners November 2, 2022 19:51

kotwanikunal force-pushed the searchable-snapshot-integ-tests branch from db495f5 to 1817e04 Compare November 2, 2022 19:52

kotwanikunal force-pushed the searchable-snapshot-integ-tests branch from 1817e04 to 2b26d2a Compare November 2, 2022 20:14

andrross reviewed Nov 2, 2022

View reviewed changes

kotwanikunal force-pushed the searchable-snapshot-integ-tests branch from 2b26d2a to e8857ea Compare November 2, 2022 21:38

kotwanikunal added backport 2.x Backport to 2.x branch backport 2.4 Backport to 2.4 branch labels Nov 2, 2022

andrross approved these changes Nov 2, 2022

View reviewed changes

kotwanikunal force-pushed the searchable-snapshot-integ-tests branch from e8857ea to cb8af5a Compare November 3, 2022 18:04

andrross approved these changes Nov 3, 2022

View reviewed changes

kotwanikunal force-pushed the searchable-snapshot-integ-tests branch from cb8af5a to 3741502 Compare November 3, 2022 19:59

aabukhalil approved these changes Nov 3, 2022

View reviewed changes

kotwanikunal mentioned this pull request Nov 4, 2022

Add integration tests for searchable snapshots, Fix length calculation for block based storage #5068

Merged

6 tasks

andrross removed the backport 2.4 Backport to 2.4 branch label Nov 4, 2022

kotwanikunal force-pushed the searchable-snapshot-integ-tests branch from 3741502 to 8f2a00f Compare November 4, 2022 17:04

aabukhalil approved these changes Nov 4, 2022

View reviewed changes

Add allocation tests for remote shards

118cca1

Signed-off-by: Kunal Kotwani <kkotwani@amazon.com>

kotwanikunal force-pushed the searchable-snapshot-integ-tests branch from 8f2a00f to 118cca1 Compare November 4, 2022 17:46

kotwanikunal merged commit 082f059 into opensearch-project:main Nov 4, 2022

opensearch-trigger-bot bot mentioned this pull request Nov 4, 2022

[Backport 2.x] Add allocation tests for searchable snapshot remote shards #5086

Merged

kotwanikunal deleted the searchable-snapshot-integ-tests branch January 12, 2023 17:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add allocation tests for searchable snapshot remote shards #5048

Add allocation tests for searchable snapshot remote shards #5048

kotwanikunal commented Nov 2, 2022

github-actions bot commented Nov 2, 2022

github-actions bot commented Nov 2, 2022

andrross Nov 2, 2022

kotwanikunal Nov 2, 2022

andrross Nov 2, 2022

kotwanikunal Nov 2, 2022

github-actions bot commented Nov 2, 2022

github-actions bot commented Nov 2, 2022

andrross Nov 2, 2022

kotwanikunal Nov 3, 2022

github-actions bot commented Nov 3, 2022

github-actions bot commented Nov 3, 2022

github-actions bot commented Nov 4, 2022

github-actions bot commented Nov 4, 2022

kotwanikunal commented Nov 4, 2022

github-actions bot commented Nov 4, 2022

github-actions bot commented Nov 4, 2022

Add allocation tests for searchable snapshot remote shards #5048

Add allocation tests for searchable snapshot remote shards #5048

Conversation

kotwanikunal commented Nov 2, 2022

Description

Issues Resolved

Check List

github-actions bot commented Nov 2, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Nov 2, 2022

Gradle Check (Jenkins) Run Completed with:

andrross Nov 2, 2022

Choose a reason for hiding this comment

kotwanikunal Nov 2, 2022

Choose a reason for hiding this comment

andrross Nov 2, 2022

Choose a reason for hiding this comment

kotwanikunal Nov 2, 2022

Choose a reason for hiding this comment

github-actions bot commented Nov 2, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Nov 2, 2022

Gradle Check (Jenkins) Run Completed with:

andrross Nov 2, 2022

Choose a reason for hiding this comment

kotwanikunal Nov 3, 2022

Choose a reason for hiding this comment

github-actions bot commented Nov 3, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Nov 3, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Nov 4, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Nov 4, 2022

Gradle Check (Jenkins) Run Completed with:

kotwanikunal commented Nov 4, 2022

github-actions bot commented Nov 4, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Nov 4, 2022

Gradle Check (Jenkins) Run Completed with: