[Flaky Test] SnapshotResiliencyTests.testConcurrentSnapshotDeleteAndDeleteIndex if flaky #8771

mch2 · 2023-07-19T05:29:31Z

https://build.ci.opensearch.org/job/gradle-check/20332/

REPRODUCE WITH: ./gradlew ':server:test' --tests "org.opensearch.snapshots.SnapshotResiliencyTests.testConcurrentSnapshotDeleteAndDeleteIndex" -Dtests.seed=B7F15DFF786CE73D -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=es-NI -Dtests.timezone=Asia/Oral -Druntime.java=17
NOTE: leaving temporary files on disk at: /var/jenkins/workspace/gradle-check/search/server/build/testrun/test/temp/org.opensearch.snapshots.SnapshotResiliencyTests_B7F15DFF786CE73D-001
NOTE: test params are: codec=Asserting(Lucene95): {}, docValues:{}, maxPointsInLeafNode=330, maxMBSortInHeap=6.1702426242178525, sim=Asserting(RandomSimilarity(queryNorm=true): {}), locale=es-NI, timezone=Asia/Oral
NOTE: Linux 5.15.0-1036-aws amd64/Eclipse Adoptium 17.0.7 (64-bit)/cpus=96,threads=1,free=189188400,total=536870912

The text was updated successfully, but these errors were encountered:

dblock · 2023-10-03T15:05:07Z

https://build.ci.opensearch.org/job/gradle-check/26641/

#10318 (comment)

andrross · 2024-02-15T16:29:57Z

#12335 (comment):

REPRODUCE WITH: ./gradlew ':server:test' --tests "org.opensearch.snapshots.SnapshotResiliencyTests.testConcurrentSnapshotDeleteAndDeleteIndex" -Dtests.seed=E816C7242330BF50 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=ms-MY -Dtests.timezone=Europe/Athens -Druntime.java=21

org.opensearch.snapshots.SnapshotResiliencyTests > testConcurrentSnapshotDeleteAndDeleteIndex FAILED
    java.lang.AssertionError: java.lang.StringIndexOutOfBoundsException: Range [13, 2) out of bounds for length 6
        at __randomizedtesting.SeedInfo.seed([E816C7242330BF50:BD5D3283E0ED3D1D]:0)
        at org.opensearch.repositories.blobstore.BlobStoreRepository.lambda$executeOneStaleIndexDelete$40(BlobStoreRepository.java:1709)
        at org.opensearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:74)
        at org.opensearch.action.ActionRunnable$2.doRun(ActionRunnable.java:89)
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)
        at org.opensearch.cluster.coordination.AbstractCoordinatorTestCase$1.run(AbstractCoordinatorTestCase.java:1498)
        at org.opensearch.cluster.coordination.DeterministicTaskQueue.runTask(DeterministicTaskQueue.java:147)
        at org.opensearch.cluster.coordination.DeterministicTaskQueue.runRandomTask(DeterministicTaskQueue.java:141)
        at org.opensearch.cluster.coordination.DeterministicTaskQueue.runAllRunnableTasks(DeterministicTaskQueue.java:91)
        at org.opensearch.snapshots.SnapshotResiliencyTests.runUntil(SnapshotResiliencyTests.java:1613)
        at org.opensearch.snapshots.SnapshotResiliencyTests.verifyReposThenStopServices(SnapshotResiliencyTests.java:335)

        Caused by:
        java.lang.StringIndexOutOfBoundsException: Range [13, 2) out of bounds for length 6
            at java.****/jdk.internal.util.Preconditions$1.apply(Preconditions.java:55)
            at java.****/jdk.internal.util.Preconditions$1.apply(Preconditions.java:52)
            at java.****/jdk.internal.util.Preconditions$4.apply(Preconditions.java:213)
            at java.****/jdk.internal.util.Preconditions$4.apply(Preconditions.java:210)
            at java.****/jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:98)
            at java.****/jdk.internal.util.Preconditions.outOfBoundsCheckFromToIndex(Preconditions.java:112)
            at java.****/jdk.internal.util.Preconditions.checkFromToIndex(Preconditions.java:349)
            at java.****/java.lang.String.checkBoundsBeginEnd(String.java:4865)
            at java.****/java.lang.String.substring(String.java:2834)
            at org.opensearch.repositories.blobstore.BlobStoreRepository.lambda$executeOneStaleIndexDelete$40(BlobStoreRepository.java:1658)
            ... 9 more

Test failure opensearch-project#8771 shows cases where certain random seeds trigger this case. The bug is clear: the substring() call should happen after the startsWith() check in case the blob name is shorter than the prefix length being used as the start index of the substring call. I don't yet know if/how this manifests in real deployments. Signed-off-by: Andrew Ross <andrross@amazon.com>

* Avoid string out of bounds error in snapshot delete Test failure #8771 shows cases where certain random seeds trigger this case. The bug is clear: the substring() call should happen after the startsWith() check in case the blob name is shorter than the prefix length being used as the start index of the substring call. I don't yet know if/how this manifests in real deployments. Signed-off-by: Andrew Ross <andrross@amazon.com> * Extract common UUID parsing method Signed-off-by: Andrew Ross <andrross@amazon.com> --------- Signed-off-by: Andrew Ross <andrross@amazon.com>

* Avoid string out of bounds error in snapshot delete Test failure #8771 shows cases where certain random seeds trigger this case. The bug is clear: the substring() call should happen after the startsWith() check in case the blob name is shorter than the prefix length being used as the start index of the substring call. I don't yet know if/how this manifests in real deployments. Signed-off-by: Andrew Ross <andrross@amazon.com> * Extract common UUID parsing method Signed-off-by: Andrew Ross <andrross@amazon.com> --------- Signed-off-by: Andrew Ross <andrross@amazon.com> (cherry picked from commit f86bfa8) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…ct#12337) * Avoid string out of bounds error in snapshot delete Test failure opensearch-project#8771 shows cases where certain random seeds trigger this case. The bug is clear: the substring() call should happen after the startsWith() check in case the blob name is shorter than the prefix length being used as the start index of the substring call. I don't yet know if/how this manifests in real deployments. Signed-off-by: Andrew Ross <andrross@amazon.com> * Extract common UUID parsing method Signed-off-by: Andrew Ross <andrross@amazon.com> --------- Signed-off-by: Andrew Ross <andrross@amazon.com>

* Avoid string out of bounds error in snapshot delete Test failure #8771 shows cases where certain random seeds trigger this case. The bug is clear: the substring() call should happen after the startsWith() check in case the blob name is shorter than the prefix length being used as the start index of the substring call. I don't yet know if/how this manifests in real deployments. * Extract common UUID parsing method --------- (cherry picked from commit f86bfa8) Signed-off-by: Andrew Ross <andrross@amazon.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…ct#12337) * Avoid string out of bounds error in snapshot delete Test failure opensearch-project#8771 shows cases where certain random seeds trigger this case. The bug is clear: the substring() call should happen after the startsWith() check in case the blob name is shorter than the prefix length being used as the start index of the substring call. I don't yet know if/how this manifests in real deployments. Signed-off-by: Andrew Ross <andrross@amazon.com> * Extract common UUID parsing method Signed-off-by: Andrew Ross <andrross@amazon.com> --------- Signed-off-by: Andrew Ross <andrross@amazon.com>

…ct#12337) * Avoid string out of bounds error in snapshot delete Test failure opensearch-project#8771 shows cases where certain random seeds trigger this case. The bug is clear: the substring() call should happen after the startsWith() check in case the blob name is shorter than the prefix length being used as the start index of the substring call. I don't yet know if/how this manifests in real deployments. Signed-off-by: Andrew Ross <andrross@amazon.com> * Extract common UUID parsing method Signed-off-by: Andrew Ross <andrross@amazon.com> --------- Signed-off-by: Andrew Ross <andrross@amazon.com> Signed-off-by: Shivansh Arora <hishiv@amazon.com>

mch2 added bug Something isn't working untriaged flaky-test Random test failure that succeeds on second run distributed framework and removed untriaged labels Jul 19, 2023

kotwanikunal mentioned this issue Jul 20, 2023

[AUTOCUT] Gradle Check Failure on push to 2.x #8787

Closed

RS146BIJAY mentioned this issue Sep 6, 2023

[Backport 2.x] Cleanup Unreferenced file on segment merge failure (#9503) #9800

Closed

6 tasks

reta mentioned this issue Sep 11, 2023

[AUTOCUT] Gradle Check Failure on push to main #9977

Closed

dblock mentioned this issue Oct 3, 2023

[AUTOCUT] Gradle Check Failure on push to main #10318

Closed

reta mentioned this issue Oct 13, 2023

[AUTOCUT] Gradle Check Failure on push to 2.11 #10477

Closed

RamakrishnaChilaka assigned yigithub Oct 24, 2023

reta changed the title ~~[Flaky Test] SnapshotResiliencyTests.testConcurrentSnapshotDeleteAndDeleteIndex on merge to 2.x~~ [Flaky Test] SnapshotResiliencyTests.testConcurrentSnapshotDeleteAndDeleteIndex if flaky Oct 27, 2023

This was referenced Oct 27, 2023

[Backport 2.x] Add tracing instrumentation for indexing paths #10962

Merged

[BWC and API enforcement] Decorate the existing APIs with proper annotations (part 2) #10950

Merged

Poojita-Raj mentioned this issue Nov 6, 2023

[2.x] Revert "Add cluster setting cluster.restrict.index.replication_type t… #11073

Merged

8 tasks

jed326 mentioned this issue Nov 7, 2023

[Backport 2.x] Disable concurrent aggs for Diversified Sampler and Sampler aggs (#11087) #11105

Merged

8 tasks

yigithub added this to Concurrent Search Nov 8, 2023

github-project-automation bot moved this to Todo in Concurrent Search Nov 8, 2023

yigithub removed this from Concurrent Search Nov 8, 2023

dreamer-89 mentioned this issue Jan 2, 2024

[AUTOCUT] Gradle Check Failure on push to 2.x #11712

Closed

reta mentioned this issue Jan 29, 2024

Bump com.gradle.enterprise from 3.16.1 to 3.16.2 #12056

Merged

8 tasks

reta mentioned this issue Feb 12, 2024

Add support of special WrappingSearchAsyncActionPhase so the onPhaseStart() will always be followed by onPhaseEnd() within AbstractSearchAsyncAction #12293

Merged

8 tasks

andrross added Storage:Snapshots and removed distributed framework labels Feb 15, 2024

andrross mentioned this issue Feb 15, 2024

Avoid string out of bounds error in snapshot delete #12337

Merged

8 tasks

Bukhtawar added this to Storage Project Board Feb 15, 2024

github-project-automation bot moved this to 🆕 New in Storage Project Board Feb 15, 2024

andrross closed this as completed in #12337 Feb 19, 2024

github-project-automation bot moved this from 🆕 New to ✅ Done in Storage Project Board Feb 19, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Flaky Test] SnapshotResiliencyTests.testConcurrentSnapshotDeleteAndDeleteIndex if flaky #8771

[Flaky Test] SnapshotResiliencyTests.testConcurrentSnapshotDeleteAndDeleteIndex if flaky #8771

mch2 commented Jul 19, 2023

dblock commented Oct 3, 2023

andrross commented Feb 15, 2024

[Flaky Test] SnapshotResiliencyTests.testConcurrentSnapshotDeleteAndDeleteIndex if flaky #8771

[Flaky Test] SnapshotResiliencyTests.testConcurrentSnapshotDeleteAndDeleteIndex if flaky #8771

Comments

mch2 commented Jul 19, 2023

dblock commented Oct 3, 2023

andrross commented Feb 15, 2024