Fix #7972 SearchBackpressureIT flaky tests

[stephen-crawford]

Description

This PR is attempt number 2 at remediating the flaky tests in **/SearchBackpressureIT. The two tests focusing on high CPU usage were prone to failure do to thread concurrency issues where the assignment of the CancellableTask values were not synchronized across the nodes.

To resolve the issue, I added in a small static class as suggested by @ketanv3 and used an atomic reference through the SetOnce class. This allows for atomic assignments of the class variables without requiring a synchronized block around the cancel(reason) method.

After the change, I ran a runner for 10 times with no failures so it appears to be fixed.

One note about this test setup: I am not sure we want tests in this vain. The way the tests are made, we end up with non-deterministic behavior. For instance, we set the HIGH_CPU tests to be 1000 iterations, but that is just an arbitrary choice for a iteration count. On a more or less powerful processor this could have different results. This is especially true with different architectures that could have stack unwinding or other features.

Related Issues

Resolves #7972

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/17562/
• CommitID: 2bfe424
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/17563/
• CommitID: ec6582b
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[stephen-crawford]

Failure does not seem related:

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':test:fixtures:krb5kdc-fixture:composeBuild'.
> Exit-code 1 when calling /usr/bin/docker-compose, stdout: Step 1/9 : FROM ubuntu:14.04
  14.04: Pulling from library/ubuntu
  Digest: sha256:64483f3496c1373bfd55348e88694d1c4d0c9b660dee6bfef5e12f43b9933b30
  Status: Downloaded newer image for ubuntu:14.04
   ---> 13b66b487594
  Step 2/9 : RUN apt update -y
   ---> Running in 0ebbcd3251a0
  �[91m
  WARNING: apt does not have a stable CLI interface yet. Use with caution in scripts.
  

[owaiskazi19]

@scrawfor99 can you rebase your branch with latest main and push again?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/17788/
• CommitID: a43920a
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/17789/
• CommitID: 75efaf2
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.indices.replication.SegmentReplicationIT.testScrollCreatedOnReplica

• URL: https://build.ci.opensearch.org/job/gradle-check/17791/
• CommitID: 5b11848
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.snapshots.DedicatedClusterSnapshotRestoreIT.testIndexDeletionDuringSnapshotCreationInQueue
      1 org.opensearch.cluster.allocation.AwarenessAllocationIT.testThreeZoneOneReplicaWithForceZoneValueAndLoadAwareness

• URL: https://build.ci.opensearch.org/job/gradle-check/17795/
• CommitID: 8bea59b
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.index.ShardIndexingPressureIT.testShardIndexingPressureTrackingDuringBulkWrites

• URL: https://build.ci.opensearch.org/job/gradle-check/17797/
• CommitID: a2d68ca
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      3 org.opensearch.remotestore.RemoteStoreRefreshListenerIT.testRemoteRefreshRetryOnFailure
      2 org.opensearch.cluster.service.MasterServiceTests.classMethod
      2 org.opensearch.cluster.service.MasterServiceTests.classMethod
      1 org.opensearch.cluster.service.MasterServiceTests.testThrottlingForMultipleTaskTypes
      1 org.opensearch.cluster.service.MasterServiceTests.testThrottlingForMultipleTaskTypes

• URL: https://build.ci.opensearch.org/job/gradle-check/17796/
• CommitID: a2d68ca
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[stephen-crawford]

@dbwiddis, @reta, or @owaiskazi19 would any of you being able to merge this? Thank you.

[owaiskazi19]

@scrawfor99 do you think we can document the test setup you tried for reproducing flaky tests like somewhere? I don't know the right place probably in TESTING.md. Let's see if folks agree to it.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/17957/
• CommitID: 2dc06ab
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT.testDropPrimaryDuringReplication

• URL: https://build.ci.opensearch.org/job/gradle-check/17958/
• CommitID: 2b96c32
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[stephen-crawford]

Hi @owaiskazi19, I can definitely add something but to be honest it was not very scientific... I use IntelliJ so I just ran the tests a handful of times with the IDE testing framework. @ketanv3 had provided quite a bit of insight here as someone who understands the use case and components around these tests much better than I do.

[stephen-crawford]

@dbwiddis @reta, think we could get this merged? :)