[BUG] testCoordinatingPrimaryThreadedUpdateToShardLimitsAndRejections is flaky

[dblock]

Describe the bug

REPRODUCE WITH: ./gradlew ':server:test' --tests "org.opensearch.index.ShardIndexingPressureConcurrentExecutionTests.testCoordinatingPrimaryThreadedUpdateToShardLimitsAndRejections" -Dtests.seed=3AA0904AC11EAEA5 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=uk-UA -Dtests.timezone=America/Argentina/Jujuy -Druntime.java=19

org.opensearch.index.ShardIndexingPressureConcurrentExecutionTests > testCoordinatingPrimaryThreadedUpdateToShardLimitsAndRejections FAILED
    java.lang.AssertionError: expected null, but was:<org.opensearch.index.stats.IndexingPressurePerShardStats@3a0179b7>
        at __randomizedtesting.SeedInfo.seed([3AA0904AC11EAEA5:BC95A9DD94E04AA8]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.junit.Assert.failNotNull(Assert.java:756)
        at org.junit.Assert.assertNull(Assert.java:738)
        at org.junit.Assert.assertNull(Assert.java:748)
        at org.opensearch.index.ShardIndexingPressureConcurrentExecutionTests.testCoordinatingPrimaryThreadedUpdateToShardLimitsAndRejections(ShardIndexingPressureConcurrentExecutionTests.java:274)

#5143

[andrross]

This is a very similar test and failure as mentioned in #4212. I suspect the fix will be the same for both. I'm capturing the notes I've collected on this so far below:

The test expects shardIndexingPressure.shardStats().getIndexingPressureShardStats(shardId1) to return null.

In theory, this close call should clean up everything being tracked and result in that assertion being true. The only suspect I have is that the close conditionally cleans up the stats being tracked. I suspect there may be a race preventing the values getting to zero tested in that condition. I haven't figured out how that might be possible though.

I can get this to fail locally somewhat reliably. However, if it fails, it always seems to fail on the first attempt. If I run it many times the retries never seem to fail. This suggests to me it is some sort of race condition.

[dblock]

https://build.ci.opensearch.org/job/gradle-check/6762/
https://build.ci.opensearch.org/job/gradle-check/6761/
https://build.ci.opensearch.org/job/gradle-check/6759/

[Rishikesh1159]

Similar to falky test : #4212 , test only fails when rejectionCount is equal to NUM_THREADS. Possible fix/workaround would be similar to this comment. Need @getsaurabh02 opinion on this to move forward with fix.

[getsaurabh02]

thanks @Rishikesh1159 . Could you pls check #4212 (comment)

[Rishikesh1159]

Closing this as #5439 is merged, which fixes the issue. Please feel free to reopen if this test fails again.