[CI] ReactiveStorageIT testScaleDuringSplitOrClone failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-periodic #8779 / openjdk21_checkpart2_java-matrix
• elasticsearch-periodic #8743 / graalvm-ce17_checkpart2_java-matrix
• elasticsearch-pull-request #74672 / part-2

Reproduction Line:

./gradlew ":x-pack:plugin:autoscaling:internalClusterTest" --tests "org.elasticsearch.xpack.autoscaling.storage.ReactiveStorageIT.testScaleDuringSplitOrClone" -Dtests.seed=3D7A1A66F2013ACD -Dtests.locale=mzn-Arab-IR -Dtests.timezone=Asia/Tomsk -Druntime.java=21

Applicable branches:
8.19

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.lang.IllegalStateException: Some shards are still open after the threadpool terminated. Something is leaking index readers or store references.

Issue Reasons:

• [8.19] 3 failures in test testScaleDuringSplitOrClone (0.4% fail rate in 764 executions)
• [8.19] 2 failures in pipeline elasticsearch-periodic (10.0% fail rate in 20 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

This has been muted on branch main

Mute Reasons:

• [main] 3 failures in test testScaleDuringSplitOrClone (1.0% fail rate in 293 executions)
• [main] 2 failures in step openjdk21_checkpart2_java-fips-matrix (33.3% fail rate in 6 executions)
• [main] 2 failures in pipeline elasticsearch-periodic (33.3% fail rate in 6 executions)

Build Scans:

• elasticsearch-periodic #8715 / openjdk21_checkpart2_java-fips-matrix
• elasticsearch-pull-request #75147 / part-2
• elasticsearch-periodic #8685 / openjdk21_checkpart2_java-fips-matrix

[elasticsearchmachine]

Pinging @elastic/es-distributed-coordination (Team:Distributed Coordination)

[DaveCTurner]

I think this one's quite serious, it suggests we're failing to shut down shards properly. It reproduces within a few (<100) iterations on a CPU-stressed machine. I'm trying to pin down the commit where it started failing.

[DaveCTurner]

My earlier estimate of the ease of reproduction was a little wrong, I've been running various different commits and some of them seemed to fail quickly while others passed for many attempts. The earliest commit on which I've seen this fail so far is 8a92b460934a7a249546aafefc5440e5362780dd which failed on the 101st iteration of ./gradlew :x-pack:plugin:autoscaling:internalClusterTest --tests org.elasticsearch.xpack.autoscaling.storage.ReactiveStorageIT.testScaleDuringSplitOrClone -Dtests.iters=20 -Dtests.failfast=true (so just over 2k iterations). I don't think this is the culprit, but I am suspicious it might be #127613 which merged only a couple of commits earlier.

[DaveCTurner]

Ok can confirm that this reproduces on 53f3ab2, i.e. the commit to which #127613 was merged. I'm now trying its parent 95a1b4b.