SearchableSnapshotsIntegTests.testMountedSnapshotHasNoReplicasByDefault failures on CI

[gwbrown]

Build scan: https://gradle-enterprise.elastic.co/s/budltddhs37os

Repro line: gradlew ':x-pack:plugin:searchable-snapshots:internalClusterTest' --tests "org.elasticsearch.xpack.searchablesnapshots.SearchableSnapshotsIntegTests.testMountedSnapshotHasNoReplicasByDefault" -Dtests.seed=EF7E78C79AB95B2C -Dtests.security.manager=true -Dtests.locale=en-IE -Dtests.timezone=America/Argentina/Cordoba -Druntime.java=11

Reproduces locally?: No

Applicable branches: master

Failure history: build stats
Failed 3 times in the past week.

Failure excerpt:

Failure stack trace:

com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=303, name=elasticsearch[node_s0][generic][T#5], state=RUNNABLE, group=TGRP-SearchableSnapshotsIntegTests]	
	at __randomizedtesting.SeedInfo.seed([EF7E78C79AB95B2C:8F1FF900D57AFBC]:0)	
Caused by: java.lang.AssertionError: shard eviction should be successful: [snapshotUUID=xqd-Lpa-Sz-VgqFdnvwe9Q, snapshotIndexName=gzzohvcxmw, shardId=[abvmnzglrp][7]]	
	at __randomizedtesting.SeedInfo.seed([EF7E78C79AB95B2C]:0)	
	at org.elasticsearch.xpack.searchablesnapshots.cache.CacheService.runIfShardMarkedAsEvictedInCache(CacheService.java:412)	
	at org.elasticsearch.xpack.searchablesnapshots.cache.CacheService$1.doRun(CacheService.java:351)	
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)	
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)	
	at java.util.concurrent.FutureTask.run(FutureTask.java:264)	
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:680)	
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)	
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)	
	at java.lang.Thread.run(Thread.java:834)

Also of interest in the log:

[2020-12-18T17:30:58,302][INFO ][o.e.c.m.MetadataDeleteIndexService] [node_s0] [abvmnzglrp/NhRyIzgXRPeH-OeBmDjm_g] deleting index	
Dec 18, 2020 5:30:58 PM com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException	
WARNING: Uncaught exception in thread: Thread[elasticsearch[node_s0][generic][T#5],5,TGRP-SearchableSnapshotsIntegTests]	
java.lang.AssertionError: shard eviction should be successful: [snapshotUUID=xqd-Lpa-Sz-VgqFdnvwe9Q, snapshotIndexName=gzzohvcxmw, shardId=[abvmnzglrp][7]]	
	at __randomizedtesting.SeedInfo.seed([EF7E78C79AB95B2C]:0)	
	at org.elasticsearch.xpack.searchablesnapshots.cache.CacheService.runIfShardMarkedAsEvictedInCache(CacheService.java:412)	
	at org.elasticsearch.xpack.searchablesnapshots.cache.CacheService$1.doRun(CacheService.java:351)	
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)	
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)	
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)	
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:680)	
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)	
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)	
	at java.base/java.lang.Thread.run(Thread.java:834)	

[elasticmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[dimitris-athanasiou]

Another failure of this exact test in https://gradle-enterprise.elastic.co/s/v5z2xk3o5wdxo

[javanna]

Another one: https://gradle-enterprise.elastic.co/s/giq2coadc6q2k

[tlrx]

The shard eviction mechanism has been improved in #67160 and the failing assertion removed in #67265. This should now be fixed, hence I'm closing this issue.