[CI] SearchableSnapshotEnableAllocationDeciderIntegTests.testAllocationDisabled assertion failed

[dimitris-athanasiou]

This test failed in a windows-2016 compatibility run. An assertion was violated and I thought it makes sense to take a look at this.

Build scan: https://gradle-enterprise.elastic.co/s/ke3iweqi5n5bc

Repro line:

gradlew ':x-pack:plugin:searchable-snapshots:internalClusterTest' --tests "org.elasticsearch.xpack.searchablesnapshots.SearchableSnapshotEnableAllocationDeciderIntegTests.testAllocationDisabled" -Dtests.seed=8354D31B87984E3F -Dtests.security.manager=true -Dtests.locale=es-PA -Dtests.timezone=America/Nuuk -Druntime.java=8

Reproduces locally?: No

Applicable branches: master

Failure history: There are 4 failures in total in the past few days, always on windows. See here

Failure excerpt:

[2020-12-21T04:13:14,949][INFO ][o.e.c.m.MetadataDeleteIndexService] [node_t1] [itidyhnpzh/gm87dpzEQhugOvR0KxfDeQ] deleting index
dic 21, 2020 4:13:15 AM com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException
ADVERTENCIA: Uncaught exception in thread: Thread[elasticsearch[node_t0][generic][T#3],5,TGRP-SearchableSnapshotEnableAllocationDeciderIntegTests]
java.lang.AssertionError: shard eviction should be successful: [snapshotUUID=gTwLcnN8RBSHOdLTsl5E_g, snapshotIndexName=vegmixdrat, shardId=[itidyhnpzh][8]]
	at __randomizedtesting.SeedInfo.seed([8354D31B87984E3F]:0)
	at org.elasticsearch.xpack.searchablesnapshots.cache.CacheService.runIfShardMarkedAsEvictedInCache(CacheService.java:412)
	at org.elasticsearch.xpack.searchablesnapshots.cache.CacheService.access$100(CacheService.java:62)
	at org.elasticsearch.xpack.searchablesnapshots.cache.CacheService$1.doRun(CacheService.java:351)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:684)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

[elasticmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[dimitris-athanasiou]

Similar assertion error in different test: https://gradle-enterprise.elastic.co/s/ngd4eact5bofo

Test: SearchableSnapshotsPrewarmingIntegTest.testConcurrentPrewarming

[tlrx]

@original-brownbear If you don't mind I'm taking this one as it is related to a change I made.

[tlrx]

The shard eviction mechanism has been improved in #67160 and the failing assertion removed in #67265, this should now be fixed so I'm closing this issue.