SearchableSnapshotsPersistentCacheIntegTests. testCacheSurviveRestart failure

[benwtrent]

Build scan:
https://gradle-enterprise.elastic.co/s/xy7c3vznahvoc
Repro line:

./gradlew ':x-pack:plugin:searchable-snapshots:internalClusterTest' --tests "org.elasticsearch.xpack.searchablesnapshots.cache.SearchableSnapshotsPersistentCacheIntegTests.testCacheSurviveRestart" -Dtests.seed=9F0422EA69B04B29 -Dtests.security.manager=true -Dtests.locale=ga -Dtests.timezone=Asia/Yakutsk -Druntime.java=11

Reproduces locally?:
No, ran it with the same seed, and many different seeds but with other params being the same
Applicable branches:
master
Failure excerpt:

org.elasticsearch.xpack.searchablesnapshots.cache.SearchableSnapshotsPersistentCacheIntegTests > testCacheSurviveRestart FAILED |  
-- | --
  | java.lang.AssertionError: |  
  | Expected: <0L> |  
  | but: was <12L> |  
  | at __randomizedtesting.SeedInfo.seed([9F0422EA69B04B29:851033F89C267B6E]:0) |  
  | at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18) |  
  | at org.junit.Assert.assertThat(Assert.java:956) |  
  | at org.junit.Assert.assertThat(Assert.java:923) |  
  | at org.elasticsearch.xpack.searchablesnapshots.cache.SearchableSnapshotsPersistentCacheIntegTests.testCacheSurviveRestart(SearchableSnapshotsPersistentCacheIntegTests.java:155)

[elasticmachine]

Pinging @elastic/es-search (Team:Search)

[benwtrent]

Another failure: https://gradle-enterprise.elastic.co/s/jpxhkvlcqq3ze/console-log?task=:x-pack:plugin:searchable-snapshots:internalClusterTest

org.elasticsearch.xpack.searchablesnapshots.cache.SearchableSnapshotsPersistentCacheIntegTests > testCacheSurviveRestart FAILED |  
-- | --
  | java.lang.AssertionError: |  
  | Expected: <0L> |  
  | but: was <10L> |  
  | at __randomizedtesting.SeedInfo.seed([98F3DDD081D6BC00:82E7CCC274408C47]:0) |  
  | at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18) |  
  | at org.junit.Assert.assertThat(Assert.java:956) |  
  | at org.junit.Assert.assertThat(Assert.java:923)

[tlrx]

It will be muted in #66289

[elasticmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[tvernum]

This failed again (sorry @tlrx)

https://gradle-enterprise.elastic.co/s/7vw5xavdodpwg/tests/:x-pack:plugin:searchable-snapshots:internalClusterTest/org.elasticsearch.xpack.searchablesnapshots.cache.SearchableSnapshotsPersistentCacheIntegTests/testCacheSurviveRestart#1

[tlrx]

Thanks @tvernum for reopening, and sorry for the noise. I opened another fix.

[costin]

Another instance at
https://gradle-enterprise.elastic.co/s/6l445sanhzsty

[tvernum]

This is still failing, and it looks like there's an open PR but it has stalled (due to holidays?)

• Recent Failure
• History
• Stalled PR

Is the PR likely to pick up again, or should we mute?

[martijnvg]

Another instance of this failure that I ran into in pr CI for #66178:
https://gradle-enterprise.elastic.co/s/2h3uxcs2ta6zy/tests/:x-pack:plugin:searchable-snapshots:internalClusterTest/org.elasticsearch.xpack.searchablesnapshots.cache.SearchableSnapshotsPersistentCacheIntegTests/testCacheSurviveRestart#1

[tlrx]

The shard eviction mechanism has been improved in #67160, which should in turn fix this issue.