[CI] IndexShardIT.testShardHasMemoryBufferOnTranslogRecover failure

[romseygeek]

This reproduces on master:

REPRODUCE WITH: ./gradlew :server:integTest \
  -Dtests.seed=9EC7D98FCA15BCE6 \
  -Dtests.class=org.elasticsearch.index.shard.IndexShardIT \
  -Dtests.method="testShardHasMemoryBufferOnTranslogRecover" \
  -Dtests.security.manager=true \
  -Dtests.locale=sv \
  -Dtests.timezone=Pacific/Auckland \
  -Dcompiler.java=11 \
  -Druntime.java=8

The resulting error is:

ERROR   4.25s | IndexShardIT.testShardHasMemoryBufferOnTranslogRecover <<< FAILURES!
   > Throwable #1: [test/rCQ777wBRcmIfm1spp2jiA][[test][0]] IndexShardRecoveryException[failed to recover from gateway]; nested: EngineCreationFailureException[failed to create engine]; nested: NoSuchFileException[file "_0.cfs" is already pending delete];
   >    at __randomizedtesting.SeedInfo.seed([9EC7D98FCA15BCE6:42E398DA205C6200]:0)
   >    at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:429)
   >    at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromStore$0(StoreRecovery.java:95)
   >    at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:302)
   >    at org.elasticsearch.index.shard.StoreRecovery.recoverFromStore(StoreRecovery.java:93)
   >    at org.elasticsearch.index.shard.IndexShard.recoverFromStore(IndexShard.java:1657)
   >    at org.elasticsearch.index.shard.IndexShardIT.recoverShard(IndexShardIT.java:636)
   >    at org.elasticsearch.index.shard.IndexShardIT.testShardHasMemoryBufferOnTranslogRecover(IndexShardIT.java:560)
   >    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   >    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   >    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   >    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
   >    at java.base/java.lang.Thread.run(Thread.java:834)
   > Caused by: [test/rCQ777wBRcmIfm1spp2jiA][[test][0]] EngineCreationFailureException[failed to create engine]; nested: NoSuchFileException[file "_0.cfs" is already pending delete];
   >    at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:204)
   >    at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:165)
   >    at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:25)
   >    at org.elasticsearch.index.shard.IndexShard.innerOpenEngineAndTranslog(IndexShard.java:1418)
   >    at org.elasticsearch.index.shard.IndexShard.openEngineAndRecoverFromTranslog(IndexShard.java:1372)
   >    at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:424)
   >    ... 42 more
   > Caused by: java.nio.file.NoSuchFileException: file "_0.cfs" is already pending delete
   >    at org.apache.lucene.store.FSDirectory.deleteFile(FSDirectory.java:338)
   >    at org.apache.lucene.store.FileSwitchDirectory.deleteFile(FileSwitchDirectory.java:151)
   >    at org.apache.lucene.store.FilterDirectory.deleteFile(FilterDirectory.java:63)
   >    at org.elasticsearch.index.store.ByteSizeCachingDirectory.deleteFile(ByteSizeCachingDirectory.java:175)
   >    at org.apache.lucene.store.FilterDirectory.deleteFile(FilterDirectory.java:63)
   >    at org.elasticsearch.index.store.Store$StoreDirectory.deleteFile(Store.java:721)
   >    at org.elasticsearch.index.store.Store$StoreDirectory.deleteFile(Store.java:726)
   >    at org.apache.lucene.store.LockValidatingDirectoryWrapper.deleteFile(LockValidatingDirectoryWrapper.java:38)
   >    at org.apache.lucene.index.IndexFileDeleter.deleteFile(IndexFileDeleter.java:696)
   >    at org.apache.lucene.index.IndexFileDeleter.deleteFiles(IndexFileDeleter.java:690)
   >    at org.apache.lucene.index.IndexFileDeleter.<init>(IndexFileDeleter.java:238)
   >    at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:898)
   >    at org.elasticsearch.index.engine.InternalEngine$AssertingIndexWriter.<init>(InternalEngine.java:2580)
   >    at org.elasticsearch.index.engine.InternalEngine.createWriter(InternalEngine.java:2107)
   >    at org.elasticsearch.index.engine.InternalEngine.createWriter(InternalEngine.java:2097)
   >    at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:199)
   >    ... 47 more

FileSwitchDirectory makes me think this is related to the addition of hybridfs, especially given that this only started failing yesterday afternoon, after #36668 was merged.

[elasticmachine]

Pinging @elastic/es-distributed