IndexStatsIT#testFilterCacheStats fails

[dnhatn]

I think this failure relates to Lucene-7.5.

I can't reproduce this locally with this command:

./gradlew :server:integTest -Dtests.seed=528FC320ADC916D8 \
-Dtests.class=org.elasticsearch.indices.stats.IndexStatsIT \
-Dtests.method="testFilterCacheStats" \
-Dtests.security.manager=true \
-Dtests.locale=kk-KZ \
-Dtests.timezone=America/Scoresbysund

FAILURE 0.63s J0 | IndexStatsIT.testFilterCacheStats <<< FAILURES!
   > Throwable #1: java.lang.AssertionError: 
   > Expected: a value greater than <0L>
   >      but: <0L> was equal to <0L>
   > 	at __randomizedtesting.SeedInfo.seed([528FC320ADC916D8:37B03E651DC111A1]:0)
   > 	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
   > 	at org.elasticsearch.indices.stats.IndexStatsIT.testFilterCacheStats(IndexStatsIT.java:1046)
   > 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   > 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   > 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   > 	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
   > 	at java.base/java.lang.Thread.run(Thread.java:834)

CI: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+matrix-java-periodic/ES_BUILD_JAVA=java10,ES_RUNTIME_JAVA=java11,nodes=virtual&&linux/210/consoleText

Log: testFilterCacheStats.log

[elasticmachine]

Pinging @elastic/es-distributed

[dnhatn]

I muted this test on the master and 6.x.

[original-brownbear]

I was able to reproduce this locally by simply running the test in a loop with @Repeat (fails in ~ 1% of runs)

[dnhatn]

@original-brownbear I did not look at this yet but I think it should be caused by a background merge.

[original-brownbear]

@dnhatn I spent a little time today reading up on the background here and using trace logging I could verify that this is caused by background merges.
Tried a fix in #36143 that empirically works fine, but I'm not a 100% sure that's the best/right approach at fixing this admittedly.