Reproducible Failure in org.elasticsearch.index.store.SearchableSnapshotDirectoryStatsTests.testCachedBytesReadsAndWrites #70000
Description
Just ran into this locally working on the multiple page sizes cache but it reproduces on master as well:
./gradlew ':x-pack:plugin:searchable-snapshots:test' --tests "org.elasticsearch.index.store.SearchableSnapshotDirectoryStatsTests.testCachedBytesReadsAndWrites" -Dtests.seed=163C909BF9F0F844 -Dtests.security.manager=true -Dtests.locale=sk-SK -Dtests.timezone=Etc/UTC -Druntime.java=15
fails with
[2021-03-04T20:58:27,650][INFO ][o.e.i.s.SearchableSnapshotDirectoryStatsTests] [testCachedBytesReadsAndWrites] before test
[2021-03-04T20:58:27,696][INFO ][o.e.e.NodeEnvironment ] [testCachedBytesReadsAndWrites] using [3] data paths, mounts [[/ (/dev/nvme0n1p2)]], net usable_space [121.6gb], net total_space [467.9gb], types [ext4]
[2021-03-04T20:58:27,697][INFO ][o.e.e.NodeEnvironment ] [testCachedBytesReadsAndWrites] heap size [15.6gb], compressed ordinary object pointers [true]
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.mockito.cglib.core.ReflectUtils$2 (file:/home/brownbear/.gradle/caches/modules-2/files-2.1/org.elasticsearch/securemock/1.2/98201d4ad5ac93f6b415ae9172d52b5e7cda490e/securemock-1.2.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
WARNING: Please consider reporting this to the maintainers of org.mockito.cglib.core.ReflectUtils$2
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
[2021-03-04T20:58:27,922][INFO ][o.e.x.s.c.PersistentCache] [testCachedBytesReadsAndWrites] persistent cache index loaded
[2021-03-04T20:58:38,115][INFO ][o.e.i.s.SearchableSnapshotDirectoryStatsTests] [testCachedBytesReadsAndWrites] after test
REPRODUCE WITH: ./gradlew 'null' --tests "org.elasticsearch.index.store.SearchableSnapshotDirectoryStatsTests.testCachedBytesReadsAndWrites" -Dtests.seed=163C909BF9F0F844 -Dtests.locale=sk-SK -Dtests.timezone=Etc/UTC -Druntime.java=15
java.lang.AssertionError:
Expected: <9641L>
but: was <9885L>
Expected :<9641L>
Actual :<9885L>
<Click to see difference>
at __randomizedtesting.SeedInfo.seed([163C909BF9F0F844:D38D21E9FBFE3FD4]:0)
at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
at org.junit.Assert.assertThat(Assert.java:956)
at org.junit.Assert.assertThat(Assert.java:923)
at org.elasticsearch.index.store.SearchableSnapshotDirectoryStatsTests.lambda$testCachedBytesReadsAndWrites$2(SearchableSnapshotDirectoryStatsTests.java:131)
at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:955)
at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:928)
at org.elasticsearch.index.store.SearchableSnapshotDirectoryStatsTests.lambda$testCachedBytesReadsAndWrites$3(SearchableSnapshotDirectoryStatsTests.java:129)
at org.elasticsearch.index.store.SearchableSnapshotDirectoryStatsTests.executeTestCase(SearchableSnapshotDirectoryStatsTests.java:683)
at org.elasticsearch.index.store.SearchableSnapshotDirectoryStatsTests.executeTestCaseWithCache(SearchableSnapshotDirectoryStatsTests.java:588)
at org.elasticsearch.index.store.SearchableSnapshotDirectoryStatsTests.testCachedBytesReadsAndWrites(SearchableSnapshotDirectoryStatsTests.java:116)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:64)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:375)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:824)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:475)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:375)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:831)
at java.base/java.lang.Thread.run(Thread.java:832)
Suppressed: java.lang.AssertionError:
Expected: <9641L>
but: was <9885L>
at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
at org.junit.Assert.assertThat(Assert.java:956)
at org.junit.Assert.assertThat(Assert.java:923)
at org.elasticsearch.index.store.SearchableSnapshotDirectoryStatsTests.lambda$testCachedBytesReadsAndWrites$2(SearchableSnapshotDirectoryStatsTests.java:131)
at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:943)
... 42 more
Suppressed: java.lang.AssertionError:
Expected: <9641L>
but: was <9885L>
at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
at org.junit.Assert.assertThat(Assert.java:956)
at org.junit.Assert.assertThat(Assert.java:923)
at org.elasticsearch.index.store.SearchableSnapshotDirectoryStatsTests.lambda$testCachedBytesReadsAndWrites$2(SearchableSnapshotDirectoryStatsTests.java:131)
at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:943)
... 42 more
Suppressed: java.lang.AssertionError:
on current master.