Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Reproducible Failure in org.elasticsearch.index.store.SearchableSnapshotDirectoryStatsTests.testCachedBytesReadsAndWrites #70000

Closed
original-brownbear opened this issue Mar 4, 2021 · 1 comment · Fixed by #70006
Closed

Reproducible Failure in org.elasticsearch.index.store.SearchableSnapshotDirectoryStatsTests.testCachedBytesReadsAndWrites #70000

original-brownbear opened this issue Mar 4, 2021 · 1 comment · Fixed by #70006
Assignees
@tlrx
Labels
:Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination. >test-failure Triaged test failures from CI

Comments

@original-brownbear
Copy link
Member

original-brownbear commented Mar 4, 2021 •
edited
Loading

Just ran into this locally working on the multiple page sizes cache but it reproduces on master as well:

./gradlew ':x-pack:plugin:searchable-snapshots:test' --tests "org.elasticsearch.index.store.SearchableSnapshotDirectoryStatsTests.testCachedBytesReadsAndWrites" -Dtests.seed=163C909BF9F0F844 -Dtests.security.manager=true -Dtests.locale=sk-SK -Dtests.timezone=Etc/UTC -Druntime.java=15

fails with

[2021-03-04T20:58:27,650][INFO ][o.e.i.s.SearchableSnapshotDirectoryStatsTests] [testCachedBytesReadsAndWrites] before test
[2021-03-04T20:58:27,696][INFO ][o.e.e.NodeEnvironment    ] [testCachedBytesReadsAndWrites] using [3] data paths, mounts [[/ (/dev/nvme0n1p2)]], net usable_space [121.6gb], net total_space [467.9gb], types [ext4]
[2021-03-04T20:58:27,697][INFO ][o.e.e.NodeEnvironment    ] [testCachedBytesReadsAndWrites] heap size [15.6gb], compressed ordinary object pointers [true]
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.mockito.cglib.core.ReflectUtils$2 (file:/home/brownbear/.gradle/caches/modules-2/files-2.1/org.elasticsearch/securemock/1.2/98201d4ad5ac93f6b415ae9172d52b5e7cda490e/securemock-1.2.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
WARNING: Please consider reporting this to the maintainers of org.mockito.cglib.core.ReflectUtils$2
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
[2021-03-04T20:58:27,922][INFO ][o.e.x.s.c.PersistentCache] [testCachedBytesReadsAndWrites] persistent cache index loaded
[2021-03-04T20:58:38,115][INFO ][o.e.i.s.SearchableSnapshotDirectoryStatsTests] [testCachedBytesReadsAndWrites] after test
REPRODUCE WITH: ./gradlew 'null' --tests "org.elasticsearch.index.store.SearchableSnapshotDirectoryStatsTests.testCachedBytesReadsAndWrites" -Dtests.seed=163C909BF9F0F844 -Dtests.locale=sk-SK -Dtests.timezone=Etc/UTC -Druntime.java=15

java.lang.AssertionError: 
Expected: <9641L>
     but: was <9885L>
Expected :<9641L>
Actual   :<9885L>
<Click to see difference>


	at __randomizedtesting.SeedInfo.seed([163C909BF9F0F844:D38D21E9FBFE3FD4]:0)
	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
	at org.junit.Assert.assertThat(Assert.java:956)
	at org.junit.Assert.assertThat(Assert.java:923)
	at org.elasticsearch.index.store.SearchableSnapshotDirectoryStatsTests.lambda$testCachedBytesReadsAndWrites$2(SearchableSnapshotDirectoryStatsTests.java:131)
	at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:955)
	at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:928)
	at org.elasticsearch.index.store.SearchableSnapshotDirectoryStatsTests.lambda$testCachedBytesReadsAndWrites$3(SearchableSnapshotDirectoryStatsTests.java:129)
	at org.elasticsearch.index.store.SearchableSnapshotDirectoryStatsTests.executeTestCase(SearchableSnapshotDirectoryStatsTests.java:683)
	at org.elasticsearch.index.store.SearchableSnapshotDirectoryStatsTests.executeTestCaseWithCache(SearchableSnapshotDirectoryStatsTests.java:588)
	at org.elasticsearch.index.store.SearchableSnapshotDirectoryStatsTests.testCachedBytesReadsAndWrites(SearchableSnapshotDirectoryStatsTests.java:116)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:64)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:564)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:375)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:824)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:475)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:375)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:831)
	at java.base/java.lang.Thread.run(Thread.java:832)
	Suppressed: java.lang.AssertionError: 
Expected: <9641L>
     but: was <9885L>
		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
		at org.junit.Assert.assertThat(Assert.java:956)
		at org.junit.Assert.assertThat(Assert.java:923)
		at org.elasticsearch.index.store.SearchableSnapshotDirectoryStatsTests.lambda$testCachedBytesReadsAndWrites$2(SearchableSnapshotDirectoryStatsTests.java:131)
		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:943)
		... 42 more
	Suppressed: java.lang.AssertionError: 
Expected: <9641L>
     but: was <9885L>
		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
		at org.junit.Assert.assertThat(Assert.java:956)
		at org.junit.Assert.assertThat(Assert.java:923)
		at org.elasticsearch.index.store.SearchableSnapshotDirectoryStatsTests.lambda$testCachedBytesReadsAndWrites$2(SearchableSnapshotDirectoryStatsTests.java:131)
		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:943)
		... 42 more
	Suppressed: java.lang.AssertionError:

on current master.
@original-brownbear original-brownbear added :Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs >test-failure Triaged test failures from CI labels Mar 4, 2021
@elasticmachine elasticmachine added the Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination. label Mar 4, 2021
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-distributed (Team:Distributed)

tlrx added a commit to tlrx/elasticsearch that referenced this issue Mar 5, 2021
@tlrx
Exclude partially cached .cfs file from SearchableSnapshotDirectorySt…
9d075e1 
…atsTests

Closes elastic#70000
@tlrx tlrx mentioned this issue Mar 5, 2021
Merged
tlrx added a commit that referenced this issue Mar 5, 2021
@tlrx
Exclude partially cached .cfs file from SearchableSnapshotDirectorySt…
b3fd16d 
…atsTests (#70006)

Since #69861 CFS files read from FrozenIndexInput create 
dedicated frozen shared cache files when they are sliced. 
This does not play well with some tests that use the 
randomReadAndSlice to read files: this method can create 
overlapping slice/clone reads operations which makes it 
difficult to assert anything about CFS files with partial cache.

This commit prevent the tests to generate a .cfs file name 
when the partial cache is randomly picked up. As a follow 
up we should rework those test to make them more realistic 
with the new behavior.

Closes #70000
tlrx added a commit to tlrx/elasticsearch that referenced this issue Mar 5, 2021
@tlrx
Exclude partially cached .cfs file from SearchableSnapshotDirectorySt…
23c59de 
…atsTests (elastic#70006)

Since elastic#69861 CFS files read from FrozenIndexInput create 
dedicated frozen shared cache files when they are sliced. 
This does not play well with some tests that use the 
randomReadAndSlice to read files: this method can create 
overlapping slice/clone reads operations which makes it 
difficult to assert anything about CFS files with partial cache.

This commit prevent the tests to generate a .cfs file name 
when the partial cache is randomly picked up. As a follow 
up we should rework those test to make them more realistic 
with the new behavior.

Closes elastic#70000
@tlrx tlrx mentioned this issue Mar 5, 2021
Merged
tlrx added a commit to tlrx/elasticsearch that referenced this issue Mar 5, 2021
@tlrx
Exclude partially cached .cfs file from SearchableSnapshotDirectorySt…
2e6b9c9 
…atsTests (elastic#70006)

Since elastic#69861 CFS files read from FrozenIndexInput create 
dedicated frozen shared cache files when they are sliced. 
This does not play well with some tests that use the 
randomReadAndSlice to read files: this method can create 
overlapping slice/clone reads operations which makes it 
difficult to assert anything about CFS files with partial cache.

This commit prevent the tests to generate a .cfs file name 
when the partial cache is randomly picked up. As a follow 
up we should rework those test to make them more realistic 
with the new behavior.

Closes elastic#70000
@tlrx tlrx mentioned this issue Mar 5, 2021
Merged
tlrx added a commit that referenced this issue Mar 5, 2021
@tlrx
Exclude partially cached .cfs file from SearchableSnapshotDirectorySt…
dc0f58f 
…atsTests (#70006) (#70019)

Since #69861 CFS files read from FrozenIndexInput create 
dedicated frozen shared cache files when they are sliced. 
This does not play well with some tests that use the 
randomReadAndSlice to read files: this method can create 
overlapping slice/clone reads operations which makes it 
difficult to assert anything about CFS files with partial cache.

This commit prevent the tests to generate a .cfs file name 
when the partial cache is randomly picked up. As a follow 
up we should rework those test to make them more realistic 
with the new behavior.

Closes #70000
tlrx added a commit that referenced this issue Mar 5, 2021
@tlrx
Exclude partially cached .cfs file from SearchableSnapshotDirectorySt…
26fa9d3 
…atsTests (#70006) (#70018)

Since #69861 CFS files read from FrozenIndexInput create 
dedicated frozen shared cache files when they are sliced. 
This does not play well with some tests that use the 
randomReadAndSlice to read files: this method can create 
overlapping slice/clone reads operations which makes it 
difficult to assert anything about CFS files with partial cache.

This commit prevent the tests to generate a .cfs file name 
when the partial cache is randomly picked up. As a follow 
up we should rework those test to make them more realistic 
with the new behavior.

Closes #70000
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tlrx tlrx

Labels
:Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination. >test-failure Triaged test failures from CI
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Exclude partially cached .cfs file from SearchableSnapshotDirectoryStatsTests tlrx/elasticsearch
3 participants
@tlrx @original-brownbear @elasticmachine