Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Failure in CloseWhileRelocatingShardsIT #49403

Closed
DaveCTurner opened this issue Nov 20, 2019 · 1 comment · Fixed by #49461
Closed

Failure in CloseWhileRelocatingShardsIT #49403

DaveCTurner opened this issue Nov 20, 2019 · 1 comment · Fixed by #49461
Assignees
@dnhatn
Labels
:Distributed Indexing/Recovery Anything around constructing a new shard, either from a local or a remote source. >test-failure Triaged test failures from CI

Comments

@DaveCTurner
Copy link
Contributor

This build failed as follows:

java.lang.AssertionError: [index-2][0], node[47Pd3heUS86aYOtaDIkfdA], relocating [Ei8GUN6gT-mv72m6aEdh7Q], [P], s[RELOCATING], a[id=ON9IncKrSZGA7iNl9csiUw, rId=qjOMwwczSSKTcn38ATktng], expected_shard_size[6238507] has unreleased snapshotted index commits
	at __randomizedtesting.SeedInfo.seed([4E92C49D3105483F:6119DA081E9B1E85]:0)
	at org.junit.Assert.fail(Assert.java:88)
	at org.junit.Assert.assertTrue(Assert.java:41)
	at org.junit.Assert.assertFalse(Assert.java:64)
	at org.elasticsearch.test.InternalTestCluster.lambda$assertNoSnapshottedIndexCommit$10(InternalTestCluster.java:1365)
	at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:891)
	at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:864)
	at org.elasticsearch.test.InternalTestCluster.assertNoSnapshottedIndexCommit(InternalTestCluster.java:1357)
	at org.elasticsearch.test.InternalTestCluster.beforeIndexDeletion(InternalTestCluster.java:1287)
	at org.elasticsearch.test.ESIntegTestCase.beforeIndexDeletion(ESIntegTestCase.java:614)
	at org.elasticsearch.test.ESIntegTestCase.afterInternal(ESIntegTestCase.java:584)
	at org.elasticsearch.test.ESIntegTestCase.cleanUpCluster(ESIntegTestCase.java:2110)
	at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)
	Suppressed: java.lang.AssertionError: [index-2][1], node[pk2CqV3qSSuBbrD7XG_lGQ], relocating [Ei8GUN6gT-mv72m6aEdh7Q], [P], s[RELOCATING], a[id=YigPiA4lRE27xyAbsksJag, rId=78tkoVggRdqJWaHOcMg-2Q], expected_shard_size[6327479] has unreleased snapshotted index commits
		at org.junit.Assert.fail(Assert.java:88)
		at org.junit.Assert.assertTrue(Assert.java:41)
		at org.junit.Assert.assertFalse(Assert.java:64)
		at org.elasticsearch.test.InternalTestCluster.lambda$assertNoSnapshottedIndexCommit$10(InternalTestCluster.java:1365)
		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:879)
		... 40 more
	Suppressed: java.lang.AssertionError: [index-2][1], node[pk2CqV3qSSuBbrD7XG_lGQ], relocating [Ei8GUN6gT-mv72m6aEdh7Q], [P], s[RELOCATING], a[id=YigPiA4lRE27xyAbsksJag, rId=78tkoVggRdqJWaHOcMg-2Q], expected_shard_size[6327479] has unreleased snapshotted index commits
		at org.junit.Assert.fail(Assert.java:88)
		at org.junit.Assert.assertTrue(Assert.java:41)
		at org.junit.Assert.assertFalse(Assert.java:64)
		at org.elasticsearch.test.InternalTestCluster.lambda$assertNoSnapshottedIndexCommit$10(InternalTestCluster.java:1365)
		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:879)
		... 40 more
	Suppressed: java.lang.AssertionError: [index-2][1], node[pk2CqV3qSSuBbrD7XG_lGQ], relocating [Ei8GUN6gT-mv72m6aEdh7Q], [P], s[RELOCATING], a[id=YigPiA4lRE27xyAbsksJag, rId=78tkoVggRdqJWaHOcMg-2Q], expected_shard_size[6327479] has unreleased snapshotted index commits
		at org.junit.Assert.fail(Assert.java:88)
		at org.junit.Assert.assertTrue(Assert.java:41)
		at org.junit.Assert.assertFalse(Assert.java:64)
		at org.elasticsearch.test.InternalTestCluster.lambda$assertNoSnapshottedIndexCommit$10(InternalTestCluster.java:1365)
		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:879)
		... 40 more
	Suppressed: java.lang.AssertionError: [index-2][1], node[pk2CqV3qSSuBbrD7XG_lGQ], relocating [Ei8GUN6gT-mv72m6aEdh7Q], [P], s[RELOCATING], a[id=YigPiA4lRE27xyAbsksJag, rId=78tkoVggRdqJWaHOcMg-2Q], expected_shard_size[6327479] has unreleased snapshotted index commits
		at org.junit.Assert.fail(Assert.java:88)
		at org.junit.Assert.assertTrue(Assert.java:41)
		at org.junit.Assert.assertFalse(Assert.java:64)
		at org.elasticsearch.test.InternalTestCluster.lambda$assertNoSnapshottedIndexCommit$10(InternalTestCluster.java:1365)
		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:879)
		... 40 more
	Suppressed: java.lang.AssertionError: [index-2][1], node[pk2CqV3qSSuBbrD7XG_lGQ], relocating [Ei8GUN6gT-mv72m6aEdh7Q], [P], s[RELOCATING], a[id=YigPiA4lRE27xyAbsksJag, rId=78tkoVggRdqJWaHOcMg-2Q], expected_shard_size[6327479] has unreleased snapshotted index commits
		at org.junit.Assert.fail(Assert.java:88)
		at org.junit.Assert.assertTrue(Assert.java:41)
		at org.junit.Assert.assertFalse(Assert.java:64)
		at org.elasticsearch.test.InternalTestCluster.lambda$assertNoSnapshottedIndexCommit$10(InternalTestCluster.java:1365)
		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:879)
		... 40 more
	Suppressed: java.lang.AssertionError: [index-2][1], node[pk2CqV3qSSuBbrD7XG_lGQ], relocating [Ei8GUN6gT-mv72m6aEdh7Q], [P], s[RELOCATING], a[id=YigPiA4lRE27xyAbsksJag, rId=78tkoVggRdqJWaHOcMg-2Q], expected_shard_size[6327479] has unreleased snapshotted index commits
		at org.junit.Assert.fail(Assert.java:88)
		at org.junit.Assert.assertTrue(Assert.java:41)
		at org.junit.Assert.assertFalse(Assert.java:64)
		at org.elasticsearch.test.InternalTestCluster.lambda$assertNoSnapshottedIndexCommit$10(InternalTestCluster.java:1365)
		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:879)
		... 40 more
	Suppressed: java.lang.AssertionError: [index-2][1], node[pk2CqV3qSSuBbrD7XG_lGQ], relocating [Ei8GUN6gT-mv72m6aEdh7Q], [P], s[RELOCATING], a[id=YigPiA4lRE27xyAbsksJag, rId=78tkoVggRdqJWaHOcMg-2Q], expected_shard_size[6327479] has unreleased snapshotted index commits
		at org.junit.Assert.fail(Assert.java:88)
		at org.junit.Assert.assertTrue(Assert.java:41)
		at org.junit.Assert.assertFalse(Assert.java:64)
		at org.elasticsearch.test.InternalTestCluster.lambda$assertNoSnapshottedIndexCommit$10(InternalTestCluster.java:1365)
		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:879)
		... 40 more
	Suppressed: java.lang.AssertionError: [index-2][1], node[pk2CqV3qSSuBbrD7XG_lGQ], relocating [Ei8GUN6gT-mv72m6aEdh7Q], [P], s[RELOCATING], a[id=YigPiA4lRE27xyAbsksJag, rId=78tkoVggRdqJWaHOcMg-2Q], expected_shard_size[6327479] has unreleased snapshotted index commits
		at org.junit.Assert.fail(Assert.java:88)
		at org.junit.Assert.assertTrue(Assert.java:41)
		at org.junit.Assert.assertFalse(Assert.java:64)
		at org.elasticsearch.test.InternalTestCluster.lambda$assertNoSnapshottedIndexCommit$10(InternalTestCluster.java:1365)
		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:879)
		... 40 more
	Suppressed: java.lang.AssertionError: [index-2][1], node[pk2CqV3qSSuBbrD7XG_lGQ], relocating [Ei8GUN6gT-mv72m6aEdh7Q], [P], s[RELOCATING], a[id=YigPiA4lRE27xyAbsksJag, rId=78tkoVggRdqJWaHOcMg-2Q], expected_shard_size[6327479] has unreleased snapshotted index commits
		at org.junit.Assert.fail(Assert.java:88)
		at org.junit.Assert.assertTrue(Assert.java:41)
		at org.junit.Assert.assertFalse(Assert.java:64)
		at org.elasticsearch.test.InternalTestCluster.lambda$assertNoSnapshottedIndexCommit$10(InternalTestCluster.java:1365)
		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:879)
		... 40 more
	Suppressed: java.lang.AssertionError: [index-2][1], node[pk2CqV3qSSuBbrD7XG_lGQ], relocating [Ei8GUN6gT-mv72m6aEdh7Q], [P], s[RELOCATING], a[id=YigPiA4lRE27xyAbsksJag, rId=78tkoVggRdqJWaHOcMg-2Q], expected_shard_size[6327479] has unreleased snapshotted index commits
		at org.junit.Assert.fail(Assert.java:88)
		at org.junit.Assert.assertTrue(Assert.java:41)
		at org.junit.Assert.assertFalse(Assert.java:64)
		at org.elasticsearch.test.InternalTestCluster.lambda$assertNoSnapshottedIndexCommit$10(InternalTestCluster.java:1365)
		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:879)
		... 40 more
	Suppressed: java.lang.AssertionError: [index-2][1], node[pk2CqV3qSSuBbrD7XG_lGQ], relocating [Ei8GUN6gT-mv72m6aEdh7Q], [P], s[RELOCATING], a[id=YigPiA4lRE27xyAbsksJag, rId=78tkoVggRdqJWaHOcMg-2Q], expected_shard_size[6327479] has unreleased snapshotted index commits
		at org.junit.Assert.fail(Assert.java:88)
		at org.junit.Assert.assertTrue(Assert.java:41)
		at org.junit.Assert.assertFalse(Assert.java:64)
		at org.elasticsearch.test.InternalTestCluster.lambda$assertNoSnapshottedIndexCommit$10(InternalTestCluster.java:1365)
		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:879)
		... 40 more
	Suppressed: java.lang.AssertionError: [index-2][0], node[47Pd3heUS86aYOtaDIkfdA], relocating [Ei8GUN6gT-mv72m6aEdh7Q], [P], s[RELOCATING], a[id=ON9IncKrSZGA7iNl9csiUw, rId=qjOMwwczSSKTcn38ATktng], expected_shard_size[6238507] has unreleased snapshotted index commits
		at org.junit.Assert.fail(Assert.java:88)
		at org.junit.Assert.assertTrue(Assert.java:41)
		at org.junit.Assert.assertFalse(Assert.java:64)
		at org.elasticsearch.test.InternalTestCluster.lambda$assertNoSnapshottedIndexCommit$10(InternalTestCluster.java:1365)
		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:879)
		... 40 more
	Suppressed: java.lang.AssertionError: [index-2][0], node[47Pd3heUS86aYOtaDIkfdA], relocating [Ei8GUN6gT-mv72m6aEdh7Q], [P], s[RELOCATING], a[id=ON9IncKrSZGA7iNl9csiUw, rId=qjOMwwczSSKTcn38ATktng], expected_shard_size[6238507] has unreleased snapshotted index commits
		at org.junit.Assert.fail(Assert.java:88)
		at org.junit.Assert.assertTrue(Assert.java:41)
		at org.junit.Assert.assertFalse(Assert.java:64)
		at org.elasticsearch.test.InternalTestCluster.lambda$assertNoSnapshottedIndexCommit$10(InternalTestCluster.java:1365)
		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:879)
		... 40 more
@DaveCTurner DaveCTurner added >test-failure Triaged test failures from CI :Distributed Indexing/Recovery Anything around constructing a new shard, either from a local or a remote source. labels Nov 20, 2019
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-distributed (:Distributed/Recovery)

@dnhatn dnhatn self-assigned this Nov 20, 2019
@dnhatn dnhatn mentioned this issue Nov 21, 2019
Merged
dnhatn added a commit that referenced this issue Nov 22, 2019
@dnhatn
Increase timeout while checking for no snapshotted commit (#49461)
5d00bee 
If some replica is performing a file-based recovery, then the check 
assertNoSnapshottedIndexCommit would fail. We should increase the
timeout for this check so that we can wait until all recoveries done
or aborted.

Closes #49403
dnhatn added a commit that referenced this issue Nov 24, 2019
@dnhatn
Increase timeout while checking for no snapshotted commit (#49461)
8260cba 
If some replica is performing a file-based recovery, then the check 
assertNoSnapshottedIndexCommit would fail. We should increase the
timeout for this check so that we can wait until all recoveries done
or aborted.

Closes #49403
dnhatn added a commit that referenced this issue Nov 24, 2019
@dnhatn
Increase timeout while checking for no snapshotted commit (#49461)
3a96f4e 
If some replica is performing a file-based recovery, then the check 
assertNoSnapshottedIndexCommit would fail. We should increase the
timeout for this check so that we can wait until all recoveries done
or aborted.

Closes #49403
dnhatn added a commit that referenced this issue Nov 24, 2019
@dnhatn
Increase timeout while checking for no snapshotted commit (#49461)
5c98c50 
If some replica is performing a file-based recovery, then the check 
assertNoSnapshottedIndexCommit would fail. We should increase the
timeout for this check so that we can wait until all recoveries done
or aborted.

Closes #49403
dnhatn added a commit that referenced this issue Nov 25, 2019
@dnhatn
Increase timeout while checking for no snapshotted commit (#49461)
7ddf540 
If some replica is performing a file-based recovery, then the check 
assertNoSnapshottedIndexCommit would fail. We should increase the
timeout for this check so that we can wait until all recoveries done
or aborted.

Closes #49403
@williamrandolph williamrandolph mentioned this issue Mar 13, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dnhatn dnhatn

Labels
:Distributed Indexing/Recovery Anything around constructing a new shard, either from a local or a remote source. >test-failure Triaged test failures from CI
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Increase timeout while checking for no snapshotted commit dnhatn/elasticsearch
3 participants
@DaveCTurner @dnhatn @elasticmachine