Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CI] FlushIT.testSyncedFlushWithConcurrentIndexing failure #47060

Closed
bizybot opened this issue Sep 25, 2019 · 5 comments · Fixed by #47186
Closed

[CI] FlushIT.testSyncedFlushWithConcurrentIndexing failure #47060

bizybot opened this issue Sep 25, 2019 · 5 comments · Fixed by #47186
Assignees
@dnhatn
Labels
:Distributed Indexing/Distributed A catch all label for anything in the Distributed Area. Please avoid if you can. >test-failure Triaged test failures from CI

Comments

@bizybot
Copy link
Contributor

bizybot commented Sep 25, 2019

The test has failed multiple times:
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+matrix-java-periodic/ES_BUILD_JAVA=openjdk12,ES_RUNTIME_JAVA=zulu11,nodes=general-purpose/225/console
and on different branches
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.4+matrix-java-periodic/ES_BUILD_JAVA=openjdk12,ES_RUNTIME_JAVA=corretto11,nodes=general-purpose/48/console

I could not reproduce locally

./gradlew ':server:integTest' --tests "org.elasticsearch.indices.flush.FlushIT.testSyncedFlushWithConcurrentIndexing" \
  -Dtests.seed=68566A4079D35F31 \
  -Dtests.security.manager=true \
  -Dtests.locale=rm-CH \
  -Dtests.timezone=Atlantic/Azores \
  -Dcompiler.java=12 \
  -Druntime.java=11
@bizybot bizybot added >test-failure Triaged test failures from CI :Distributed Indexing/Distributed A catch all label for anything in the Distributed Area. Please avoid if you can. labels Sep 25, 2019
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-distributed

@dnhatn
Copy link
Member

dnhatn commented Sep 25, 2019

Some shard did not pass checkIndex.

  2> REPRODUCE WITH: ./gradlew ':server:integTest' --tests "org.elasticsearch.indices.flush.FlushIT.testSyncedFlushWithConcurrentIndexing" -Dtests.seed=68566A4079D35F31 -Dtests.security.manager=true -Dtests.locale=rm-CH -Dtests.timezone=Atlantic/Azores -Dcompiler.java=12 -Druntime.java=11
  2> java.lang.AssertionError: at least one shard failed CheckIndex

@dnhatn dnhatn self-assigned this Sep 25, 2019
@dnhatn
Copy link
Member

dnhatn commented Sep 25, 2019

This relates to #45263. I am working on the fix.

@dnhatn dnhatn mentioned this issue Sep 26, 2019
Merged
@romseygeek
Copy link
Contributor

Another failure here: https://www.google.com/url?q=https%3A%2F%2Fgradle-enterprise.elastic.co%2Fs%2Fndxlpxwumgnuk&sa=D&sntz=1&usg=AFQjCNGV7r_O-5hr3LWrv9RqX1QAxh_-GQ

dnhatn added a commit that referenced this issue Nov 7, 2019
@dnhatn
Closed shard should never open new engine (#47186)
d029e18 
We should not open new engines if a shard is closed. We break this
assumption in #45263 where we stop verifying the shard state before
creating an engine but only before swapping the engine reference.
We can fail to snapshot the store metadata or checkIndex a closed shard
if there's some IndexWriter holding the index lock.

Closes #47060
dnhatn added a commit that referenced this issue Nov 9, 2019
@dnhatn
Closed shard should never open new engine (#47186)
2a0b430 
We should not open new engines if a shard is closed. We break this
assumption in #45263 where we stop verifying the shard state before
creating an engine but only before swapping the engine reference.
We can fail to snapshot the store metadata or checkIndex a closed shard
if there's some IndexWriter holding the index lock.

Closes #47060
dnhatn added a commit that referenced this issue Nov 9, 2019
@dnhatn
Closed shard should never open new engine (#47186)
ff6c121 
We should not open new engines if a shard is closed. We break this
assumption in #45263 where we stop verifying the shard state before
creating an engine but only before swapping the engine reference.
We can fail to snapshot the store metadata or checkIndex a closed shard
if there's some IndexWriter holding the index lock.

Closes #47060
This was referenced Feb 3, 2020
Closed
Closed
@nknize
Copy link
Contributor

nknize commented Sep 15, 2020 •
edited
Loading

Here's a failure on 7.9. Looks like a time out waiting for green state:

14:25:27   2> REPRODUCE WITH: ./gradlew ':server:internalClusterTest' --tests "org.elasticsearch.indices.flush.FlushIT.testSyncedFlushWithConcurrentIndexing" -Dtests.seed=95D587608E40DC2C -Dtests.security.manager=true -Dtests.locale=de-DE -Dtests.timezone=Australia/Tasmania -Druntime.java=11 -Dtests.fips.enabled=true

14:24:47 org.elasticsearch.indices.flush.FlushIT > testSyncedFlushWithConcurrentIndexing FAILED
14:24:47     java.lang.AssertionError: timed out waiting for green state
14:24:47         at __randomizedtesting.SeedInfo.seed([95D587608E40DC2C:182200A22BB331D6]:0)
14:24:47         at org.junit.Assert.fail(Assert.java:88)
14:24:47         at org.elasticsearch.test.ESIntegTestCase.ensureColor(ESIntegTestCase.java:952)
14:24:47         at org.elasticsearch.test.ESIntegTestCase.ensureGreen(ESIntegTestCase.java:905)
14:24:47         at org.elasticsearch.test.ESIntegTestCase.ensureGreen(ESIntegTestCase.java:894)
14:24:47         at org.elasticsearch.indices.flush.FlushIT.testSyncedFlushWithConcurrentIndexing(FlushIT.java:245)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dnhatn dnhatn

Labels
:Distributed Indexing/Distributed A catch all label for anything in the Distributed Area. Please avoid if you can. >test-failure Triaged test failures from CI
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Closed shard should never open new engine dnhatn/elasticsearch
5 participants
@nknize @bizybot @romseygeek @dnhatn @elasticmachine