Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CI] SplitIndexIT fails with Lucene-7.4 snapshot #30416

Closed
dnhatn opened this issue May 7, 2018 · 6 comments · Fixed by #30571
Closed

[CI] SplitIndexIT fails with Lucene-7.4 snapshot #30416

dnhatn opened this issue May 7, 2018 · 6 comments · Fixed by #30571
Assignees
@s1monw
Labels
:Distributed Indexing/Recovery Anything around constructing a new shard, either from a local or a remote source. >test-failure Triaged test failures from CI

Comments

@dnhatn
Copy link
Member

dnhatn commented May 7, 2018 •
edited
Loading

This test starts failing more often after upgrading to Lucene-7.4 snapshot. Most of the failures are on Windows. It may relate to this Lucene failure https://elasticsearch-ci.elastic.co/job/apache+lucene-solr+master/27765/consoleText.

Error: still has pending deleted files; cannot initialize IndexWriter

  1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\16\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter
  1> 	at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13]
  1> 	at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?]
  1> 	at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?]
  1> 	at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?]
  1> 	at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?]
  1> 	at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?]
  1> 	at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?]
  1> 	at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?]
  1> 	at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?]
  1> 	... 4 more

Some CI instances:

  1. https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob-windows-compatibility/1608/consoleText

  2. https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob-windows-compatibility/1609/console

Log:
shrink.txt
@dnhatn dnhatn added >test Issues or PRs that are addressing/adding tests :Distributed Indexing/Recovery Anything around constructing a new shard, either from a local or a remote source. labels May 7, 2018
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-distributed

@dnhatn
Copy link
Member Author

dnhatn commented May 7, 2018

Ping @simonw and @elastic/es-search-aggs

dnhatn added a commit that referenced this issue May 7, 2018
@dnhatn
AwaitsFix testCreateShrinkIndexToN
9c77eab 
Relates #30416
dnhatn added a commit that referenced this issue May 7, 2018
@dnhatn
AwaitsFix testCreateShrinkIndexToN
16d6a0b 
Relates #30416
@dnhatn
Copy link
Member Author

dnhatn commented May 7, 2018

I muted the test at 9c77eab.

colings86 pushed a commit that referenced this issue May 8, 2018
@dnhatn @colings86
AwaitsFix testCreateShrinkIndexToN
eb51318 
Relates #30416
@bleskes bleskes changed the title [CI] testCreateShrinkIndexToN fails with Lucene-7.4 snapshot [CI] SplitIndexIT failures with Lucene-7.4 snapshot May 9, 2018
@bleskes
Copy link
Contributor

bleskes commented May 9, 2018

I muted the entire suit as we have similar failures in many tests.

@bleskes bleskes mentioned this issue May 9, 2018
Closed
bleskes added a commit that referenced this issue May 9, 2018
@bleskes
mute SplitIndexIT due to #30416
240a338
bleskes added a commit that referenced this issue May 9, 2018
@bleskes
mute SplitIndexIT due to #30416
54122d8
@s1monw s1monw self-assigned this May 9, 2018
@s1monw
Copy link
Contributor

s1monw commented May 9, 2018

yeah this will not work on windows. I can look how to fix soon

@polyfractal polyfractal added >test-failure Triaged test failures from CI and removed >test Issues or PRs that are addressing/adding tests labels May 9, 2018
@jasontedor jasontedor mentioned this issue May 10, 2018
Closed
dnhatn added a commit that referenced this issue May 10, 2018
@dnhatn
Merge branch 'master' into ccr
a5be414 
* master:
  Upgrade to Lucene-7.4-snapshot-6705632810 (#30519)
  add version compatibility from 6.4.0 after backport, see #30319 (#30390)
  Security: Simplify security index listeners (#30466)
  Add proper longitude validation in geo_polygon_query (#30497)
  Remove Discovery.AckListener.onTimeout() (#30514)
  Build: move generated-resources to build (#30366)
  Reindex: Fold "with all deps" project into reindex (#30154)
  Isolate REST client single host tests (#30504)
  Solve Gradle deprecation warnings around shadowJar (#30483)
  SAML: Process only signed data (#30420)
  Remove BWC repository test (#30500)
  Build: Remove xpack specific run task (#30487)
  AwaitsFix IntegTestZipClientYamlTestSuiteIT#indices.split tests
  LLClient: Add setJsonEntity (#30447)
  Expose CommonStatsFlags directly in IndicesStatsRequest. (#30163)
  Silence IndexUpgradeIT test failures. (#30430)
  Bump Gradle heap to 1792m (#30484)
  [docs] add warning for read-write indices in force merge documentation (#28869)
  Avoid deadlocks in cache (#30461)
  Test: remove hardcoded list of unconfigured ciphers (#30367)
  mute SplitIndexIT due to #30416
  Docs: Test examples that recreate lang analyzers  (#29535)
  BulkProcessor to retry based on status code (#29329)
  Add GET Repository High Level REST API (#30362)
  add a comment explaining the need for RetryOnReplicaException on missing mappings
  Add `coordinating_only` node selector (#30313)
  Stop forking groovyc (#30471)
  Avoid setting connection request timeout (#30384)
  Use date format in `date_range` mapping before fallback to default (#29310)
  Watcher: Increase HttpClient parallel sent requests (#30130)

# Conflicts:
#	x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/LocalStateCompositeXPackPlugin.java
dnhatn added a commit that referenced this issue May 10, 2018
@dnhatn
Merge branch '6.x' into ccr-6.x
4785696 
* 6.x:
  Upgrade to Lucene-7.4-snapshot-6705632810 (#30519)
  Remove Discovery.AckListener.onTimeout() (#30514)
  Build: move generated-resources to build (#30366)
  Reindex: Fold "with all deps" project into reindex (#30154)
  Isolate REST client single host tests (#30504)
  Remove BWC repository test (#30500)
  Build: Remove xpack specific run task (#30487)
  AwaitsFix IntegTestZipClientYamlTestSuiteIT#indices.split tests
  LLClient: Add setJsonEntity (#30447)
  [docs] add warning for read-write indices in force merge documentation (#28869)
  Avoid deadlocks in cache (#30461)
  BulkProcessor to retry based on status code (#29329)
  Avoid setting connection request timeout (#30384)
  Test: remove hardcoded list of unconfigured ciphers (#30367)
  Add GET Repository High Level REST API (#30362)
  mute SplitIndexIT due to #30416
  Docs: Test examples that recreate lang analyzers  (#29535)
  add a comment explaining the need for RetryOnReplicaException on missing mappings
  Pass the task to broadcast actions (#29672)
  Stop forking groovyc (#30471)
  Add `coordinating_only` node selector (#30313)
  Fix accidental error in changelog
  Use date format in `date_range` mapping before fallback to default (#29310)
  Watcher: Increase HttpClient parallel sent requests (#30130)
  [Security][Tests] Azeri(Turkish) locale tripps opensaml dependency
dnhatn added a commit that referenced this issue May 13, 2018
@dnhatn
Mute ShrinkIndexIT suite
9d1e687 
Relates #30416
dnhatn added a commit that referenced this issue May 13, 2018
@dnhatn
Mute ShrinkIndexIT suite
73ec90f 
Relates #30416
@s1monw s1monw mentioned this issue May 14, 2018
Closed
@s1monw
Copy link
Contributor

s1monw commented May 14, 2018

I opened #30567 for this

s1monw added a commit to s1monw/elasticsearch that referenced this issue May 14, 2018
@s1monw
Side-step pending deletes check
05663e4 
When we split/shrink an index we open several IndexWriter instances
causeing file-deletes to be pending on windows. This subsequently fails
when we open an IW to bootstrap the index history due to pending deletes.
This change sidesteps the check since we know our history goes forward
in terms of files and segments.

Closes elastic#30416
@s1monw s1monw mentioned this issue May 14, 2018
Merged
@dnhatn dnhatn changed the title [CI] SplitIndexIT failures with Lucene-7.4 snapshot [CI] SplitIndexIT fails with Lucene-7.4 snapshot May 15, 2018
s1monw added a commit that referenced this issue May 15, 2018
@s1monw
Side-step pending deletes check (#30571)
b50cf3c 
When we split/shrink an index we open several IndexWriter instances
causeing file-deletes to be pending on windows. This subsequently fails
when we open an IW to bootstrap the index history due to pending deletes.
This change sidesteps the check since we know our history goes forward
in terms of files and segments.

Closes #30416
@s1monw s1monw mentioned this issue May 15, 2018
Merged
s1monw added a commit that referenced this issue May 15, 2018
@s1monw
Side-step pending deletes check (#30607)
be587bf 
When we split/shrink an index we open several IndexWriter instances
causeing file-deletes to be pending on windows. This subsequently fails
when we open an IW to bootstrap the index history due to pending deletes.
This change sidesteps the check since we know our history goes forward
in terms of files and segments.

Closes #30416
jasontedor pushed a commit that referenced this issue Aug 17, 2018
@dnhatn @jasontedor
AwaitsFix testCreateShrinkIndexToN
0413813 
Relates #30416
@droberts195 droberts195 mentioned this issue Sep 26, 2018
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@s1monw s1monw

Labels
:Distributed Indexing/Recovery Anything around constructing a new shard, either from a local or a remote source. >test-failure Triaged test failures from CI
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Side-step pending deletes check s1monw/elasticsearch
5 participants
@s1monw @bleskes @polyfractal @dnhatn @elasticmachine