Make PerThreadIDAndVersionLookup per-segment #14070
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
|
LGTM, this is a nice cleanup (1 TermsEnum per segment core, instead of 1 per top-level reader X segment core). I benchmarked indexing at defaults and it looks like any performance change is within noise. |
Contributor
|
LGTM too - I think as a followup we can remove the support for payloads in master since we don't support indices with payloads anymore. |
clintongormley
added
>enhancement
:Core/Infra/Core
Contributor
Author
|
I added some tests for cache eviction of readers and so on here. I think its ready. |
Contributor
|
LGTM thanks for all the tests |
rmuir
added a commit
that referenced
this pull request
Oct 13, 2015
Closes #14070 Squashed commit of the following: commit ad5829e Author: Robert Muir <rmuir@apache.org> Date: Tue Oct 13 06:50:25 2015 -0400 Add more tests for version map caching commit 617333e Author: Robert Muir <rmuir@apache.org> Date: Mon Oct 12 13:25:51 2015 -0400 fix javadocs ref commit 9626dd0 Author: Robert Muir <rmuir@apache.org> Date: Mon Oct 12 13:23:16 2015 -0400 make the version map cache per-segment Conflicts: core/src/main/java/org/elasticsearch/common/lucene/uid/Versions.java
rmuir
added a commit
that referenced
this pull request
Oct 13, 2015
Closes #14070 Squashed commit of the following: commit ad5829e Author: Robert Muir <rmuir@apache.org> Date: Tue Oct 13 06:50:25 2015 -0400 Add more tests for version map caching commit 617333e Author: Robert Muir <rmuir@apache.org> Date: Mon Oct 12 13:25:51 2015 -0400 fix javadocs ref commit 9626dd0 Author: Robert Muir <rmuir@apache.org> Date: Mon Oct 12 13:23:16 2015 -0400 make the version map cache per-segment Conflicts: core/src/main/java/org/elasticsearch/common/lucene/uid/Versions.java
rmuir
added a commit
that referenced
this pull request
Oct 13, 2015
Closes #14070 Squashed commit of the following: commit ad5829e Author: Robert Muir <rmuir@apache.org> Date: Tue Oct 13 06:50:25 2015 -0400 Add more tests for version map caching commit 617333e Author: Robert Muir <rmuir@apache.org> Date: Mon Oct 12 13:25:51 2015 -0400 fix javadocs ref commit 9626dd0 Author: Robert Muir <rmuir@apache.org> Date: Mon Oct 12 13:23:16 2015 -0400 make the version map cache per-segment Conflicts: core/src/main/java/org/elasticsearch/common/lucene/uid/Versions.java Conflicts: core/src/main/java/org/elasticsearch/common/lucene/uid/PerThreadIDAndVersionLookup.java
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Currently this caches some state per-top-level-reader.
This currently has a downside in that it makes managing the cache tricky when readers are wrapped, since the map is cleared by
addReaderClosedListener.As a per-segment cache, things are simpler for that case since we can just use
addCoreClosedListener, which will evict from this cache when the underlying segment's core is merged away. Its effectively required for reader wrappers to delegategetCoreCacheKeyfor good performance with elasticsearch (otherwise you would e.g. build fielddata and so on per-query), so its not a limitation.It also addresses the TODO in the code about handling reopen...
If @mikemccand benchmarking is ok, I will add unit tests to this PR, we should be able to easily test this thing.