Increase Size and lower TTL on DLS BitSet Cache #50535
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The Document Level Security BitSet Cache (see elastic#43669) had a default configuration of "small size, long lifetime". However, this is not a very useful default as the cache is most valuable for BitSets that take a long time to construct, which is (generally speaking) the same ones that operate over a large number of documents and contain many bytes. This commit changes the cache to be "large size, short lifetime" so that it can hold bitsets representing billions of documents, but releases memory quickly. The new defaults are 10% of heap, and 2 hours. This also adds some logging when a single BitSet exceeds the size of the cache and when the cache is full.
tvernum
added
>enhancement
:Security/Authorization
|
Pinging @elastic/es-security (:Security/Authorization) |
albertzaharovits
approved these changes
Jan 2, 2020
There was a problem hiding this comment.
LGTM - the prev value of 50Mb was too low and a ratio value makes sense to me. I don't feel qualified to answer about the 10% I don't have enough practical experience with real life memory numbers.
I sense it might be rather large, but the 2hrs expiry time should mitigate it.
I think we can expose to the DocumentSubsetBitsetCache that this roleQuery pertains from a script, and have a dedicated cache for those, i.e. a new Cache instance member. The original TTL of 1 week makes sense for non-templated role queries. However, templated role queries are rather difficult to set up and users that employ them really need them, which makes me assume they most likely use dynamic attributes of the user, or time, thereby flooding the cache which turns our 1 week TTL assumptions on their head.
jpountz
approved these changes
Jan 9, 2020
...ava/org/elasticsearch/xpack/core/security/authz/accesscontrol/DocumentSubsetBitsetCache.java
Outdated
Show resolved
Hide resolved
…security/authz/accesscontrol/DocumentSubsetBitsetCache.java Co-Authored-By: Adrien Grand <jpountz@gmail.com>
@elasticmachine run elasticsearch-ci/bwc |
tvernum
added a commit
to tvernum/elasticsearch
that referenced
this pull request
Jan 14, 2020
The Document Level Security BitSet Cache (see elastic#43669) had a default configuration of "small size, long lifetime". However, this is not a very useful default as the cache is most valuable for BitSets that take a long time to construct, which is (generally speaking) the same ones that operate over a large number of documents and contain many bytes. This commit changes the cache to be "large size, short lifetime" so that it can hold bitsets representing billions of documents, but releases memory quickly. The new defaults are 10% of heap, and 2 hours. This also adds some logging when a single BitSet exceeds the size of the cache and when the cache is full. Resolves: elastic#49260 Backport of: elastic#50535
tvernum
added a commit
that referenced
this pull request
Jan 14, 2020
The Document Level Security BitSet Cache (see #43669) had a default configuration of "small size, long lifetime". However, this is not a very useful default as the cache is most valuable for BitSets that take a long time to construct, which is (generally speaking) the same ones that operate over a large number of documents and contain many bytes. This commit changes the cache to be "large size, short lifetime" so that it can hold bitsets representing billions of documents, but releases memory quickly. The new defaults are 10% of heap, and 2 hours. This also adds some logging when a single BitSet exceeds the size of the cache and when the cache is full. Backport of: #50535
SivagurunathanV
pushed a commit
to SivagurunathanV/elasticsearch
that referenced
this pull request
Jan 23, 2020
The Document Level Security BitSet Cache (see elastic#43669) had a default configuration of "small size, long lifetime". However, this is not a very useful default as the cache is most valuable for BitSets that take a long time to construct, which is (generally speaking) the same ones that operate over a large number of documents and contain many bytes. This commit changes the cache to be "large size, short lifetime" so that it can hold bitsets representing billions of documents, but releases memory quickly. The new defaults are 10% of heap, and 2 hours. This also adds some logging when a single BitSet exceeds the size of the cache and when the cache is full. Resolves: elastic#49260
This was referenced Feb 3, 2020
tvernum
added a commit
to tvernum/elasticsearch
that referenced
this pull request
Apr 29, 2022
In elastic#50535 (ES v7.6) the default values for the `DocumentSubsetBitsetCache` settings were changed. However, the docs were not updated at that tim, and still reflect the old values for these settings
elasticsearchmachine
pushed a commit
that referenced
this pull request
May 2, 2022
In #50535 (ES v7.6) the default values for the `DocumentSubsetBitsetCache` settings were changed. However, the docs were not updated at that time, and still reflect the old values for these settings
tvernum
added a commit
to tvernum/elasticsearch
that referenced
this pull request
May 2, 2022
In elastic#50535 (ES v7.6) the default values for the `DocumentSubsetBitsetCache` settings were changed. However, the docs were not updated at that time, and still reflect the old values for these settings
tvernum
added a commit
to tvernum/elasticsearch
that referenced
this pull request
May 2, 2022
In elastic#50535 (ES v7.6) the default values for the `DocumentSubsetBitsetCache` settings were changed. However, the docs were not updated at that time, and still reflect the old values for these settings
tvernum
added a commit
to tvernum/elasticsearch
that referenced
this pull request
May 2, 2022
In elastic#50535 (ES v7.6) the default values for the `DocumentSubsetBitsetCache` settings were changed. However, the docs were not updated at that time, and still reflect the old values for these settings
elasticsearchmachine
pushed a commit
that referenced
this pull request
May 2, 2022
elasticsearchmachine
pushed a commit
that referenced
this pull request
May 2, 2022
elasticsearchmachine
pushed a commit
that referenced
this pull request
May 2, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The Document Level Security BitSet Cache (see #43669) had a default
configuration of "small size, long lifetime". However, this is not
a very useful default as the cache is most valuable for BitSets that
take a long time to construct, which is (generally speaking) the same
ones that operate over a large number of documents and contain many
bytes.
This commit changes the cache to be "large size, short lifetime" so
that it can hold bitsets representing billions of documents, but
releases memory quickly.
The new defaults are 10% of heap, and 2 hours.
This also adds some logging when a single BitSet exceeds the size of
the cache and when the cache is full.
Resolves: #49260