Add partial searchable snapshot support for a frozen tier #68509
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This adds all the necessary infrastructure to use the reusable, single-file cache in practice: * Create cache file in a data directory instead of a temp directory * Fully pre-allocate it (the existing solution would at least on Linux still do a sparse allocation) * Manage file channel resource by ref counting * Add minimal abstraction in place of exposing `FileChannel` to allow for adjusting the concrete paging approach under the hood in a follow-up
This commit introduces a new flag, `?partial_local_copy`, indicating that the local copy of a searchable snapshot should be partial rather than complete, enabling the frozen tier functionality.
DaveCTurner
approved these changes
Feb 5, 2021
tlrx
approved these changes
Feb 5, 2021
ywelsch
added a commit
that referenced
this pull request
Feb 5, 2021
A frozen tier is backed by an external object store (like S3) and caches only a small portion of data on local disks. In this way, users can reduce hardware costs substantially for infrequently accessed data. For the frozen tier we only pull in the parts of the files that are actually needed to run a given search. Further, we don't require the node to have enough space to host all the files. We therefore have a cache that manages which file parts are available, and which ones not. This node-level shared cache is bounded in size (typically in relation to the disk size), and will evict items based on a LFU policy, as we expect some parts of the Lucene files to be used more frequently than other parts. The level of granularity for evictions is at the level of regions of a file, and does not require evicting full files. The on-disk representation that was chosen for the cold tier is not a good fit here, as it won't allow evicting parts of a file. Instead we are using fixed-size pre-allocated files and have implemented our own memory management logic to map regions of the shard's original Lucene files onto regions in these node-level shared files that are representing the on-disk cache. This PR adds the core functionality to searchable snapshots to power such a frozen tier: - It adds the node-level shared cache that evicts file regions based on a LFU policy - It adds the machinery to dynamically download file regions into this cache and serve their contents when searches execute. - It extends the mount API with a new parameter, `storage`, which selects the kind of local storage used to accelerate searches of the mounted index. If set to `full_copy` (default, used for cold tier), each node holding a shard of the searchable snapshot index makes a full copy of the shard to its local storage. If set to `shared_cache`, the shard uses the newly introduced shared cache, only holding a partial copy of the index on disk (used for frozen tier). Co-authored-by: Tanguy Leroux <tlrx.dev@gmail.com> Co-authored-by: Armin Braun <me@obrown.io> Co-authored-by: David Turner <david.turner@elastic.co>
ywelsch
added a commit
that referenced
this pull request
Feb 5, 2021
dakrone
added a commit
to dakrone/elasticsearch
that referenced
this pull request
Feb 8, 2021
This commit adds support for the recently introduced partial searchable snapshot (elastic#68509) to ILM. Searchable snapshot ILM actions may now be specified with a `storage` option, specifying either `full_copy` or `shared_cache` (similar to the "mount" API) to mount either a full or partial searchable snapshot: ```json PUT _ilm/policy/my_policy { "policy": { "phases": { "cold": { "actions": { "searchable_snapshot" : { "snapshot_repository" : "backing_repo", "storage": "shared_cache" } } } } } } ``` Internally, If more than one searchable snapshot action is specified (for example, a full searchable snapshot in the "cold" phase and a partial searchable snapshot in the "frozen" phase) ILM will re-use the existing snapshot when doing the second mount since a second snapshot is not required. Currently this is allowed for actions that use the same repository, however, multiple `searchable_snapshot` actions for the same index that use different repositories is not allowed (the ERROR state is entered). We plan to allow this in the future in subsequent work. If the `storage` option is not specified in the `searchable_snapshot` action, the mount type defaults to "shared_cache" in the frozen phase and "full_copy" in all other phases. Relates to elastic#68605
elasticmachine
pushed a commit
to costin/elasticsearch
that referenced
this pull request
Feb 9, 2021
This commit adds support for the recently introduced partial searchable snapshot (elastic#68509) to ILM. Searchable snapshot ILM actions may now be specified with a `storage` option, specifying either `full_copy` or `shared_cache` (similar to the "mount" API) to mount either a full or partial searchable snapshot: ```json PUT _ilm/policy/my_policy { "policy": { "phases": { "cold": { "actions": { "searchable_snapshot" : { "snapshot_repository" : "backing_repo", "storage": "shared_cache" } } } } } } ``` Internally, If more than one searchable snapshot action is specified (for example, a full searchable snapshot in the "cold" phase and a partial searchable snapshot in the "frozen" phase) ILM will re-use the existing snapshot when doing the second mount since a second snapshot is not required. Currently this is allowed for actions that use the same repository, however, multiple `searchable_snapshot` actions for the same index that use different repositories is not allowed (the ERROR state is entered). We plan to allow this in the future in subsequent work. If the `storage` option is not specified in the `searchable_snapshot` action, the mount type defaults to "shared_cache" in the frozen phase and "full_copy" in all other phases. Relates to elastic#68605
dakrone
added a commit
to dakrone/elasticsearch
that referenced
this pull request
Feb 9, 2021
This commit adds support for the recently introduced partial searchable snapshot (elastic#68509) to ILM. Searchable snapshot ILM actions may now be specified with a `storage` option, specifying either `full_copy` or `shared_cache` (similar to the "mount" API) to mount either a full or partial searchable snapshot: ```json PUT _ilm/policy/my_policy { "policy": { "phases": { "cold": { "actions": { "searchable_snapshot" : { "snapshot_repository" : "backing_repo", "storage": "shared_cache" } } } } } } ``` Internally, If more than one searchable snapshot action is specified (for example, a full searchable snapshot in the "cold" phase and a partial searchable snapshot in the "frozen" phase) ILM will re-use the existing snapshot when doing the second mount since a second snapshot is not required. Currently this is allowed for actions that use the same repository, however, multiple `searchable_snapshot` actions for the same index that use different repositories is not allowed (the ERROR state is entered). We plan to allow this in the future in subsequent work. If the `storage` option is not specified in the `searchable_snapshot` action, the mount type defaults to "shared_cache" in the frozen phase and "full_copy" in all other phases. Relates to elastic#68605
dakrone
added a commit
that referenced
this pull request
Feb 9, 2021
…68762) This commit adds support for the recently introduced partial searchable snapshot (#68509) to ILM. Searchable snapshot ILM actions may now be specified with a `storage` option, specifying either `full_copy` or `shared_cache` (similar to the "mount" API) to mount either a full or partial searchable snapshot: `json PUT _ilm/policy/my_policy { "policy": { "phases": { "cold": { "actions": { "searchable_snapshot" : { "snapshot_repository" : "backing_repo", "storage": "shared_cache" } } } } } } ` Internally, If more than one searchable snapshot action is specified (for example, a full searchable snapshot in the "cold" phase and a partial searchable snapshot in the "frozen" phase) ILM will re-use the existing snapshot when doing the second mount since a second snapshot is not required. Currently this is allowed for actions that use the same repository, however, multiple `searchable_snapshot` actions for the same index that use different repositories is not allowed (the ERROR state is entered). We plan to allow this in the future in subsequent work. If the `storage` option is not specified in the `searchable_snapshot` action, the mount type defaults to "shared_cache" in the frozen phase and "full_copy" in all other phases. Relates to #68605
34 tasks
ywelsch
added a commit
that referenced
this pull request
Mar 16, 2021
Reenables BWC for the searchable snapshot usage stats test. Relates #68509
ywelsch
added a commit
to ywelsch/elasticsearch
that referenced
this pull request
Mar 16, 2021
Reenables BWC for the searchable snapshot usage stats test. Relates elastic#68509
ywelsch
added a commit
that referenced
this pull request
Mar 16, 2021
ywelsch
added a commit
that referenced
this pull request
Mar 16, 2021
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
A frozen tier is backed by an external object store (like S3) and caches only a small portion of data on local disks. In this way, users can reduce hardware costs substantially for infrequently accessed data. For the frozen tier we only pull in the parts of the files that are actually needed to run a given search. Further, we don't require the node to have enough space to host all the files. We therefore have a cache that manages which file parts are available, and which ones not. This node-level shared cache is bounded in size (typically in relation to the disk size), and will evict items based on a LFU policy, as we expect some parts of the Lucene files to be used more frequently than other parts. The level of granularity for evictions is at the level of regions of a file, and does not require evicting full files. The on-disk representation that was chosen for the cold tier is not a good fit here, as it won't allow evicting parts of a file. Instead we are using fixed-size pre-allocated files and have implemented our own memory management logic to map regions of the shard's original Lucene files onto regions in these node-level shared files that are representing the on-disk cache.
This PR adds the core functionality to searchable snapshots to power such a frozen tier:
storage, which selects the kind of local storage used to acceleratesearches of the mounted index. If set to
full_copy(default, used for cold tier), each node holding a shard of the searchable snapshot index makes a full copy of the shard to its local storage. If set toshared_cache, the shard uses the newly introduced shared cache, only holding a partial copy of the index on disk (used for frozen tier).Co-authored-by: Tanguy Leroux <...>
Co-authored-by: Armin Braun <...>
Co-authored-by: David Turner <...>