[Meta] Remote store-based warm index

[andrross]

Goals

Create a proof-of-concept that shows end-to-end functionality of a remote-backed index where the data may not all reside locally and can be fetched on-demand from the remote store when necessary. This is the initial implementation of the feature described in #6528. This will build upon the design and prototype started in #7331 in order to demonstrate an end-to-end capability.

This code touches much of the same code as the remote store feature, which is nearing promotion out from behind a feature flag. In order to avoid complicating that effort in the immediate short term, we’ll start development on a feature branch. Once remote store is no longer behind a feature flag, then we’ll move this effort from the feature branch to behind a feature flag on main.

Non-goals

Make final decisions on naming or APIs. The term “warm” is used extensively here as that is a sort of term-of-art and is generally well understood, but one of the larger goals of these efforts is to remove the need for users to think about discrete storage tiers and allow the system to more intelligently optimize based on usage patterns.

Tasks

• Implement option to create a new warm remote index (with zero replicas) #8447
• Restore file cache state upon restart of a node that contains a warm remote index #8448
• Handle shard relocation for a warm remote index (add node, shard rebalancing, etc) #8449
• Implement support for replicas with a warm remote index #8450
• [Warm Remote PoC] Remove hard-coded composite directory from refresh listener #8802
• [Warm Remote PoC] Add integration tests to test file-offload and block-based download functionality #8803
• [Warm Remote PoC] Evalute integrating the new file tracker with o.o.index.translog.transfer.FileTransferTracker #8804

---

The above tasks are the initial priority for building the basic functionality. After that, we will implement the functionality described below to dynamically change the "warm" property on an index:

• Implement support for adding “warm” setting to hot index (hot-to-warm)
  • All files on local disk are logically moved to the file cache (no physical movement). If the file cache is filled, then files will be deleted from local disk based on a TBD heuristic.
  • The index will remain readable and writable with no extended downtime.
• Implement support for removing “warm” setting from warm index (warm-to-hot)
  • All block-based files will be deleted from disk and complete files will be restored from the remote store.
  • The index will remain readable and writable with no extended downtime.

[anasalkouz]

All block-based files will be deleted from disk and complete files will be restored from the remote store.

Why this is required since we will have the hybrid directory and we can read from both complete files and block-based files?

[ankitkala]

Issues for FileCache+Block level fetch(apart from the ones already mentioned in the description):

• FileCache integration with warm indices #11789
• Implement CacheLifecycleManager for the FileCache #11790
• [Observability] Expose the FileCache stats & metrics  #11791
• Support for encryption with FileCache #11792
• Working set computation for warm shard #11793

[ankitkala]

Here is the sorted list of tasks we have to start the efforts on writable warm.

• Add initial bootstrap code for warm primary (issue1 issue2).
  • Uses file cache & composite directory.
  • Implement IndexInput for local files with file cache.
• Support for replicas (issue).
  • Replica recovery flow (start replica with partial data on remote)
  • Supporting segment replication on warm replicas.
    • Prefetch optimizations for replica (data needed to open a reader).
    • Add support for remote store backed index inputs.
• Primary promotion (replica to primary, primary relocation) (issue).
  • Testing write path on primary with tiered data.
• Support encryption with writable warm(issue)
• Prefetch optimizations (TBD).
• Support for File cache evictions. TTL & overflow criteria based.
  • open questions: how long to retain the new files locally on primary.
  • Overflow criteria for File Cache.
• [Observability] Expose stats & metrics for FileCache (issue).

[ns-sladani]

Hey @andrross,

Thank you for your & team work on the search functionality for remote-backed indexes. Could you please provide an estimated timeline (ETA) for the release of this feature? I have reviewed several GitHub issues, including #6528 and #8446, but did not find an ETA mentioned.

We are planning to implement remote-backed indexes for our on-premises OpenSearch deployment, and this feature will significantly help us reduce local disk storage by leveraging AWS S3 object storage.

Thank you once again! Very excited for this feature.

[andrross]

Hey @ns-sladani, thanks for your interest! Unfortunately I don't have an answer to your question as I am not actively working on this feature myself. Tagging some others who might be able to help: @ankitkala @rayshrey