[Feature Request] Add index setting for custom remote store path component

[msfroh]

Is your feature request related to a problem? Please describe

As part of #17957, we would like to be able to set up redundant shard writers for availability and to support rolling upgrades (as described in this comment).

In order to do that, we need the ability to dynamically configure searchers that replicate from specific writers. Unfortunately, based on a PoC that I did using a filesystem-based repository, it looks like there's no identifier for the writer node in the remote store paths, which look like:

remote_repository/111001101110010/5PuruvK_S463AQam0Oe1aw/0/segments/data/_0.cfs__M_V1zZcBij817kr599dh
remote_repository/111001101110010/5PuruvK_S463AQam0Oe1aw/0/segments/data/_0.si__MfV1zZcBij817kr599dc
remote_repository/111001101110010/5PuruvK_S463AQam0Oe1aw/0/segments/data/segments_3__LfV1zZcBij817kr599dE
remote_repository/111001101110010/5PuruvK_S463AQam0Oe1aw/0/segments/data/_0.cfe__L_V1zZcBij817kr599dY

The 111001101110010 part is a hash prefix based on the same attributes used to build the other parts of the path (AFAIK). The 5PuruvK_S463AQam0Oe1aw part is the index UUID. The 0 is the shard ID. I believe this means that two writers for the same shard (not normally possible in an OpenSearch cluster, but doable in our "clusterless" configuration) would try writing to the same path and potentially step on each others toes.

Describe the solution you'd like

I've been trying to think of a good solution and the best idea I've had so far is to inject a unique identifier (maybe node name, ID, or ephemeral ID -- probably not ephemeral ID, because I don't think I want it changing on restart) for each writer, so the resulting paths look something like:

remote_repository/111001101110010/5PuruvK_S463AQam0Oe1aw/0/<writer-node-id>/segments/data/_0.cfs__M_V1zZcBij817kr599dh
remote_repository/111001101110010/5PuruvK_S463AQam0Oe1aw/0/<writer-node-id>/segments/data/_0.si__MfV1zZcBij817kr599dc
remote_repository/111001101110010/5PuruvK_S463AQam0Oe1aw/0/<writer-node-id>/segments/data/segments_3__LfV1zZcBij817kr599dE
remote_repository/111001101110010/5PuruvK_S463AQam0Oe1aw/0/<writer-node-id>/segments/data/_0.cfe__L_V1zZcBij817kr599dY

Or maybe:

remote_repository/111001101110010/<writer-node-id>/5PuruvK_S463AQam0Oe1aw/0/segments/data/_0.cfs__M_V1zZcBij817kr599dh
remote_repository/111001101110010/<writer-node-id>/5PuruvK_S463AQam0Oe1aw/0/segments/data/_0.si__MfV1zZcBij817kr599dc
remote_repository/111001101110010/<writer-node-id>/5PuruvK_S463AQam0Oe1aw/0/segments/data/segments_3__LfV1zZcBij817kr599dE
remote_repository/111001101110010/<writer-node-id>/5PuruvK_S463AQam0Oe1aw/0/segments/data/_0.cfe__L_V1zZcBij817kr599dY

(I don't have a strong preference for whether the writer node identifier is at the beginning or end of the path.)

Unlike the existing remote store paths, I would like this path component to be more dynamic, so I can add a shard to a running searcher and tell it which writer to replicate from. By extension, I can add searchers that replicate from a particular writer (or set of writers) to a given coordinator's routing table.

I believe that I can achieve that by adding a new index setting (similar to cluster.remote_store.segments.path.prefix, but at the index level) that is null or the empty string by default. For existing remote store use-cases, we don't need to inject this new path component. For my team's uses (or anyone else who wants to get on the clusterless train once it's ready), the setting is there and waiting to be populated.

Specifically, I'm proposing that we add an index setting as follows:

    public static final Setting<String> INDEX_REMOTE_STORE_SEGMENT_PATH_PREFIX = Setting.simpleString(
        "index.remote_store.segment.path_prefix",
        Property.IndexScope,
        Property.InternalIndex
    );

Related component

Storage:Remote

Describe alternatives you've considered

Index UUIDs

Since the clusterless implementation involves cluster state that's created on each OpenSearch node, I considered the idea of using different index UUIDs for shards written by different writers. Unfortunately, the index UUID needs to get propagated all the way to the coordinator, who would now see searchers that replicate from different writers as hosting different indexes. Coordinators shouldn't care which writer a given searcher replicates from (unless our control plane tells them to exclude specific searchers -- but that's a control plane concern, not a coordinator concern).

Static configuration

I also thought about using one of the existing remote store static settings (like cluster.remote_store.segments.path.prefix), but that would involve starting searchers only after the writers have started, so that they replicate from specific writers. If I want to flip a searcher to replicate from another writer, I would need to remove it from coordinators, remove shards from the searcher, shut down the process, update the static config in opensearch.yml, start it up, add shards back to the searcher, wait for it to replicate, and then add it back to the coordinator. The dynamic setting lets me replace the three middle steps (stop process, update config, start process) with "update index setting on searcher", which is an easier task for our control plane.

Static named repositories

Finally, I had a terrible idea where I'd statically configure a bunch of repositories over the same bucket (with boring names like repo1, repo2, .... repoN) and associate different writers with different repos (in their index settings) and mirror those repo-selector settings to the index settings on searchers to pick which writer to replicate from. Functionally, it's kind of like what I'm proposing here, but I feel like just adding an explicit index-level remote store path prefix setting is much cleaner.

Additional context

No response

[rgsriram]

@msfroh - can you please assign this issue to me?.