Remote Store: GA in 2.10 release

[sachinpkale]

Overview

Remote store feature was introduced in OpenSearch 2.3 under experimental feature flag. The goal of the feature, as mentioned in the feature proposal (#1968), is to improve durability of the ingested data. It is ready to go GA in 2.10 release. Between 2.3 and 2.10, we added a bunch of incremental functionalities to this feature, added more unit and integration tests, ran performance tests. Following sections provides checklist to take the feature to GA.

Release Meta: #9805

Checklist

Configuration

1. Remote store feature is built on top of existing repository plugin interface. In order to enable the feature, repository plugin needs to be installed on the cluster.
2. The feature can be enabled at a cluster level by providing repository details. Once a cluster is remote store enabled, all the indices created in the cluster are remote store enabled.
3. For remote store enabled index, replication type will always be selected as Segment Replication

Functionality

1. Request level durability of the indexed data
2. Remote store integration in Replication, Failover and Peer Recovery flow
3. Interoperability of remote store with snapshot
4. Store and Restore index metadata from remote store

Meta Issues

1. [Remote Store] Experimental Release Tracking #3974
2. [Remote Store][META] Request Level Durability for for remote-backed indexes (Experimental Release Tracking for 2.5) #5671
3. [Meta] Remote Store: 2.8.0 - Release Tracking #7382
4. [Meta] Remote Store: 2.10.0 - Release Tracking #9805
5. [META] Remote Cluster State Issues #9344

Testing

1. Unit tests and Integ tests are added as part of each PR with required code coverage
2. Ran existing integ tests by enabling the remote store settings - As this is quite intrusive change to enable remote store for all the existing tests, the changes were done in private fork and carried out tests with https://github.com/sachinpkale/OpenSearch/tree/remote-store-enabled-its

Performance

• Remote store performance run for different workloads, cluster configurations and test configurations #9790

Documentation

• [DOC] Remote Store feature documentation changes for 2.10.0 release documentation-website#4875