[DOC] Add sub-page for performance/benchmark runs in the main page for remote store feature

[ashking94]

What do you want to do?

• Request a change to existing documentation
• Add new documentation
• Report a technical problem with the documentation
• Other

Tell us about your request. Provide a summary of the request and all versions that are affected.

## Benchmarks

We have run benchmark for multiple workload options available within [opensearch benchmark](https://opensearch.org/docs/latest/benchmark/index/) tool and multiple cluster configuration. Below, we have summarised the benchmark results for [StackOverflow](https://github.com/opensearch-project/opensearch-benchmark-workloads/tree/main/so), [HTTP logs](https://github.com/opensearch-project/opensearch-benchmark-workloads/tree/main/http_logs) and [NYC taxis](https://github.com/opensearch-project/opensearch-benchmark-workloads/tree/main/nyc_taxis), for multiple configurations of bulk indexing client which simulates concurrency of requests.


>Your results may vary based on the cluster topology, hardware used, shard count, and merge settings.

### Cluster, shard & test configuration

* 3 Nodes with data, ingest, master role
* Node instance: AWS EC2 r6g.xlarge
* OpenSearch benchmark host: Single AWS EC2 m5.2xlarge
* Shard configuration: 3 shards with 1 replica
* repository-s3 plugin installed with default S3 settings 

### StackOverflow

The following table lists the benchmarking results for the `so` workload with the mentioned configuration and remote translog buffer interval of 250ms.

|	|	|8 bulk indexing clients (Default)	|16 bulk indexing clients	|24 bulk indexing clients	|
|---	|---	|---	|---	|---	|
|	|	|Document Replication	|remote enabled	|Percent difference	|Document Replication	|remote enabled	|Percent difference	|Document Replication	|remote enabled	|Percent difference	|
|Indexing Throughput	|Mean	|29582.5	|40667.4	|37.47	|31154.9	|47862.3	|53.63	|31777.2	|51123.2	|60.88	|
|P50	|28915.4	|40343.4	|39.52	|30406.4	|47472.5	|56.13	|30852.1	|50547.2	|63.84	|
|Indexing Latency	|P90	|1716.34	|1469.5	|-14.38	|3709.77	|2799.82	|-24.53	|5768.68	|3794.13	|-34.23	|

### HTTP logs

The following table lists the benchmarking results for the `http_logs` workload with the mentioned configuration and remote translog buffer interval of 200ms.

|	|	|8 bulk indexing clients (Default)	|16 bulk indexing clients	|24 bulk indexing clients	|
|---	|---	|---	|---	|---	|
|	|	|Document Replication	|remote enabled	|Percent difference	|Document Replication	|remote enabled	|Percent difference	|Document Replication	|remote enabled	|Percent difference	|
|Indexing Throughput	|Mean	|149062	|82198.7	|-44.86	|134696	|148749	|10.43	|133050	|197239	|48.24	|
|P50	|148123	|81656.1	|-44.87	|133591	|148859	|11.43	|132872	|197455	|48.61	|
|Indexing Latency	|P90	|327.011	|610.036	|86.55	|751.705	|669.073	|-10.99	|1145.19	|817.185	|-28.64	|

### NYC taxis

The following table lists the benchmarking results for the `http_logs` workload with the mentioned configuration and remote translog buffer interval of 250ms.

|	|	|8 bulk indexing clients (Default)	|16 bulk indexing clients	|24 bulk indexing clients	|
|---	|---	|---	|---	|---	|
|	|	|Document Replication	|remote enabled	|Percent difference	|Document Replication	|remote enabled	|Percent difference	|Document Replication	|remote enabled	|Percent difference	|
|Indexing Throughput	|Mean	|93383.9	|94186.1	|0.86	|91624.8	|125770	|37.27	|93627.7	|132006	|40.99	|
|P50	|91645.1	|93906.7	|2.47	|89659.8	|125443	|39.91	|91120.3	|132166	|45.05	|
|Indexing Latency	|P90	|995.217	|1014.01	|1.89	|2236.33	|1750.06	|-21.74	|3353.45	|2472	|-26.28	|

We see consistent gains for cases where the indexing latency is more than the average remote upload time. When we increase the bulk indexing clients, remote enabled configuration provides indexing throughput gains upto 60-65%. Detailed results are present on GitHub - https://github.com/opensearch-project/OpenSearch/issues/9790

What other resources are available? Provide links to related issues, POCs, steps for testing, etc.

[hdhalter]

@Naarcha-AWS - Can we close this?

[ashking94]

Closing this as this is done already.