OpenSearch 1.2.0 Release run Performance test cases manually

[peternied]

Due to #126, performance tests are not run via automation and instead must be started manually, here are the high level steps:

• Create performance test clusters
• Trigger performance test cases
• Confirm all tests were successful
• Publish performance test data

Follow up; Fork with all changes for the manual deployment process on private branch, https://github.com/peternied/opensearch-infra/tree/1.2.0-perf-tests

[peternied]

CDK deployment of all nodes

cdk deploy OpenSearch-1-2-0--982-arm64-disable --require-approval never
cdk deploy OpenSearch-1-2-0--982-arm64-enable --require-approval never
cdk deploy OpenSearch-1-2-0--982-x64-disable --require-approval never
cdk deploy OpenSearch-1-2-0--982-x64-enable --require-approval never

Private IP Addresses for performance test sytems

OpenSearch-1-2-0--982-arm64-disable.PrivateIp = 172.XXX.XXX.191
OpenSearch-1-2-0--982-arm64-enable.PrivateIp = 172.XXX.XXX.153
OpenSearch-1-2-0--982-x64-disable.PrivateIp = 172.XXX.XXX.81
OpenSearch-1-2-0--982-x64-enable.PrivateIp = 172.XXX.XXX.12

Trigger of performance test cases, via python client

DEBUG:root:Response body: {'testExecutionId': 'c189c624-c50c-4a1c-b234-6d933a6a8b4e'}
DEBUG:root:Response body: {'testExecutionId': 'f76c89f3-24ce-4c30-a285-209b750f3d68'}
DEBUG:root:Response body: {'testExecutionId': '423c836c-69b8-4d5f-b2db-2e13f53b9bd7'}
DEBUG:root:Response body: {'testExecutionId': 'cd07932d-f330-4ad2-88d6-9ce3b9cfb716'}

[peternied]

Test ID	dataNodeCount	masterNodeCount	dataNodeInstanceType	elasticsearchDistributionType	ec2Type	description	securityEnabled	index/opCount	index/opErrorCount	index/opErrorRate	index/requestsPerSecond/p0	index/requestsPerSecond/p50	index/requestsPerSecond/p100	index/requestsPerSecond/mean	index/latencyMillis/p50	index/latencyMillis/p90	index/latencyMillis/p99	index/latencyMillis/p100	index/latencyMillis/mean	query/opCount	query/opErrorCount	query/opErrorRate	query/requestsPerSecond/p0	query/requestsPerSecond/p50	query/requestsPerSecond/p100	query/requestsPerSecond/mean	query/latencyMillis/p50	query/latencyMillis/p90	query/latencyMillis/p99	query/latencyMillis/p100	query/latencyMillis/mean
71612843-d1dc-48ee-a8c8-40dd42e099b3	1	0	m6g.xlarge	OpenSearch 1.1	m6g.xlarge	1.1.0 without security- ARM64 - An anomaly of type: "HighErrorRate"	false	46731	0	0	33139.07031	33918.02721	40019.22266	34577.09389	2288.726649	3140.288187	4562.160723	7352.362793	2396.590689	906	0	0	2.011380672	2.017324944	2.034316818	2.019013157	256.4610302	265.1434822	268.585367	272.3462236	256.4490771
423c836c-69b8-4d5f-b2db-2e13f53b9bd7	1	0	m6g.xlarge	OpenSearch 1.2	m6g.xlarge	Performance test for build #892 without security	false	45974	0	0	41751.06641	43666.16274	50803.41016	44510.289	1775.419774	2494.474046	4009.649458	7165.874512	1881.452933	1510	0	0	1.726428151	1.741399515	1.760671973	1.741374951	468.9042711	480.6914964	487.0442009	489.0307142	469.9505166
b61240fa-52f6-4dd9-819c-4ac432f69b6a	1	0	m6g.xlarge	OpenSearch 1.1	m6g.xlarge	1.1.0 with security - ARM64	true	46135	0	0	40435.89844	42583.4836	48557.03906	43251.389	1840.582512	2409.78671	3128.396697	5320.119141	1909.018949	1510	0	0	1.749189579	1.753851861	1.767739952	1.755281913	169.8501698	177.0108099	198.4141157	205.2669571	170.2442214
f76c89f3-24ce-4c30-a285-209b750f3d68	1	0	m6g.xlarge	OpenSearch 1.2	m6g.xlarge	Performance test for build #892 with security	true	46577	0	0	35508.82422	36845.19366	42095.35938	37580.61338	2085.864094	2846.570023	4563.322096	6995.331055	2201.55233	1510	0	0	1.700284028	1.746885089	1.758681083	1.740247074	458.849396	477.6829893	488.6582034	498.0121841	463.7348075
092c4e4c-6daf-47b8-a8e0-509ba83fb68d	1	0	m5.xlarge	OpenSearch 1.1	m5.xlarge	1.1.0 without security X64 - An anomaly of type: "HighErrorRate"	false	47190	0	0	27202.90234	27913.82578	33456.10156	28529.47535	2724.511434	3741.991913	5886.840206	9118.582031	2918.882662	906	0	0	2.000571132	2.017082612	2.034460982	2.017949551	265.6344746	360.9502751	371.6241313	379.5752988	278.9982757
c189c624-c50c-4a1c-b234-6d933a6a8b4e	1	0	m5.xlarge	OpenSearch 1.2	m5.xlarge	Performance test for build #892 without security	false	46826	0	0	31416.14844	32603.31875	38654.39063	33355.57969	2364.828497	3221.024066	5780.55462	11444.31348	2523.899404	1510	0	0	1.747188342	1.751349894	1.762653327	1.752470286	378.2717085	394.0483365	408.6399221	439.290294	380.9430005
b3d8dda4-7e98-4b0c-a713-e77903063a54	1	0	m5.xlarge	OpenSearch 1.1	m5.xlarge	1.1.0 with security enabled	true	46653	0	0	34369.55859	35485.80078	41167.10938	36147.37563	2202.995803	2923.21963	4059.734399	7425.819824	2296.914207	1510	0	0	1.748971367	1.753621143	1.766879165	1.754886255	177.2797585	188.0642809	195.8570168	212.0040726	178.424238
cd07932d-f330-4ad2-88d6-9ce3b9cfb716	1	0	m5.xlarge	OpenSearch 1.2	m5.xlarge	Performance test for build #892 with security	true	46946	0	0	30978.55859	32080.78649	36967.53125	32736.42583	2403.528676	3250.280279	5576.647506	10806.48242	2550.426156	1510	0	0	1.746573925	1.750395426	1.760551703	1.751394197	383.3537136	392.0092588	403.4703856	462.6974915	384.6670792

[peternied]

Quick note on how that table was generated, you'll need access to the internal performance data store

Query for recent test data:

GET /mensor-results*/_search?filter_path=-hits.hits._source.systemUnderTest.clusterIdentity,-hits.hits._source.testResults.operations,hits.hits._source.systemUnderTest,hits.hits._source.testResults.operationsSummary,hits.hits._source.testExecutionId
{
  "query": {
    "bool": {
      "must": [
        {
          "match": {
            "testOwner": "opensearch-devops"
          }
        },
        {
          "match_phrase": {
            "systemUnderTest.clusterIdentity.clusterName": "*OpenSearch-1-2-0--982*"
          }
        }
      ]
    }
  }
}

Query for OpenSearch 1.1 test data:

GET /mensor-results*/_search?filter_path=-hits.hits._source.systemUnderTest.clusterIdentity,-hits.hits._source.testResults.operations,hits.hits._source.systemUnderTest,hits.hits._source.testResults.operationsSummary,hits.hits._source.testExecutionId
{
  "query": {
    "bool": {
      "minimum_should_match": 1, 
      "should": [
        {
          "term": {
            "testExecutionId.keyword": "b3d8dda4-7e98-4b0c-a713-e77903063a54"
          }
        },
        {
          "term": {
            "testExecutionId.keyword": "092c4e4c-6daf-47b8-a8e0-509ba83fb68d"
          }
        },
        {
          "term": {
            "testExecutionId.keyword": "b61240fa-52f6-4dd9-819c-4ac432f69b6a"
          }
        },
        {
          "term": {
            "testExecutionId.keyword": "71612843-d1dc-48ee-a8c8-40dd42e099b3"
          }
        }
      ]
    }
  }
}

• Plug the results into https://www.convertjson.com/json-to-html-table.htm
• Massage missing fields from OpenSearch 1.1 data to line up
• Add not that the OpenSearch 1.1 results without security had an anomaly
• Convert over to a markdown table https://tableconvert.com/excel-to-markdown
• Pasted into the above using the below annotations

<html>
<body>
<!--StartFragment-->

<!--EndFragment-->
</body>
</html>

[peternied]

@mch2 We have performance numbers, can you poke someone to look into the data?

[peternied]

I've created an issue in the OpenSearch repo to track the analysis