From 71e24c17d088b03973dec827720670310ee7efb5 Mon Sep 17 00:00:00 2001 From: Michael Oviedo Date: Fri, 11 Oct 2024 12:54:35 -0700 Subject: [PATCH 1/5] add documentation for new aggregate command + auto aggregation Signed-off-by: Michael Oviedo --- _benchmark/reference/commands/aggregate.md | 81 +++++++++++++++++++ .../reference/commands/command-flags.md | 34 ++++++++ 2 files changed, 115 insertions(+) create mode 100644 _benchmark/reference/commands/aggregate.md diff --git a/_benchmark/reference/commands/aggregate.md b/_benchmark/reference/commands/aggregate.md new file mode 100644 index 0000000000..54daf0d084 --- /dev/null +++ b/_benchmark/reference/commands/aggregate.md @@ -0,0 +1,81 @@ +--- +layout: default +title: aggregate +nav_order: 85 +parent: Command reference +grand_parent: OpenSearch Benchmark Reference +redirect_from: + - /benchmark/commands/aggregate/ +--- + +# Aggregation + +OpenSearch Benchmark now supports result aggregation, allowing users to combine multiple test executions into a single aggregated result. This feature enhances the benchmarking workflow by providing a more streamlined way to conduct and analyze multiple test runs. + +## Auto-aggregation + +OpenSearch Benchmark allows users to run multiple iterations of benchmark tests and automatically aggregate the results, all within a single command. New flags have been added to the existing `execute` command to support this functionality. + +### Usage + +To use auto-aggregation, you can use the `execute` command with the new flags: + +```bash +opensearch-benchmark execute --test-iterations= --aggregate=true [other_options] +``` + +For example: +``` +opensearch-benchmark execute --test-iterations=2 --aggregate=true --workload=geonames --target-hosts=127.0.0.1:9200 +``` + +This command will run the geonames workload twice and aggregate the results. + +### New Flags + +The following new flags have been added to support auto-aggregation: + +- `--test-iterations`: Specifies the number of times to run the workload (default: 1). +- `--aggregate`: Determines whether to aggregate the results of multiple test executions (default: true). +- `--sleep-timer`: Specifies the number of seconds to sleep before starting the next test execution (default: 5). +- `--cancel-on-error`: When set, stops executing tests if an error occurs in one of the test iterations (default: false). +All existing arguments and flags for test execution remain compatible with these new options, allowing users to customize their benchmark runs as needed. For instance: +``` +opensearch-benchmark execute --test-iterations=2 --aggregate=true --workload=geonames --target-hosts=127.0.0.1:9200 --test-mode --kill-running-processes +``` + +## Manual Aggregation + +In addition to auto-aggregation, OpenSearch Benchmark also allows users to manually aggregate results from multiple test executions that have already been run. + +### Usage + +To aggregate multiple test executions manually, you can use the aggregate command: +``` +opensearch-benchmark aggregate --test-executions=,,... +``` + +Sample Output +``` + ____ _____ __ ____ __ __ + / __ \____ ___ ____ / ___/___ ____ ___________/ /_ / __ )___ ____ _____/ /_ ____ ___ ____ ______/ /__ + / / / / __ \/ _ \/ __ \\__ \/ _ \/ __ `/ ___/ ___/ __ \ / __ / _ \/ __ \/ ___/ __ \/ __ `__ \/ __ `/ ___/ //_/ +/ /_/ / /_/ / __/ / / /__/ / __/ /_/ / / / /__/ / / / / /_/ / __/ / / / /__/ / / / / / / / / /_/ / / / ,< +\____/ .___/\___/_/ /_/____/\___/\__,_/_/ \___/_/ /_/ /_____/\___/_/ /_/\___/_/ /_/_/ /_/ /_/\__,_/_/ /_/|_| + /_/ + +Aggregate test execution ID: aggregate_results_geonames_9aafcfb8-d3b7-4583-864e-4598b5886c4f + +------------------------------- +[INFO] SUCCESS (took 1 seconds) +------------------------------- +``` + +The results will be aggregated into one test execution and stored under the ID shown in the output. + +Additional Options + +- `--test-execution-id`: Define a unique ID for the aggregated test execution. +- `--results-file`: Write the aggregated results to the provided file. +- `--workload-repository`: Define the repository from where OpenSearch Benchmark will load workloads (default: default). + diff --git a/_benchmark/reference/commands/command-flags.md b/_benchmark/reference/commands/command-flags.md index 6520f80803..aecc30f289 100644 --- a/_benchmark/reference/commands/command-flags.md +++ b/_benchmark/reference/commands/command-flags.md @@ -328,3 +328,37 @@ Sets what fraction of randomized query values can be repeated. Takes values betw Sets how many distinct repeatable pair values are generated for each operation when randomization is used. Default is `5000`. This setting does not work when `--randomization-enabled` is not used. + + +## test-iterations + + +Specifies the number of times to run the workload. This flag accepts an integer value. Default is `1`. + +Example: `--test-iterations=3` will run the workload three times. + + +## aggregate + + +Determines whether OpenSearch Benchmark should aggregate the results of multiple test executions. This flag accepts a boolean value (true/false). Default is `true`. + +When set to `true`, OpenSearch Benchmark will combine the results from all iterations into a single aggregated report. When set to `false`, results from each iteration will be reported separately. + +Example: `--aggregate=false` + + +## sleep-timer + + +Specifies the number of seconds to sleep before starting the next test execution. This flag accepts an integer value. Default is `5`. + +Example: `--sleep-timer=10` will cause OpenSearch Benchmark to wait 10 seconds between test iterations. + + +## cancel-on-error + + +When set, this flag instructs OpenSearch Benchmark to stop executing tests if an error occurs in one of the test iterations. This is a boolean flag and does not require a value. Default is `false` (not set). + +Example: `--cancel-on-error` From e6026365b5f3dba5edddc0af25ed5e99482d9951 Mon Sep 17 00:00:00 2001 From: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Date: Wed, 16 Oct 2024 15:31:36 -0500 Subject: [PATCH 2/5] Update aggregate.md Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> --- _benchmark/reference/commands/aggregate.md | 42 +++++++++++----------- 1 file changed, 20 insertions(+), 22 deletions(-) diff --git a/_benchmark/reference/commands/aggregate.md b/_benchmark/reference/commands/aggregate.md index 54daf0d084..618313a7fa 100644 --- a/_benchmark/reference/commands/aggregate.md +++ b/_benchmark/reference/commands/aggregate.md @@ -8,54 +8,52 @@ redirect_from: - /benchmark/commands/aggregate/ --- -# Aggregation +# aggregate -OpenSearch Benchmark now supports result aggregation, allowing users to combine multiple test executions into a single aggregated result. This feature enhances the benchmarking workflow by providing a more streamlined way to conduct and analyze multiple test runs. +The `aggregate` command combines multiple test executions into a single aggregated result, providing a more streamlined way to conduct and analyze multiple test runs. There are two methods of aggregation: + +- [Auto-aggregation](#auto-aggregation) +- [Manual aggregation(#manual-aggregation) ## Auto-aggregation -OpenSearch Benchmark allows users to run multiple iterations of benchmark tests and automatically aggregate the results, all within a single command. New flags have been added to the existing `execute` command to support this functionality. +Auto-aggregation runs multiple iterations of benchmark tests and automatically aggregate the results, all within a single command. You can use the flags outlined in this with the `execute` command. ### Usage -To use auto-aggregation, you can use the `execute` command with the new flags: +The following example runs the `geonames` workload and aggregates the results twice: ```bash -opensearch-benchmark execute --test-iterations= --aggregate=true [other_options] -``` - -For example: -``` opensearch-benchmark execute --test-iterations=2 --aggregate=true --workload=geonames --target-hosts=127.0.0.1:9200 ``` +{% include copy-curl.html %} -This command will run the geonames workload twice and aggregate the results. +### Auto-aggregation flags -### New Flags - -The following new flags have been added to support auto-aggregation: +The following new flags can be used to customized auto-aggregation: - `--test-iterations`: Specifies the number of times to run the workload (default: 1). - `--aggregate`: Determines whether to aggregate the results of multiple test executions (default: true). - `--sleep-timer`: Specifies the number of seconds to sleep before starting the next test execution (default: 5). - `--cancel-on-error`: When set, stops executing tests if an error occurs in one of the test iterations (default: false). -All existing arguments and flags for test execution remain compatible with these new options, allowing users to customize their benchmark runs as needed. For instance: -``` -opensearch-benchmark execute --test-iterations=2 --aggregate=true --workload=geonames --target-hosts=127.0.0.1:9200 --test-mode --kill-running-processes -``` -## Manual Aggregation +## Manual aggregation -In addition to auto-aggregation, OpenSearch Benchmark also allows users to manually aggregate results from multiple test executions that have already been run. +Use the `aggregate` command to manually aggregate results from multiple test executions: ### Usage To aggregate multiple test executions manually, you can use the aggregate command: -``` + +```bash opensearch-benchmark aggregate --test-executions=,,... ``` +{% include copy-curl.html %} + +### Response + +OpenSearch Benchmark responds with the following results: -Sample Output ``` ____ _____ __ ____ __ __ / __ \____ ___ ____ / ___/___ ____ ___________/ /_ / __ )___ ____ _____/ /_ ____ ___ ____ ______/ /__ @@ -73,7 +71,7 @@ Aggregate test execution ID: aggregate_results_geonames_9aafcfb8-d3b7-4583-864e The results will be aggregated into one test execution and stored under the ID shown in the output. -Additional Options + - `--test-execution-id`: Define a unique ID for the aggregated test execution. - `--results-file`: Write the aggregated results to the provided file. From 87aba137391469ec2f7523ac82378606d2f235dd Mon Sep 17 00:00:00 2001 From: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Date: Wed, 16 Oct 2024 15:34:51 -0500 Subject: [PATCH 3/5] Update command-flags.md Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> --- _benchmark/reference/commands/command-flags.md | 16 ++++++---------- 1 file changed, 6 insertions(+), 10 deletions(-) diff --git a/_benchmark/reference/commands/command-flags.md b/_benchmark/reference/commands/command-flags.md index aecc30f289..96948e45c7 100644 --- a/_benchmark/reference/commands/command-flags.md +++ b/_benchmark/reference/commands/command-flags.md @@ -333,32 +333,28 @@ Sets how many distinct repeatable pair values are generated for each operation w ## test-iterations -Specifies the number of times to run the workload. This flag accepts an integer value. Default is `1`. - -Example: `--test-iterations=3` will run the workload three times. +Specifies the number of times to run the workload. Default is `1`. ## aggregate -Determines whether OpenSearch Benchmark should aggregate the results of multiple test executions. This flag accepts a boolean value (true/false). Default is `true`. +Determines whether OpenSearch Benchmark should aggregate the results of multiple test executions. -When set to `true`, OpenSearch Benchmark will combine the results from all iterations into a single aggregated report. When set to `false`, results from each iteration will be reported separately. +When set to `true`, OpenSearch Benchmark will combine the results from all iterations into a single aggregated report. When set to `false`, results from each iteration will be reported separately. -Example: `--aggregate=false` +Default is `true`. ## sleep-timer -Specifies the number of seconds to sleep before starting the next test execution. This flag accepts an integer value. Default is `5`. +Specifies the number of seconds to sleep before starting the next test execution. Default is `5`. -Example: `--sleep-timer=10` will cause OpenSearch Benchmark to wait 10 seconds between test iterations. ## cancel-on-error -When set, this flag instructs OpenSearch Benchmark to stop executing tests if an error occurs in one of the test iterations. This is a boolean flag and does not require a value. Default is `false` (not set). +When set, this flag instructs OpenSearch Benchmark to stop executing tests if an error occurs in one of the test iterations. Default is `false` (not set). -Example: `--cancel-on-error` From 75ff23be2f6c263725be011d4e127fd1590b0d3b Mon Sep 17 00:00:00 2001 From: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Date: Thu, 17 Oct 2024 05:45:16 -0500 Subject: [PATCH 4/5] Apply suggestions from code review Co-authored-by: Nathan Bower Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> --- _benchmark/reference/commands/aggregate.md | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/_benchmark/reference/commands/aggregate.md b/_benchmark/reference/commands/aggregate.md index 618313a7fa..63c5fc0c99 100644 --- a/_benchmark/reference/commands/aggregate.md +++ b/_benchmark/reference/commands/aggregate.md @@ -13,11 +13,11 @@ redirect_from: The `aggregate` command combines multiple test executions into a single aggregated result, providing a more streamlined way to conduct and analyze multiple test runs. There are two methods of aggregation: - [Auto-aggregation](#auto-aggregation) -- [Manual aggregation(#manual-aggregation) +- [Manual aggregation](#manual-aggregation) ## Auto-aggregation -Auto-aggregation runs multiple iterations of benchmark tests and automatically aggregate the results, all within a single command. You can use the flags outlined in this with the `execute` command. +The auto-aggregation method runs multiple iterations of benchmark tests and automatically aggregates the results, all within a single command. You can use the flags outlined in this with the `execute` command. ### Usage @@ -30,16 +30,16 @@ opensearch-benchmark execute --test-iterations=2 --aggregate=true --workload=geo ### Auto-aggregation flags -The following new flags can be used to customized auto-aggregation: +The following new flags can be used to customize the auto-aggregation method: -- `--test-iterations`: Specifies the number of times to run the workload (default: 1). -- `--aggregate`: Determines whether to aggregate the results of multiple test executions (default: true). -- `--sleep-timer`: Specifies the number of seconds to sleep before starting the next test execution (default: 5). -- `--cancel-on-error`: When set, stops executing tests if an error occurs in one of the test iterations (default: false). +- `--test-iterations`: Specifies the number of times to run the workload (default is `1`). +- `--aggregate`: Determines whether to aggregate the results of multiple test executions (default is `true`). +- `--sleep-timer`: Specifies the number of seconds to sleep before starting the next test execution (default is `5`). +- `--cancel-on-error`: When set, stops executing tests if an error occurs in one of the test iterations (default is `false`). ## Manual aggregation -Use the `aggregate` command to manually aggregate results from multiple test executions: +You can use the `aggregate` command to manually aggregate results from multiple test executions. ### Usage @@ -52,7 +52,7 @@ opensearch-benchmark aggregate --test-executions=, Date: Thu, 17 Oct 2024 05:46:20 -0500 Subject: [PATCH 5/5] Update aggregate.md Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> --- _benchmark/reference/commands/aggregate.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/_benchmark/reference/commands/aggregate.md b/_benchmark/reference/commands/aggregate.md index 63c5fc0c99..17612f1164 100644 --- a/_benchmark/reference/commands/aggregate.md +++ b/_benchmark/reference/commands/aggregate.md @@ -43,7 +43,7 @@ You can use the `aggregate` command to manually aggregate results from multiple ### Usage -To aggregate multiple test executions manually, you can use the aggregate command: +To aggregate multiple test executions manually, specify the `test_execution_ids` you would like to aggregate, as shown in the following example: ```bash opensearch-benchmark aggregate --test-executions=,,... @@ -71,8 +71,6 @@ Aggregate test execution ID: aggregate_results_geonames_9aafcfb8-d3b7-4583-864e The results will be aggregated into one test execution and stored under the ID shown in the output: - - - `--test-execution-id`: Define a unique ID for the aggregated test execution. - `--results-file`: Write the aggregated results to the provided file. - `--workload-repository`: Define the repository from which OpenSearch Benchmark will load workloads (default is `default`).