Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adds command line argument to specify throughput percentiles to display #449

Merged
merged 1 commit into from
Jan 30, 2024

Conversation

peteralfonsi
Copy link
Contributor

Description

Adds a new optional argument, --throughput-percentiles. Similar to #441, the user can send a comma-separated list of throughput percentiles they'd like to see displayed at the end of the run. The original stats (max, min, median, mean) are always displayed regardless of --throughput-percentiles.

Example usage:
opensearch-benchmark execute-test --pipeline=benchmark-only --workload-path=/home/ec2-user/osb/opensearch-benchmark-workloads/modified_nyc_taxis --target-host=http://localhost:9200/ --throughput-percentiles 10,50,75

Result (trimmed):

| Min Throughput | cheap-dropoff | 85.02 | ops/s |
| Mean Throughput | cheap-dropoff | 85.03 | ops/s |
| Median Throughput | cheap-dropoff | 85.03 | ops/s |
| Max Throughput | cheap-dropoff | 85.07 | ops/s |
| 10th percentile throughput | cheap-dropoff | 85.0169 | ms |
| 50th percentile throughput | cheap-dropoff | 85.0263 | ms |
| 75th percentile throughput | cheap-dropoff | 85.0425 | ms |
| 50th percentile latency | cheap-dropoff | 1.72014 | ms |
| 90th percentile latency | cheap-dropoff | 2.14881 | ms |
| 99th percentile latency | cheap-dropoff | 3.26386 | ms |
| 99.9th percentile latency | cheap-dropoff | 3.87494 | ms |
| 100th percentile latency | cheap-dropoff | 4.03645 | ms |
| 50th percentile service time | cheap-dropoff | 0.956658 | ms |
| 90th percentile service time | cheap-dropoff | 1.15708 | ms |
| 99th percentile service time | cheap-dropoff | 1.62473 | ms |
| 99.9th percentile service time | cheap-dropoff | 1.86019 | ms |
| 100th percentile service time | cheap-dropoff | 1.95773 | ms |
| error rate | cheap-dropoff | 0 | % |

(Note - if the user is specifying percentiles, I'm assuming they want accuracy, so I left it displaying all 4 decimal places. But if you guys think I should change it to 2 places to align with min/median/mean/max, I can change that)

Issues Resolved

Resolves #261

Testing

  • New functionality includes testing

Changes to percentile sample size function were tested in its UT. Manually tested on a workload, with all combinations of throughput and latency percentiles being on and off, to ensure the values were as expected.


By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Peter Alfonsi <petealft@amazon.com>
Copy link
Collaborator

@IanHoang IanHoang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Thanks for doing this @peteralfonsi

Since --throughput-percentiles and --latency-percentiles are new command line arguments, we should also have this reflected in the reference guide for documentation website. It's not needed ASAP but will be needed by the time of our next release. I'll create an issue for this

@IanHoang IanHoang merged commit c74c993 into opensearch-project:main Jan 30, 2024
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Additional percentile statistics for throughput metrics
2 participants