Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Remote store performance run for different workloads, cluster configurations and test configurations #9790

Closed
ashking94 opened this issue Sep 6, 2023 · 5 comments
Closed

Remote store performance run for different workloads, cluster configurations and test configurations #9790

ashking94 opened this issue Sep 6, 2023 · 5 comments
Labels
discuss Issues intended to help drive brainstorming and decision making enhancement Enhancement or improvement to existing feature or request Storage:Durability Issues and PRs related to the durability framework v2.10.0

Comments

@ashking94
Copy link
Member

ashking94 commented Sep 6, 2023 •
edited
Loading

We have run OSB for different workloads with the different configurations as below. We are able to see consistent improvements in the throughput for remote store runs.

Configuration 1 - Using r5.large, 6 shards 1 replica

  • 6 Shards 1 Replica
  • Cluster Configuration - 3 Nodes with data, ingest, master role
  • Instance Details - AWS EC2 r5.large
  • OSB host - Single AWS EC2 m5.2xlarge

Run Summary

Workload Index type Refresh interval Remote translog Buffer Interval Bulk client / Bulk Size Mean Indexing Throughput p50 Indexing Throughput Indexing Latency in ms (P90) Workload-params
so docrep 1s - 24/Default 13968.1 14280.9 12749.4 {"bulk_indexing_clients":"24","number_of_shards":"6","number_of_replicas":"1"}
so docrep 1s - 24/Default 14376.1 14440.5 13027.4 {"bulk_indexing_clients":"24","number_of_shards":"6","number_of_replicas":"1"}
so remote 10s Default (650ms) 24/Default 20013.9 19839.6 8500.81 {"bulk_indexing_clients":"24","number_of_shards":"6","number_of_replicas":"1","index_settings":{"index":{"refresh_interval":"10s"}}}
so remote 10s Default (650ms) 24/Default 19317.5 19033 8656.1 {"bulk_indexing_clients":"24","number_of_shards":"6","number_of_replicas":"1","index_settings":{"index":{"refresh_interval":"10s"}}}
http_logs docrep 1s - 24/Default 64797 65568.8 2708.08 {"bulk_indexing_clients":"24","number_of_shards":"6","number_of_replicas":"1"}
http_logs docrep 1s - 24/Default 67049.1 67361.2 2607.16 {"bulk_indexing_clients":"24","number_of_shards":"6","number_of_replicas":"1"}
http_logs remote 10s Default (650ms) 24/Default 83155.7 84021.1 2089.09 {"bulk_indexing_clients":"24","number_of_shards":"6","number_of_replicas":"1","index_settings":{"index":{"refresh_interval":"10s"}}}
http_logs remote 10s Default (650ms) 24/Default 78207.1 79566.2 2104.93 {"bulk_indexing_clients":"24","number_of_shards":"6","number_of_replicas":"1","index_settings":{"index":{"refresh_interval":"10s"}}}
nyc_taxis docrep - - 24/Default 35023.6 34574.6 9385.54 {"bulk_indexing_clients":"24","number_of_shards":"6","number_of_replicas":"1"}
nyc_taxis docrep - - 24/Default 36522.6 35788.6 9628.57 {"bulk_indexing_clients":"24","number_of_shards":"6","number_of_replicas":"1"}
nyc_taxis remote - Default (650ms) 24/Default 53046.1 52422.5 5978.72 {"bulk_indexing_clients":"24","number_of_shards":"6","number_of_replicas":"1",}
nyc_taxis remote - Default (650ms) 24/Default 51803.2 51571.8 6319.26 {"bulk_indexing_clients":"24","number_of_shards":"6","number_of_replicas":"1",}

Appendix

Abbreviations

bic - bulk indexing client
rbi - remote translog buffer interval
bs - bulk size
ri - refresh interval

Results for configuration 1

so, docrep, bic-24 (Run 1)

|                                                         Metric |                     Task |       Value |   Unit |
|---------------------------------------------------------------:|-------------------------:|------------:|-------:|
|                     Cumulative indexing time of primary shards |                          |     114.455 |    min |
|             Min cumulative indexing time across primary shards |                          |     17.2207 |    min |
|          Median cumulative indexing time across primary shards |                          |     19.3778 |    min |
|             Max cumulative indexing time across primary shards |                          |     21.4053 |    min |
|            Cumulative indexing throttle time of primary shards |                          |           0 |    min |
|    Min cumulative indexing throttle time across primary shards |                          |           0 |    min |
| Median cumulative indexing throttle time across primary shards |                          |           0 |    min |
|    Max cumulative indexing throttle time across primary shards |                          |           0 |    min |
|                        Cumulative merge time of primary shards |                          |     26.7914 |    min |
|                       Cumulative merge count of primary shards |                          |          25 |        |
|                Min cumulative merge time across primary shards |                          |     1.87663 |    min |
|             Median cumulative merge time across primary shards |                          |     4.68072 |    min |
|                Max cumulative merge time across primary shards |                          |     6.18075 |    min |
|               Cumulative merge throttle time of primary shards |                          |     9.04755 |    min |
|       Min cumulative merge throttle time across primary shards |                          |    0.703417 |    min |
|    Median cumulative merge throttle time across primary shards |                          |     1.50548 |    min |
|       Max cumulative merge throttle time across primary shards |                          |     2.11802 |    min |
|                      Cumulative refresh time of primary shards |                          |      24.988 |    min |
|                     Cumulative refresh count of primary shards |                          |         197 |        |
|              Min cumulative refresh time across primary shards |                          |     3.12418 |    min |
|           Median cumulative refresh time across primary shards |                          |     4.40324 |    min |
|              Max cumulative refresh time across primary shards |                          |     4.56643 |    min |
|                        Cumulative flush time of primary shards |                          |     16.6149 |    min |
|                       Cumulative flush count of primary shards |                          |          70 |        |
|                Min cumulative flush time across primary shards |                          |     1.90595 |    min |
|             Median cumulative flush time across primary shards |                          |     2.70427 |    min |
|                Max cumulative flush time across primary shards |                          |     3.84727 |    min |
|                                        Total Young Gen GC time |                          |       61.17 |      s |
|                                       Total Young Gen GC count |                          |         642 |        |
|                                          Total Old Gen GC time |                          |           0 |      s |
|                                         Total Old Gen GC count |                          |           0 |        |
|                                                     Store size |                          |     70.8439 |     GB |
|                                                  Translog size |                          | 6.14673e-07 |     GB |
|                                         Heap used for segments |                          |           0 |     MB |
|                                       Heap used for doc values |                          |           0 |     MB |
|                                            Heap used for terms |                          |           0 |     MB |
|                                            Heap used for norms |                          |           0 |     MB |
|                                           Heap used for points |                          |           0 |     MB |
|                                    Heap used for stored fields |                          |           0 |     MB |
|                                                  Segment count |                          |         217 |        |
|                                                 Min Throughput |             index-append |     12459.7 | docs/s |
|                                                Mean Throughput |             index-append |     13968.1 | docs/s |
|                                              Median Throughput |             index-append |     14280.9 | docs/s |
|                                                 Max Throughput |             index-append |       16311 | docs/s |
|                                        50th percentile latency |             index-append |     9352.44 |     ms |
|                                        90th percentile latency |             index-append |     12749.4 |     ms |
|                                        99th percentile latency |             index-append |     21783.8 |     ms |
|                                      99.9th percentile latency |             index-append |       35196 |     ms |
|                                     99.99th percentile latency |             index-append |             |        |
|                                       100th percentile latency |             index-append |       50567 |     ms |
|                                   50th percentile service time |             index-append |     9352.44 |     ms |
|                                   90th percentile service time |             index-append |     12749.4 |     ms |
|                                   99th percentile service time |             index-append |     21783.8 |     ms |
|                                 99.9th percentile service time |             index-append |       35196 |     ms |
|                                99.99th percentile service time |             index-append |             |        |
|                                  100th percentile service time |             index-append |       50567 |     ms |
|                                                     error rate |             index-append |           0 |      % |
|                                                 Min Throughput | wait-until-merges-finish |        0.02 |  ops/s |
|                                                Mean Throughput | wait-until-merges-finish |        0.02 |  ops/s |
|                                              Median Throughput | wait-until-merges-finish |        0.02 |  ops/s |
|                                                 Max Throughput | wait-until-merges-finish |        0.02 |  ops/s |
|                                        50th percentile latency | wait-until-merges-finish |             |        |
|                                        90th percentile latency | wait-until-merges-finish |             |        |
|                                        99th percentile latency | wait-until-merges-finish |             |        |
|                                      99.9th percentile latency | wait-until-merges-finish |             |        |
|                                     99.99th percentile latency | wait-until-merges-finish |             |        |
|                                       100th percentile latency | wait-until-merges-finish |     55241.8 |     ms |
|                                   50th percentile service time | wait-until-merges-finish |             |        |
|                                   90th percentile service time | wait-until-merges-finish |             |        |
|                                   99th percentile service time | wait-until-merges-finish |             |        |
|                                 99.9th percentile service time | wait-until-merges-finish |             |        |
|                                99.99th percentile service time | wait-until-merges-finish |             |        |
|                                  100th percentile service time | wait-until-merges-finish |     55241.8 |     ms |
|                                                     error rate | wait-until-merges-finish |           0 |      % |

so, docrep, bic-24 (Run 2)

|                                                         Metric |                     Task |       Value |   Unit |
|---------------------------------------------------------------:|-------------------------:|------------:|-------:|
|                     Cumulative indexing time of primary shards |                          |     110.292 |    min |
|             Min cumulative indexing time across primary shards |                          |     17.1682 |    min |
|          Median cumulative indexing time across primary shards |                          |      18.236 |    min |
|             Max cumulative indexing time across primary shards |                          |     20.0695 |    min |
|            Cumulative indexing throttle time of primary shards |                          |           0 |    min |
|    Min cumulative indexing throttle time across primary shards |                          |           0 |    min |
| Median cumulative indexing throttle time across primary shards |                          |           0 |    min |
|    Max cumulative indexing throttle time across primary shards |                          |           0 |    min |
|                        Cumulative merge time of primary shards |                          |     25.2738 |    min |
|                       Cumulative merge count of primary shards |                          |          27 |        |
|                Min cumulative merge time across primary shards |                          |     1.33825 |    min |
|             Median cumulative merge time across primary shards |                          |     4.55728 |    min |
|                Max cumulative merge time across primary shards |                          |     5.59335 |    min |
|               Cumulative merge throttle time of primary shards |                          |     9.53137 |    min |
|       Min cumulative merge throttle time across primary shards |                          |    0.549917 |    min |
|    Median cumulative merge throttle time across primary shards |                          |     1.52705 |    min |
|       Max cumulative merge throttle time across primary shards |                          |     2.57405 |    min |
|                      Cumulative refresh time of primary shards |                          |     25.6115 |    min |
|                     Cumulative refresh count of primary shards |                          |         201 |        |
|              Min cumulative refresh time across primary shards |                          |     3.76517 |    min |
|           Median cumulative refresh time across primary shards |                          |     4.09333 |    min |
|              Max cumulative refresh time across primary shards |                          |      5.5187 |    min |
|                        Cumulative flush time of primary shards |                          |     16.8697 |    min |
|                       Cumulative flush count of primary shards |                          |          69 |        |
|                Min cumulative flush time across primary shards |                          |     2.16655 |    min |
|             Median cumulative flush time across primary shards |                          |     2.74971 |    min |
|                Max cumulative flush time across primary shards |                          |     3.78457 |    min |
|                                        Total Young Gen GC time |                          |      59.655 |      s |
|                                       Total Young Gen GC count |                          |         622 |        |
|                                          Total Old Gen GC time |                          |           0 |      s |
|                                         Total Old Gen GC count |                          |           0 |        |
|                                                     Store size |                          |      72.084 |     GB |
|                                                  Translog size |                          | 6.14673e-07 |     GB |
|                                         Heap used for segments |                          |           0 |     MB |
|                                       Heap used for doc values |                          |           0 |     MB |
|                                            Heap used for terms |                          |           0 |     MB |
|                                            Heap used for norms |                          |           0 |     MB |
|                                           Heap used for points |                          |           0 |     MB |
|                                    Heap used for stored fields |                          |           0 |     MB |
|                                                  Segment count |                          |         235 |        |
|                                                 Min Throughput |             index-append |     12729.1 | docs/s |
|                                                Mean Throughput |             index-append |     14376.1 | docs/s |
|                                              Median Throughput |             index-append |     14440.5 | docs/s |
|                                                 Max Throughput |             index-append |     17443.4 | docs/s |
|                                        50th percentile latency |             index-append |     9016.47 |     ms |
|                                        90th percentile latency |             index-append |     13027.4 |     ms |
|                                        99th percentile latency |             index-append |     18164.9 |     ms |
|                                      99.9th percentile latency |             index-append |     27884.7 |     ms |
|                                     99.99th percentile latency |             index-append |             |        |
|                                       100th percentile latency |             index-append |     60584.9 |     ms |
|                                   50th percentile service time |             index-append |     9016.47 |     ms |
|                                   90th percentile service time |             index-append |     13027.4 |     ms |
|                                   99th percentile service time |             index-append |     18164.9 |     ms |
|                                 99.9th percentile service time |             index-append |     27884.7 |     ms |
|                                99.99th percentile service time |             index-append |             |        |
|                                  100th percentile service time |             index-append |     60584.9 |     ms |
|                                                     error rate |             index-append |           0 |      % |
|                                                 Min Throughput | wait-until-merges-finish |        0.01 |  ops/s |
|                                                Mean Throughput | wait-until-merges-finish |        0.01 |  ops/s |
|                                              Median Throughput | wait-until-merges-finish |        0.01 |  ops/s |
|                                                 Max Throughput | wait-until-merges-finish |        0.01 |  ops/s |
|                                        50th percentile latency | wait-until-merges-finish |             |        |
|                                        90th percentile latency | wait-until-merges-finish |             |        |
|                                        99th percentile latency | wait-until-merges-finish |             |        |
|                                      99.9th percentile latency | wait-until-merges-finish |             |        |
|                                     99.99th percentile latency | wait-until-merges-finish |             |        |
|                                       100th percentile latency | wait-until-merges-finish |      100100 |     ms |
|                                   50th percentile service time | wait-until-merges-finish |             |        |
|                                   90th percentile service time | wait-until-merges-finish |             |        |
|                                   99th percentile service time | wait-until-merges-finish |             |        |
|                                 99.9th percentile service time | wait-until-merges-finish |             |        |
|                                99.99th percentile service time | wait-until-merges-finish |             |        |
|                                  100th percentile service time | wait-until-merges-finish |      100100 |     ms |
|                                                     error rate | wait-until-merges-finish |           0 |      % |


----------------------------------
[INFO] SUCCESS (took 3029 seconds)
----------------------------------

so,remote,bic-24,ri-10s,bri-650ms (Run 1)

|                                                         Metric |                     Task |       Value |   Unit |
|---------------------------------------------------------------:|-------------------------:|------------:|-------:|
|                     Cumulative indexing time of primary shards |                          |      172.04 |    min |
|             Min cumulative indexing time across primary shards |                          |     26.9224 |    min |
|          Median cumulative indexing time across primary shards |                          |     28.5569 |    min |
|             Max cumulative indexing time across primary shards |                          |     30.3519 |    min |
|            Cumulative indexing throttle time of primary shards |                          |           0 |    min |
|    Min cumulative indexing throttle time across primary shards |                          |           0 |    min |
| Median cumulative indexing throttle time across primary shards |                          |           0 |    min |
|    Max cumulative indexing throttle time across primary shards |                          |           0 |    min |
|                        Cumulative merge time of primary shards |                          |     150.705 |    min |
|                       Cumulative merge count of primary shards |                          |         127 |        |
|                Min cumulative merge time across primary shards |                          |     14.3817 |    min |
|             Median cumulative merge time across primary shards |                          |     26.6883 |    min |
|                Max cumulative merge time across primary shards |                          |     30.7443 |    min |
|               Cumulative merge throttle time of primary shards |                          |      98.651 |    min |
|       Min cumulative merge throttle time across primary shards |                          |     7.89045 |    min |
|    Median cumulative merge throttle time across primary shards |                          |     18.1521 |    min |
|       Max cumulative merge throttle time across primary shards |                          |     19.9526 |    min |
|                      Cumulative refresh time of primary shards |                          |     30.4949 |    min |
|                     Cumulative refresh count of primary shards |                          |         683 |        |
|              Min cumulative refresh time across primary shards |                          |     4.75125 |    min |
|           Median cumulative refresh time across primary shards |                          |     4.95989 |    min |
|              Max cumulative refresh time across primary shards |                          |     5.55813 |    min |
|                        Cumulative flush time of primary shards |                          |     2.37727 |    min |
|                       Cumulative flush count of primary shards |                          |          26 |        |
|                Min cumulative flush time across primary shards |                          |      0.1545 |    min |
|             Median cumulative flush time across primary shards |                          |    0.313425 |    min |
|                Max cumulative flush time across primary shards |                          |    0.722817 |    min |
|                                        Total Young Gen GC time |                          |      17.347 |      s |
|                                       Total Young Gen GC count |                          |         396 |        |
|                                          Total Old Gen GC time |                          |           0 |      s |
|                                         Total Old Gen GC count |                          |           0 |        |
|                                                     Store size |                          |     73.5375 |     GB |
|                                                  Translog size |                          | 6.14673e-07 |     GB |
|                                         Heap used for segments |                          |           0 |     MB |
|                                       Heap used for doc values |                          |           0 |     MB |
|                                            Heap used for terms |                          |           0 |     MB |
|                                            Heap used for norms |                          |           0 |     MB |
|                                           Heap used for points |                          |           0 |     MB |
|                                    Heap used for stored fields |                          |           0 |     MB |
|                                                  Segment count |                          |         163 |        |
|                                                 Min Throughput |             index-append |     18806.5 | docs/s |
|                                                Mean Throughput |             index-append |     20013.9 | docs/s |
|                                              Median Throughput |             index-append |     19839.6 | docs/s |
|                                                 Max Throughput |             index-append |     22824.7 | docs/s |
|                                        50th percentile latency |             index-append |     6066.08 |     ms |
|                                        90th percentile latency |             index-append |     8500.81 |     ms |
|                                        99th percentile latency |             index-append |     11980.2 |     ms |
|                                      99.9th percentile latency |             index-append |     14263.2 |     ms |
|                                     99.99th percentile latency |             index-append |             |        |
|                                       100th percentile latency |             index-append |     15782.9 |     ms |
|                                   50th percentile service time |             index-append |     6066.08 |     ms |
|                                   90th percentile service time |             index-append |     8500.81 |     ms |
|                                   99th percentile service time |             index-append |     11980.2 |     ms |
|                                 99.9th percentile service time |             index-append |     14263.2 |     ms |
|                                99.99th percentile service time |             index-append |             |        |
|                                  100th percentile service time |             index-append |     15782.9 |     ms |
|                                                     error rate |             index-append |           0 |      % |
|                                                 Min Throughput | wait-until-merges-finish |           0 |  ops/s |
|                                                Mean Throughput | wait-until-merges-finish |           0 |  ops/s |
|                                              Median Throughput | wait-until-merges-finish |           0 |  ops/s |
|                                                 Max Throughput | wait-until-merges-finish |           0 |  ops/s |
|                                        50th percentile latency | wait-until-merges-finish |             |        |
|                                        90th percentile latency | wait-until-merges-finish |             |        |
|                                        99th percentile latency | wait-until-merges-finish |             |        |
|                                      99.9th percentile latency | wait-until-merges-finish |             |        |
|                                     99.99th percentile latency | wait-until-merges-finish |             |        |
|                                       100th percentile latency | wait-until-merges-finish |      857268 |     ms |
|                                   50th percentile service time | wait-until-merges-finish |             |        |
|                                   90th percentile service time | wait-until-merges-finish |             |        |
|                                   99th percentile service time | wait-until-merges-finish |             |        |
|                                 99.9th percentile service time | wait-until-merges-finish |             |        |
|                                99.99th percentile service time | wait-until-merges-finish |             |        |
|                                  100th percentile service time | wait-until-merges-finish |      857268 |     ms |
|                                                     error rate | wait-until-merges-finish |           0 |      % |

so,remote,bic-24,ri-10s,bri-650ms (Run 2)

|                                                         Metric |                     Task |      Value |   Unit |
|---------------------------------------------------------------:|-------------------------:|-----------:|-------:|
|                     Cumulative indexing time of primary shards |                          |    142.299 |    min |
|             Min cumulative indexing time across primary shards |                          |    0.09665 |    min |
|          Median cumulative indexing time across primary shards |                          |    28.4065 |    min |
|             Max cumulative indexing time across primary shards |                          |     30.238 |    min |
|            Cumulative indexing throttle time of primary shards |                          |          0 |    min |
|    Min cumulative indexing throttle time across primary shards |                          |          0 |    min |
| Median cumulative indexing throttle time across primary shards |                          |          0 |    min |
|    Max cumulative indexing throttle time across primary shards |                          |          0 |    min |
|                        Cumulative merge time of primary shards |                          |    86.1354 |    min |
|                       Cumulative merge count of primary shards |                          |        110 |        |
|                Min cumulative merge time across primary shards |                          |   0.474883 |    min |
|             Median cumulative merge time across primary shards |                          |      14.41 |    min |
|                Max cumulative merge time across primary shards |                          |     27.334 |    min |
|               Cumulative merge throttle time of primary shards |                          |    50.0146 |    min |
|       Min cumulative merge throttle time across primary shards |                          |   0.249433 |    min |
|    Median cumulative merge throttle time across primary shards |                          |    7.83618 |    min |
|       Max cumulative merge throttle time across primary shards |                          |    18.6518 |    min |
|                      Cumulative refresh time of primary shards |                          |    25.5143 |    min |
|                     Cumulative refresh count of primary shards |                          |        687 |        |
|              Min cumulative refresh time across primary shards |                          |     0.0328 |    min |
|           Median cumulative refresh time across primary shards |                          |    4.93388 |    min |
|              Max cumulative refresh time across primary shards |                          |     5.7074 |    min |
|                        Cumulative flush time of primary shards |                          |    2.72127 |    min |
|                       Cumulative flush count of primary shards |                          |         29 |        |
|                Min cumulative flush time across primary shards |                          |  0.0228333 |    min |
|             Median cumulative flush time across primary shards |                          |   0.479742 |    min |
|                Max cumulative flush time across primary shards |                          |    0.82545 |    min |
|                                        Total Young Gen GC time |                          |     17.257 |      s |
|                                       Total Young Gen GC count |                          |        361 |        |
|                                          Total Old Gen GC time |                          |          0 |      s |
|                                         Total Old Gen GC count |                          |          0 |        |
|                                                     Store size |                          |    65.7364 |     GB |
|                                                  Translog size |                          | 0.00556364 |     GB |
|                                         Heap used for segments |                          |          0 |     MB |
|                                       Heap used for doc values |                          |          0 |     MB |
|                                            Heap used for terms |                          |          0 |     MB |
|                                            Heap used for norms |                          |          0 |     MB |
|                                           Heap used for points |                          |          0 |     MB |
|                                    Heap used for stored fields |                          |          0 |     MB |
|                                                  Segment count |                          |        174 |        |
|                                                 Min Throughput |             index-append |    18290.8 | docs/s |
|                                                Mean Throughput |             index-append |    19317.5 | docs/s |
|                                              Median Throughput |             index-append |      19033 | docs/s |
|                                                 Max Throughput |             index-append |    21558.2 | docs/s |
|                                        50th percentile latency |             index-append |    6322.65 |     ms |
|                                        90th percentile latency |             index-append |     8656.1 |     ms |
|                                        99th percentile latency |             index-append |    11184.8 |     ms |
|                                      99.9th percentile latency |             index-append |    13370.4 |     ms |
|                                     99.99th percentile latency |             index-append |            |        |
|                                       100th percentile latency |             index-append |    15010.1 |     ms |
|                                   50th percentile service time |             index-append |    6322.65 |     ms |
|                                   90th percentile service time |             index-append |     8656.1 |     ms |
|                                   99th percentile service time |             index-append |    11184.8 |     ms |
|                                 99.9th percentile service time |             index-append |    13370.4 |     ms |
|                                99.99th percentile service time |             index-append |            |        |
|                                  100th percentile service time |             index-append |    15010.1 |     ms |
|                                                     error rate |             index-append |       0.06 |      % |
|                                                 Min Throughput | wait-until-merges-finish |          0 |  ops/s |
|                                                Mean Throughput | wait-until-merges-finish |          0 |  ops/s |
|                                              Median Throughput | wait-until-merges-finish |          0 |  ops/s |
|                                                 Max Throughput | wait-until-merges-finish |          0 |  ops/s |
|                                        50th percentile latency | wait-until-merges-finish |            |        |
|                                        90th percentile latency | wait-until-merges-finish |            |        |
|                                        99th percentile latency | wait-until-merges-finish |            |        |
|                                      99.9th percentile latency | wait-until-merges-finish |            |        |
|                                     99.99th percentile latency | wait-until-merges-finish |            |        |
|                                       100th percentile latency | wait-until-merges-finish |     722053 |     ms |
|                                   50th percentile service time | wait-until-merges-finish |            |        |
|                                   90th percentile service time | wait-until-merges-finish |            |        |
|                                   99th percentile service time | wait-until-merges-finish |            |        |
|                                 99.9th percentile service time | wait-until-merges-finish |            |        |
|                                99.99th percentile service time | wait-until-merges-finish |            |        |
|                                  100th percentile service time | wait-until-merges-finish |     722053 |     ms |
|                                                     error rate | wait-until-merges-finish |          0 |      % |

http_logs,docrep,bic-24 (Run 1)

|                                                         Metric |                                                   Task |       Value |    Unit |
|---------------------------------------------------------------:|-------------------------------------------------------:|------------:|--------:|
|                     Cumulative indexing time of primary shards |                                                        |     133.333 |     min |
|             Min cumulative indexing time across primary shards |                                                        |           0 |     min |
|          Median cumulative indexing time across primary shards |                                                        |    0.916483 |     min |
|             Max cumulative indexing time across primary shards |                                                        |     17.3902 |     min |
|            Cumulative indexing throttle time of primary shards |                                                        |           0 |     min |
|    Min cumulative indexing throttle time across primary shards |                                                        |           0 |     min |
| Median cumulative indexing throttle time across primary shards |                                                        |           0 |     min |
|    Max cumulative indexing throttle time across primary shards |                                                        |           0 |     min |
|                        Cumulative merge time of primary shards |                                                        |     29.5956 |     min |
|                       Cumulative merge count of primary shards |                                                        |          70 |         |
|                Min cumulative merge time across primary shards |                                                        |           0 |     min |
|             Median cumulative merge time across primary shards |                                                        |   0.0892667 |     min |
|                Max cumulative merge time across primary shards |                                                        |     5.09587 |     min |
|               Cumulative merge throttle time of primary shards |                                                        |     3.80785 |     min |
|       Min cumulative merge throttle time across primary shards |                                                        |           0 |     min |
|    Median cumulative merge throttle time across primary shards |                                                        |           0 |     min |
|       Max cumulative merge throttle time across primary shards |                                                        |    0.881333 |     min |
|                      Cumulative refresh time of primary shards |                                                        |      20.318 |     min |
|                     Cumulative refresh count of primary shards |                                                        |         818 |         |
|              Min cumulative refresh time across primary shards |                                                        |           0 |     min |
|           Median cumulative refresh time across primary shards |                                                        |    0.218783 |     min |
|              Max cumulative refresh time across primary shards |                                                        |     2.56795 |     min |
|                        Cumulative flush time of primary shards |                                                        |     13.1379 |     min |
|                       Cumulative flush count of primary shards |                                                        |         146 |         |
|                Min cumulative flush time across primary shards |                                                        |           0 |     min |
|             Median cumulative flush time across primary shards |                                                        |   0.0421417 |     min |
|                Max cumulative flush time across primary shards |                                                        |     2.78772 |     min |
|                                        Total Young Gen GC time |                                                        |     166.944 |       s |
|                                       Total Young Gen GC count |                                                        |        2075 |         |
|                                          Total Old Gen GC time |                                                        |           0 |       s |
|                                         Total Old Gen GC count |                                                        |           0 |         |
|                                                     Store size |                                                        |     33.8708 |      GB |
|                                                  Translog size |                                                        | 4.91738e-06 |      GB |
|                                         Heap used for segments |                                                        |           0 |      MB |
|                                       Heap used for doc values |                                                        |           0 |      MB |
|                                            Heap used for terms |                                                        |           0 |      MB |
|                                            Heap used for norms |                                                        |           0 |      MB |
|                                           Heap used for points |                                                        |           0 |      MB |
|                                    Heap used for stored fields |                                                        |           0 |      MB |
|                                                  Segment count |                                                        |          42 |         |
|                                                 Min Throughput |                                           index-append |     60717.2 |  docs/s |
|                                                Mean Throughput |                                           index-append |       64797 |  docs/s |
|                                              Median Throughput |                                           index-append |     65568.8 |  docs/s |
|                                                 Max Throughput |                                           index-append |     66959.9 |  docs/s |
|                                        50th percentile latency |                                           index-append |     1746.95 |      ms |
|                                        90th percentile latency |                                           index-append |     2708.08 |      ms |
|                                        99th percentile latency |                                           index-append |     4070.75 |      ms |
|                                      99.9th percentile latency |                                           index-append |     10895.9 |      ms |
|                                     99.99th percentile latency |                                           index-append |     22404.4 |      ms |
|                                       100th percentile latency |                                           index-append |     26528.9 |      ms |
|                                   50th percentile service time |                                           index-append |     1746.95 |      ms |
|                                   90th percentile service time |                                           index-append |     2708.08 |      ms |
|                                   99th percentile service time |                                           index-append |     4070.75 |      ms |
|                                 99.9th percentile service time |                                           index-append |     10895.9 |      ms |
|                                99.99th percentile service time |                                           index-append |     22404.4 |      ms |
|                                  100th percentile service time |                                           index-append |     26528.9 |      ms |
|                                                     error rate |                                           index-append |           0 |       % |
|                                                 Min Throughput |                               wait-until-merges-finish |       27.55 |   ops/s |
|                                                Mean Throughput |                               wait-until-merges-finish |       27.55 |   ops/s |
|                                              Median Throughput |                               wait-until-merges-finish |       27.55 |   ops/s |
|                                                 Max Throughput |                               wait-until-merges-finish |       27.55 |   ops/s |
|                                        50th percentile latency |                               wait-until-merges-finish |             |         |
|                                        90th percentile latency |                               wait-until-merges-finish |             |         |
|                                        99th percentile latency |                               wait-until-merges-finish |             |         |
|                                      99.9th percentile latency |                               wait-until-merges-finish |             |         |
|                                     99.99th percentile latency |                               wait-until-merges-finish |             |         |
|                                       100th percentile latency |                               wait-until-merges-finish |     35.6845 |      ms |
|                                   50th percentile service time |                               wait-until-merges-finish |             |         |
|                                   90th percentile service time |                               wait-until-merges-finish |             |         |
|                                   99th percentile service time |                               wait-until-merges-finish |             |         |
|                                 99.9th percentile service time |                               wait-until-merges-finish |             |         |
|                                99.99th percentile service time |                               wait-until-merges-finish |             |         |
|                                  100th percentile service time |                               wait-until-merges-finish |     35.6845 |      ms |
|                                                     error rate |                               wait-until-merges-finish |           0 |       % |
|                                                 Min Throughput |                                                default |        7.97 |   ops/s |
|                                                Mean Throughput |                                                default |        7.97 |   ops/s |
|                                              Median Throughput |                                                default |        7.97 |   ops/s |
|                                                 Max Throughput |                                                default |        7.97 |   ops/s |
|                                        50th percentile latency |                                                default |     7.02868 |      ms |
|                                        90th percentile latency |                                                default |      8.5953 |      ms |
|                                        99th percentile latency |                                                default |      12.487 |      ms |
|                                      99.9th percentile latency |                                                default |             |         |
|                                     99.99th percentile latency |                                                default |             |         |
|                                       100th percentile latency |                                                default |     13.3263 |      ms |
|                                   50th percentile service time |                                                default |     5.29421 |      ms |
|                                   90th percentile service time |                                                default |     6.83067 |      ms |
|                                   99th percentile service time |                                                default |      11.439 |      ms |
|                                 99.9th percentile service time |                                                default |             |         |
|                                99.99th percentile service time |                                                default |             |         |
|                                  100th percentile service time |                                                default |     11.6446 |      ms |
|                                                     error rate |                                                default |           0 |       % |
|                                                 Min Throughput |                                                   term |       49.45 |   ops/s |
|                                                Mean Throughput |                                                   term |       49.48 |   ops/s |
|                                              Median Throughput |                                                   term |       49.48 |   ops/s |
|                                                 Max Throughput |                                                   term |        49.5 |   ops/s |
|                                        50th percentile latency |                                                   term |     8.11407 |      ms |
|                                        90th percentile latency |                                                   term |     10.2131 |      ms |
|                                        99th percentile latency |                                                   term |     12.1307 |      ms |
|                                      99.9th percentile latency |                                                   term |             |         |
|                                     99.99th percentile latency |                                                   term |             |         |
|                                       100th percentile latency |                                                   term |     13.0884 |      ms |
|                                   50th percentile service time |                                                   term |     6.70074 |      ms |
|                                   90th percentile service time |                                                   term |     8.91412 |      ms |
|                                   99th percentile service time |                                                   term |     11.1013 |      ms |
|                                 99.9th percentile service time |                                                   term |             |         |
|                                99.99th percentile service time |                                                   term |             |         |
|                                  100th percentile service time |                                                   term |     11.2426 |      ms |
|                                                     error rate |                                                   term |           0 |       % |
|                                                 Min Throughput |                                                  range |           1 |   ops/s |
|                                                Mean Throughput |                                                  range |        1.01 |   ops/s |
|                                              Median Throughput |                                                  range |        1.01 |   ops/s |
|                                                 Max Throughput |                                                  range |        1.01 |   ops/s |
|                                        50th percentile latency |                                                  range |      11.354 |      ms |
|                                        90th percentile latency |                                                  range |     13.8147 |      ms |
|                                        99th percentile latency |                                                  range |      16.696 |      ms |
|                                      99.9th percentile latency |                                                  range |             |         |
|                                     99.99th percentile latency |                                                  range |             |         |
|                                       100th percentile latency |                                                  range |     16.7178 |      ms |
|                                   50th percentile service time |                                                  range |     9.73689 |      ms |
|                                   90th percentile service time |                                                  range |     11.9682 |      ms |
|                                   99th percentile service time |                                                  range |     15.2398 |      ms |
|                                 99.9th percentile service time |                                                  range |             |         |
|                                99.99th percentile service time |                                                  range |             |         |
|                                  100th percentile service time |                                                  range |     15.4227 |      ms |
|                                                     error rate |                                                  range |           0 |       % |
|                                                 Min Throughput |                                          200s-in-range |       32.83 |   ops/s |
|                                                Mean Throughput |                                          200s-in-range |       32.84 |   ops/s |
|                                              Median Throughput |                                          200s-in-range |       32.84 |   ops/s |
|                                                 Max Throughput |                                          200s-in-range |       32.85 |   ops/s |
|                                        50th percentile latency |                                          200s-in-range |     8.91839 |      ms |
|                                        90th percentile latency |                                          200s-in-range |     12.2244 |      ms |
|                                        99th percentile latency |                                          200s-in-range |     16.8145 |      ms |
|                                      99.9th percentile latency |                                          200s-in-range |             |         |
|                                     99.99th percentile latency |                                          200s-in-range |             |         |
|                                       100th percentile latency |                                          200s-in-range |     18.0734 |      ms |
|                                   50th percentile service time |                                          200s-in-range |     7.99701 |      ms |
|                                   90th percentile service time |                                          200s-in-range |     11.5843 |      ms |
|                                   99th percentile service time |                                          200s-in-range |     15.9256 |      ms |
|                                 99.9th percentile service time |                                          200s-in-range |             |         |
|                                99.99th percentile service time |                                          200s-in-range |             |         |
|                                  100th percentile service time |                                          200s-in-range |     17.5955 |      ms |
|                                                     error rate |                                          200s-in-range |           0 |       % |
|                                                 Min Throughput |                                          400s-in-range |       49.94 |   ops/s |
|                                                Mean Throughput |                                          400s-in-range |       49.95 |   ops/s |
|                                              Median Throughput |                                          400s-in-range |       49.95 |   ops/s |
|                                                 Max Throughput |                                          400s-in-range |       49.95 |   ops/s |
|                                        50th percentile latency |                                          400s-in-range |     6.03231 |      ms |
|                                        90th percentile latency |                                          400s-in-range |     7.23765 |      ms |
|                                        99th percentile latency |                                          400s-in-range |     10.1454 |      ms |
|                                      99.9th percentile latency |                                          400s-in-range |             |         |
|                                     99.99th percentile latency |                                          400s-in-range |             |         |
|                                       100th percentile latency |                                          400s-in-range |     12.9462 |      ms |
|                                   50th percentile service time |                                          400s-in-range |     5.17197 |      ms |
|                                   90th percentile service time |                                          400s-in-range |     6.22814 |      ms |
|                                   99th percentile service time |                                          400s-in-range |     9.22896 |      ms |
|                                 99.9th percentile service time |                                          400s-in-range |             |         |
|                                99.99th percentile service time |                                          400s-in-range |             |         |
|                                  100th percentile service time |                                          400s-in-range |     12.3969 |      ms |
|                                                     error rate |                                          400s-in-range |           0 |       % |
|                                                 Min Throughput |                                             hourly_agg |         0.2 |   ops/s |
|                                                Mean Throughput |                                             hourly_agg |         0.2 |   ops/s |
|                                              Median Throughput |                                             hourly_agg |         0.2 |   ops/s |
|                                                 Max Throughput |                                             hourly_agg |         0.2 |   ops/s |
|                                        50th percentile latency |                                             hourly_agg |      3226.2 |      ms |
|                                        90th percentile latency |                                             hourly_agg |     3350.21 |      ms |
|                                        99th percentile latency |                                             hourly_agg |     3428.95 |      ms |
|                                      99.9th percentile latency |                                             hourly_agg |             |         |
|                                     99.99th percentile latency |                                             hourly_agg |             |         |
|                                       100th percentile latency |                                             hourly_agg |      3469.4 |      ms |
|                                   50th percentile service time |                                             hourly_agg |      3223.5 |      ms |
|                                   90th percentile service time |                                             hourly_agg |     3347.06 |      ms |
|                                   99th percentile service time |                                             hourly_agg |     3425.85 |      ms |
|                                 99.9th percentile service time |                                             hourly_agg |             |         |
|                                99.99th percentile service time |                                             hourly_agg |             |         |
|                                  100th percentile service time |                                             hourly_agg |      3467.1 |      ms |
|                                                     error rate |                                             hourly_agg |           0 |       % |
|                                                 Min Throughput |                                                 scroll |       24.96 | pages/s |
|                                                Mean Throughput |                                                 scroll |       24.98 | pages/s |
|                                              Median Throughput |                                                 scroll |       24.98 | pages/s |
|                                                 Max Throughput |                                                 scroll |       24.98 | pages/s |
|                                        50th percentile latency |                                                 scroll |     538.195 |      ms |
|                                        90th percentile latency |                                                 scroll |     584.604 |      ms |
|                                        99th percentile latency |                                                 scroll |     743.932 |      ms |
|                                      99.9th percentile latency |                                                 scroll |             |         |
|                                     99.99th percentile latency |                                                 scroll |             |         |
|                                       100th percentile latency |                                                 scroll |     947.148 |      ms |
|                                   50th percentile service time |                                                 scroll |     536.743 |      ms |
|                                   90th percentile service time |                                                 scroll |     583.331 |      ms |
|                                   99th percentile service time |                                                 scroll |     742.531 |      ms |
|                                 99.9th percentile service time |                                                 scroll |             |         |
|                                99.99th percentile service time |                                                 scroll |             |         |
|                                  100th percentile service time |                                                 scroll |     945.925 |      ms |
|                                                     error rate |                                                 scroll |           0 |       % |
|                                                 Min Throughput |                                    desc_sort_timestamp |         0.5 |   ops/s |
|                                                Mean Throughput |                                    desc_sort_timestamp |         0.5 |   ops/s |
|                                              Median Throughput |                                    desc_sort_timestamp |         0.5 |   ops/s |
|                                                 Max Throughput |                                    desc_sort_timestamp |         0.5 |   ops/s |
|                                        50th percentile latency |                                    desc_sort_timestamp |     116.118 |      ms |
|                                        90th percentile latency |                                    desc_sort_timestamp |     153.241 |      ms |
|                                        99th percentile latency |                                    desc_sort_timestamp |     182.602 |      ms |
|                                      99.9th percentile latency |                                    desc_sort_timestamp |             |         |
|                                     99.99th percentile latency |                                    desc_sort_timestamp |             |         |
|                                       100th percentile latency |                                    desc_sort_timestamp |     201.038 |      ms |
|                                   50th percentile service time |                                    desc_sort_timestamp |     113.357 |      ms |
|                                   90th percentile service time |                                    desc_sort_timestamp |     150.568 |      ms |
|                                   99th percentile service time |                                    desc_sort_timestamp |     180.905 |      ms |
|                                 99.9th percentile service time |                                    desc_sort_timestamp |             |         |
|                                99.99th percentile service time |                                    desc_sort_timestamp |             |         |
|                                  100th percentile service time |                                    desc_sort_timestamp |     198.162 |      ms |
|                                                     error rate |                                    desc_sort_timestamp |           0 |       % |
|                                                 Min Throughput |                                     asc_sort_timestamp |         0.5 |   ops/s |
|                                                Mean Throughput |                                     asc_sort_timestamp |         0.5 |   ops/s |
|                                              Median Throughput |                                     asc_sort_timestamp |         0.5 |   ops/s |
|                                                 Max Throughput |                                     asc_sort_timestamp |         0.5 |   ops/s |
|                                        50th percentile latency |                                     asc_sort_timestamp |     10.1293 |      ms |
|                                        90th percentile latency |                                     asc_sort_timestamp |     11.6825 |      ms |
|                                        99th percentile latency |                                     asc_sort_timestamp |     16.9844 |      ms |
|                                      99.9th percentile latency |                                     asc_sort_timestamp |             |         |
|                                     99.99th percentile latency |                                     asc_sort_timestamp |             |         |
|                                       100th percentile latency |                                     asc_sort_timestamp |     24.4486 |      ms |
|                                   50th percentile service time |                                     asc_sort_timestamp |     7.57608 |      ms |
|                                   90th percentile service time |                                     asc_sort_timestamp |     9.23958 |      ms |
|                                   99th percentile service time |                                     asc_sort_timestamp |     14.7651 |      ms |
|                                 99.9th percentile service time |                                     asc_sort_timestamp |             |         |
|                                99.99th percentile service time |                                     asc_sort_timestamp |             |         |
|                                  100th percentile service time |                                     asc_sort_timestamp |     21.1904 |      ms |
|                                                     error rate |                                     asc_sort_timestamp |           0 |       % |
|                                                 Min Throughput |                         desc_sort_with_after_timestamp |         0.5 |   ops/s |
|                                                Mean Throughput |                         desc_sort_with_after_timestamp |        0.51 |   ops/s |
|                                              Median Throughput |                         desc_sort_with_after_timestamp |        0.51 |   ops/s |
|                                                 Max Throughput |                         desc_sort_with_after_timestamp |        0.54 |   ops/s |
|                                        50th percentile latency |                         desc_sort_with_after_timestamp |     194.517 |      ms |
|                                        90th percentile latency |                         desc_sort_with_after_timestamp |     222.342 |      ms |
|                                        99th percentile latency |                         desc_sort_with_after_timestamp |     251.231 |      ms |
|                                      99.9th percentile latency |                         desc_sort_with_after_timestamp |             |         |
|                                     99.99th percentile latency |                         desc_sort_with_after_timestamp |             |         |
|                                       100th percentile latency |                         desc_sort_with_after_timestamp |      256.04 |      ms |
|                                   50th percentile service time |                         desc_sort_with_after_timestamp |     191.776 |      ms |
|                                   90th percentile service time |                         desc_sort_with_after_timestamp |     220.132 |      ms |
|                                   99th percentile service time |                         desc_sort_with_after_timestamp |     248.773 |      ms |
|                                 99.9th percentile service time |                         desc_sort_with_after_timestamp |             |         |
|                                99.99th percentile service time |                         desc_sort_with_after_timestamp |             |         |
|                                  100th percentile service time |                         desc_sort_with_after_timestamp |     253.557 |      ms |
|                                                     error rate |                         desc_sort_with_after_timestamp |           0 |       % |
|                                                 Min Throughput |                          asc_sort_with_after_timestamp |         0.5 |   ops/s |
|                                                Mean Throughput |                          asc_sort_with_after_timestamp |        0.51 |   ops/s |
|                                              Median Throughput |                          asc_sort_with_after_timestamp |        0.51 |   ops/s |
|                                                 Max Throughput |                          asc_sort_with_after_timestamp |        0.55 |   ops/s |
|                                        50th percentile latency |                          asc_sort_with_after_timestamp |     12.4596 |      ms |
|                                        90th percentile latency |                          asc_sort_with_after_timestamp |     15.4538 |      ms |
|                                        99th percentile latency |                          asc_sort_with_after_timestamp |     18.1168 |      ms |
|                                      99.9th percentile latency |                          asc_sort_with_after_timestamp |             |         |
|                                     99.99th percentile latency |                          asc_sort_with_after_timestamp |             |         |
|                                       100th percentile latency |                          asc_sort_with_after_timestamp |     21.6304 |      ms |
|                                   50th percentile service time |                          asc_sort_with_after_timestamp |     10.0214 |      ms |
|                                   90th percentile service time |                          asc_sort_with_after_timestamp |     12.7455 |      ms |
|                                   99th percentile service time |                          asc_sort_with_after_timestamp |     15.2533 |      ms |
|                                 99.9th percentile service time |                          asc_sort_with_after_timestamp |             |         |
|                                99.99th percentile service time |                          asc_sort_with_after_timestamp |             |         |
|                                  100th percentile service time |                          asc_sort_with_after_timestamp |     18.4944 |      ms |
|                                                     error rate |                          asc_sort_with_after_timestamp |           0 |       % |
|                                                 Min Throughput |                         wait-until-merges-1-seg-finish |       29.14 |   ops/s |
|                                                Mean Throughput |                         wait-until-merges-1-seg-finish |       29.14 |   ops/s |
|                                              Median Throughput |                         wait-until-merges-1-seg-finish |       29.14 |   ops/s |
|                                                 Max Throughput |                         wait-until-merges-1-seg-finish |       29.14 |   ops/s |
|                                        50th percentile latency |                         wait-until-merges-1-seg-finish |             |         |
|                                        90th percentile latency |                         wait-until-merges-1-seg-finish |             |         |
|                                        99th percentile latency |                         wait-until-merges-1-seg-finish |             |         |
|                                      99.9th percentile latency |                         wait-until-merges-1-seg-finish |             |         |
|                                     99.99th percentile latency |                         wait-until-merges-1-seg-finish |             |         |
|                                       100th percentile latency |                         wait-until-merges-1-seg-finish |     33.7297 |      ms |
|                                   50th percentile service time |                         wait-until-merges-1-seg-finish |             |         |
|                                   90th percentile service time |                         wait-until-merges-1-seg-finish |             |         |
|                                   99th percentile service time |                         wait-until-merges-1-seg-finish |             |         |
|                                 99.9th percentile service time |                         wait-until-merges-1-seg-finish |             |         |
|                                99.99th percentile service time |                         wait-until-merges-1-seg-finish |             |         |
|                                  100th percentile service time |                         wait-until-merges-1-seg-finish |     33.7297 |      ms |
|                                                     error rate |                         wait-until-merges-1-seg-finish |           0 |       % |
|                                                 Min Throughput |            desc-sort-timestamp-after-force-merge-1-seg |           2 |   ops/s |
|                                                Mean Throughput |            desc-sort-timestamp-after-force-merge-1-seg |           2 |   ops/s |
|                                              Median Throughput |            desc-sort-timestamp-after-force-merge-1-seg |           2 |   ops/s |
|                                                 Max Throughput |            desc-sort-timestamp-after-force-merge-1-seg |           2 |   ops/s |
|                                        50th percentile latency |            desc-sort-timestamp-after-force-merge-1-seg |     276.999 |      ms |
|                                        90th percentile latency |            desc-sort-timestamp-after-force-merge-1-seg |     303.391 |      ms |
|                                        99th percentile latency |            desc-sort-timestamp-after-force-merge-1-seg |     349.492 |      ms |
|                                      99.9th percentile latency |            desc-sort-timestamp-after-force-merge-1-seg |             |         |
|                                     99.99th percentile latency |            desc-sort-timestamp-after-force-merge-1-seg |             |         |
|                                       100th percentile latency |            desc-sort-timestamp-after-force-merge-1-seg |     350.185 |      ms |
|                                   50th percentile service time |            desc-sort-timestamp-after-force-merge-1-seg |     276.213 |      ms |
|                                   90th percentile service time |            desc-sort-timestamp-after-force-merge-1-seg |     301.903 |      ms |
|                                   99th percentile service time |            desc-sort-timestamp-after-force-merge-1-seg |     347.321 |      ms |
|                                 99.9th percentile service time |            desc-sort-timestamp-after-force-merge-1-seg |             |         |
|                                99.99th percentile service time |            desc-sort-timestamp-after-force-merge-1-seg |             |         |
|                                  100th percentile service time |            desc-sort-timestamp-after-force-merge-1-seg |     348.149 |      ms |
|                                                     error rate |            desc-sort-timestamp-after-force-merge-1-seg |           0 |       % |
|                                                 Min Throughput |             asc-sort-timestamp-after-force-merge-1-seg |        2.01 |   ops/s |
|                                                Mean Throughput |             asc-sort-timestamp-after-force-merge-1-seg |        2.01 |   ops/s |
|                                              Median Throughput |             asc-sort-timestamp-after-force-merge-1-seg |        2.01 |   ops/s |
|                                                 Max Throughput |             asc-sort-timestamp-after-force-merge-1-seg |        2.01 |   ops/s |
|                                        50th percentile latency |             asc-sort-timestamp-after-force-merge-1-seg |      11.021 |      ms |
|                                        90th percentile latency |             asc-sort-timestamp-after-force-merge-1-seg |     13.0055 |      ms |
|                                        99th percentile latency |             asc-sort-timestamp-after-force-merge-1-seg |     14.8803 |      ms |
|                                      99.9th percentile latency |             asc-sort-timestamp-after-force-merge-1-seg |             |         |
|                                     99.99th percentile latency |             asc-sort-timestamp-after-force-merge-1-seg |             |         |
|                                       100th percentile latency |             asc-sort-timestamp-after-force-merge-1-seg |     15.1771 |      ms |
|                                   50th percentile service time |             asc-sort-timestamp-after-force-merge-1-seg |     9.97655 |      ms |
|                                   90th percentile service time |             asc-sort-timestamp-after-force-merge-1-seg |     11.4836 |      ms |
|                                   99th percentile service time |             asc-sort-timestamp-after-force-merge-1-seg |     13.5347 |      ms |
|                                 99.9th percentile service time |             asc-sort-timestamp-after-force-merge-1-seg |             |         |
|                                99.99th percentile service time |             asc-sort-timestamp-after-force-merge-1-seg |             |         |
|                                  100th percentile service time |             asc-sort-timestamp-after-force-merge-1-seg |     13.5759 |      ms |
|                                                     error rate |             asc-sort-timestamp-after-force-merge-1-seg |           0 |       % |
|                                                 Min Throughput | desc-sort-with-after-timestamp-after-force-merge-1-seg |         0.5 |   ops/s |
|                                                Mean Throughput | desc-sort-with-after-timestamp-after-force-merge-1-seg |        0.51 |   ops/s |
|                                              Median Throughput | desc-sort-with-after-timestamp-after-force-merge-1-seg |        0.51 |   ops/s |
|                                                 Max Throughput | desc-sort-with-after-timestamp-after-force-merge-1-seg |        0.54 |   ops/s |
|                                        50th percentile latency | desc-sort-with-after-timestamp-after-force-merge-1-seg |       200.3 |      ms |
|                                        90th percentile latency | desc-sort-with-after-timestamp-after-force-merge-1-seg |     224.152 |      ms |
|                                        99th percentile latency | desc-sort-with-after-timestamp-after-force-merge-1-seg |     252.751 |      ms |
|                                      99.9th percentile latency | desc-sort-with-after-timestamp-after-force-merge-1-seg |             |         |
|                                     99.99th percentile latency | desc-sort-with-after-timestamp-after-force-merge-1-seg |             |         |
|                                       100th percentile latency | desc-sort-with-after-timestamp-after-force-merge-1-seg |      294.99 |      ms |
|                                   50th percentile service time | desc-sort-with-after-timestamp-after-force-merge-1-seg |      197.83 |      ms |
|                                   90th percentile service time | desc-sort-with-after-timestamp-after-force-merge-1-seg |     222.666 |      ms |
|                                   99th percentile service time | desc-sort-with-after-timestamp-after-force-merge-1-seg |      250.33 |      ms |
|                                 99.9th percentile service time | desc-sort-with-after-timestamp-after-force-merge-1-seg |             |         |
|                                99.99th percentile service time | desc-sort-with-after-timestamp-after-force-merge-1-seg |             |         |
|                                  100th percentile service time | desc-sort-with-after-timestamp-after-force-merge-1-seg |     292.227 |      ms |
|                                                     error rate | desc-sort-with-after-timestamp-after-force-merge-1-seg |           0 |       % |
|                                                 Min Throughput |  asc-sort-with-after-timestamp-after-force-merge-1-seg |         0.5 |   ops/s |
|                                                Mean Throughput |  asc-sort-with-after-timestamp-after-force-merge-1-seg |        0.51 |   ops/s |
|                                              Median Throughput |  asc-sort-with-after-timestamp-after-force-merge-1-seg |        0.51 |   ops/s |
|                                                 Max Throughput |  asc-sort-with-after-timestamp-after-force-merge-1-seg |        0.55 |   ops/s |
|                                        50th percentile latency |  asc-sort-with-after-timestamp-after-force-merge-1-seg |     48.1817 |      ms |
|                                        90th percentile latency |  asc-sort-with-after-timestamp-after-force-merge-1-seg |     55.9074 |      ms |
|                                        99th percentile latency |  asc-sort-with-after-timestamp-after-force-merge-1-seg |     61.9364 |      ms |
|                                      99.9th percentile latency |  asc-sort-with-after-timestamp-after-force-merge-1-seg |             |         |
|                                     99.99th percentile latency |  asc-sort-with-after-timestamp-after-force-merge-1-seg |             |         |
|                                       100th percentile latency |  asc-sort-with-after-timestamp-after-force-merge-1-seg |     65.7088 |      ms |
|                                   50th percentile service time |  asc-sort-with-after-timestamp-after-force-merge-1-seg |     46.0766 |      ms |
|                                   90th percentile service time |  asc-sort-with-after-timestamp-after-force-merge-1-seg |     54.6314 |      ms |
|                                   99th percentile service time |  asc-sort-with-after-timestamp-after-force-merge-1-seg |     61.0294 |      ms |
|                                 99.9th percentile service time |  asc-sort-with-after-timestamp-after-force-merge-1-seg |             |         |
|                                99.99th percentile service time |  asc-sort-with-after-timestamp-after-force-merge-1-seg |             |         |
|                                  100th percentile service time |  asc-sort-with-after-timestamp-after-force-merge-1-seg |     62.9261 |      ms |
|                                                     error rate |  asc-sort-with-after-timestamp-after-force-merge-1-seg |           0 |       % |


----------------------------------
[INFO] SUCCESS (took 8747 seconds)
----------------------------------

http_logs,docrep,bic-24 (Run 2)

|                                                         Metric |                                                   Task |       Value |    Unit |
|---------------------------------------------------------------:|-------------------------------------------------------:|------------:|--------:|
|                     Cumulative indexing time of primary shards |                                                        |     131.721 |     min |
|             Min cumulative indexing time across primary shards |                                                        |           0 |     min |
|          Median cumulative indexing time across primary shards |                                                        |    0.988708 |     min |
|             Max cumulative indexing time across primary shards |                                                        |     17.8964 |     min |
|            Cumulative indexing throttle time of primary shards |                                                        |           0 |     min |
|    Min cumulative indexing throttle time across primary shards |                                                        |           0 |     min |
| Median cumulative indexing throttle time across primary shards |                                                        |           0 |     min |
|    Max cumulative indexing throttle time across primary shards |                                                        |           0 |     min |
|                        Cumulative merge time of primary shards |                                                        |     25.8192 |     min |
|                       Cumulative merge count of primary shards |                                                        |          71 |         |
|                Min cumulative merge time across primary shards |                                                        |           0 |     min |
|             Median cumulative merge time across primary shards |                                                        |   0.0899833 |     min |
|                Max cumulative merge time across primary shards |                                                        |     5.07528 |     min |
|               Cumulative merge throttle time of primary shards |                                                        |     3.17128 |     min |
|       Min cumulative merge throttle time across primary shards |                                                        |           0 |     min |
|    Median cumulative merge throttle time across primary shards |                                                        |           0 |     min |
|       Max cumulative merge throttle time across primary shards |                                                        |      0.8565 |     min |
|                      Cumulative refresh time of primary shards |                                                        |     20.4898 |     min |
|                     Cumulative refresh count of primary shards |                                                        |         819 |         |
|              Min cumulative refresh time across primary shards |                                                        |           0 |     min |
|           Median cumulative refresh time across primary shards |                                                        |    0.220358 |     min |
|              Max cumulative refresh time across primary shards |                                                        |      2.5494 |     min |
|                        Cumulative flush time of primary shards |                                                        |     10.5104 |     min |
|                       Cumulative flush count of primary shards |                                                        |         144 |         |
|                Min cumulative flush time across primary shards |                                                        |           0 |     min |
|             Median cumulative flush time across primary shards |                                                        |    0.024375 |     min |
|                Max cumulative flush time across primary shards |                                                        |     2.21608 |     min |
|                                        Total Young Gen GC time |                                                        |     165.436 |       s |
|                                       Total Young Gen GC count |                                                        |        2060 |         |
|                                          Total Old Gen GC time |                                                        |           0 |       s |
|                                         Total Old Gen GC count |                                                        |           0 |         |
|                                                     Store size |                                                        |     33.8781 |      GB |
|                                                  Translog size |                                                        | 4.91738e-06 |      GB |
|                                         Heap used for segments |                                                        |           0 |      MB |
|                                       Heap used for doc values |                                                        |           0 |      MB |
|                                            Heap used for terms |                                                        |           0 |      MB |
|                                            Heap used for norms |                                                        |           0 |      MB |
|                                           Heap used for points |                                                        |           0 |      MB |
|                                    Heap used for stored fields |                                                        |           0 |      MB |
|                                                  Segment count |                                                        |          42 |         |
|                                                 Min Throughput |                                           index-append |     62169.7 |  docs/s |
|                                                Mean Throughput |                                           index-append |     67049.1 |  docs/s |
|                                              Median Throughput |                                           index-append |     67361.2 |  docs/s |
|                                                 Max Throughput |                                           index-append |     72818.5 |  docs/s |
|                                        50th percentile latency |                                           index-append |     1757.78 |      ms |
|                                        90th percentile latency |                                           index-append |     2607.16 |      ms |
|                                        99th percentile latency |                                           index-append |     3881.16 |      ms |
|                                      99.9th percentile latency |                                           index-append |     10768.9 |      ms |
|                                     99.99th percentile latency |                                           index-append |     20432.9 |      ms |
|                                       100th percentile latency |                                           index-append |     33889.3 |      ms |
|                                   50th percentile service time |                                           index-append |     1757.78 |      ms |
|                                   90th percentile service time |                                           index-append |     2607.16 |      ms |
|                                   99th percentile service time |                                           index-append |     3881.16 |      ms |
|                                 99.9th percentile service time |                                           index-append |     10768.9 |      ms |
|                                99.99th percentile service time |                                           index-append |     20432.9 |      ms |
|                                  100th percentile service time |                                           index-append |     33889.3 |      ms |
|                                                     error rate |                                           index-append |           0 |       % |
|                                                 Min Throughput |                               wait-until-merges-finish |        0.01 |   ops/s |
|                                                Mean Throughput |                               wait-until-merges-finish |        0.01 |   ops/s |
|                                              Median Throughput |                               wait-until-merges-finish |        0.01 |   ops/s |
|                                                 Max Throughput |                               wait-until-merges-finish |        0.01 |   ops/s |
|                                        50th percentile latency |                               wait-until-merges-finish |             |         |
|                                        90th percentile latency |                               wait-until-merges-finish |             |         |
|                                        99th percentile latency |                               wait-until-merges-finish |             |         |
|                                      99.9th percentile latency |                               wait-until-merges-finish |             |         |
|                                     99.99th percentile latency |                               wait-until-merges-finish |             |         |
|                                       100th percentile latency |                               wait-until-merges-finish |     73864.1 |      ms |
|                                   50th percentile service time |                               wait-until-merges-finish |             |         |
|                                   90th percentile service time |                               wait-until-merges-finish |             |         |
|                                   99th percentile service time |                               wait-until-merges-finish |             |         |
|                                 99.9th percentile service time |                               wait-until-merges-finish |             |         |
|                                99.99th percentile service time |                               wait-until-merges-finish |             |         |
|                                  100th percentile service time |                               wait-until-merges-finish |     73864.1 |      ms |
|                                                     error rate |                               wait-until-merges-finish |           0 |       % |
|                                                 Min Throughput |                                                default |        8.01 |   ops/s |
|                                                Mean Throughput |                                                default |        8.01 |   ops/s |
|                                              Median Throughput |                                                default |        8.01 |   ops/s |
|                                                 Max Throughput |                                                default |        8.01 |   ops/s |
|                                        50th percentile latency |                                                default |     5.42994 |      ms |
|                                        90th percentile latency |                                                default |      6.1692 |      ms |
|                                        99th percentile latency |                                                default |     6.78539 |      ms |
|                                      99.9th percentile latency |                                                default |             |         |
|                                     99.99th percentile latency |                                                default |             |         |
|                                       100th percentile latency |                                                default |     7.05159 |      ms |
|                                   50th percentile service time |                                                default |     4.41577 |      ms |
|                                   90th percentile service time |                                                default |     4.63914 |      ms |
|                                   99th percentile service time |                                                default |     5.11656 |      ms |
|                                 99.9th percentile service time |                                                default |             |         |
|                                99.99th percentile service time |                                                default |             |         |
|                                  100th percentile service time |                                                default |     5.57454 |      ms |
|                                                     error rate |                                                default |           0 |       % |
|                                                 Min Throughput |                                                   term |       49.63 |   ops/s |
|                                                Mean Throughput |                                                   term |       49.64 |   ops/s |
|                                              Median Throughput |                                                   term |       49.64 |   ops/s |
|                                                 Max Throughput |                                                   term |       49.66 |   ops/s |
|                                        50th percentile latency |                                                   term |     7.34431 |      ms |
|                                        90th percentile latency |                                                   term |     8.21164 |      ms |
|                                        99th percentile latency |                                                   term |     8.69457 |      ms |
|                                      99.9th percentile latency |                                                   term |             |         |
|                                     99.99th percentile latency |                                                   term |             |         |
|                                       100th percentile latency |                                                   term |     11.0237 |      ms |
|                                   50th percentile service time |                                                   term |     6.08136 |      ms |
|                                   90th percentile service time |                                                   term |     6.60548 |      ms |
|                                   99th percentile service time |                                                   term |     7.29271 |      ms |
|                                 99.9th percentile service time |                                                   term |             |         |
|                                99.99th percentile service time |                                                   term |             |         |
|                                  100th percentile service time |                                                   term |     9.58338 |      ms |
|                                                     error rate |                                                   term |           0 |       % |
|                                                 Min Throughput |                                                  range |           1 |   ops/s |
|                                                Mean Throughput |                                                  range |        1.01 |   ops/s |
|                                              Median Throughput |                                                  range |        1.01 |   ops/s |
|                                                 Max Throughput |                                                  range |        1.01 |   ops/s |
|                                        50th percentile latency |                                                  range |      13.361 |      ms |
|                                        90th percentile latency |                                                  range |     14.8972 |      ms |
|                                        99th percentile latency |                                                  range |     17.7073 |      ms |
|                                      99.9th percentile latency |                                                  range |             |         |
|                                     99.99th percentile latency |                                                  range |             |         |
|                                       100th percentile latency |                                                  range |     21.3705 |      ms |
|                                   50th percentile service time |                                                  range |      11.371 |      ms |
|                                   90th percentile service time |                                                  range |     12.8476 |      ms |
|                                   99th percentile service time |                                                  range |     15.9071 |      ms |
|                                 99.9th percentile service time |                                                  range |             |         |
|                                99.99th percentile service time |                                                  range |             |         |
|                                  100th percentile service time |                                                  range |      18.787 |      ms |
|                                                     error rate |                                                  range |           0 |       % |
|                                                 Min Throughput |                                          200s-in-range |       32.99 |   ops/s |
|                                                Mean Throughput |                                          200s-in-range |       32.99 |   ops/s |
|                                              Median Throughput |                                          200s-in-range |       32.99 |   ops/s |
|                                                 Max Throughput |                                          200s-in-range |       32.99 |   ops/s |
|                                        50th percentile latency |                                          200s-in-range |     8.59151 |      ms |
|                                        90th percentile latency |                                          200s-in-range |      10.382 |      ms |
|                                        99th percentile latency |                                          200s-in-range |      11.525 |      ms |
|                                      99.9th percentile latency |                                          200s-in-range |             |         |
|                                     99.99th percentile latency |                                          200s-in-range |             |         |
|                                       100th percentile latency |                                          200s-in-range |     12.7522 |      ms |
|                                   50th percentile service time |                                          200s-in-range |     7.76262 |      ms |
|                                   90th percentile service time |                                          200s-in-range |     9.50643 |      ms |
|                                   99th percentile service time |                                          200s-in-range |     10.7138 |      ms |
|                                 99.9th percentile service time |                                          200s-in-range |             |         |
|                                99.99th percentile service time |                                          200s-in-range |             |         |
|                                  100th percentile service time |                                          200s-in-range |     11.6511 |      ms |
|                                                     error rate |                                          200s-in-range |           0 |       % |
|                                                 Min Throughput |                                          400s-in-range |          50 |   ops/s |
|                                                Mean Throughput |                                          400s-in-range |          50 |   ops/s |
|                                              Median Throughput |                                          400s-in-range |          50 |   ops/s |
|                                                 Max Throughput |                                          400s-in-range |          50 |   ops/s |
|                                        50th percentile latency |                                          400s-in-range |     5.20267 |      ms |
|                                        90th percentile latency |                                          400s-in-range |     6.02521 |      ms |
|                                        99th percentile latency |                                          400s-in-range |     6.58512 |      ms |
|                                      99.9th percentile latency |                                          400s-in-range |             |         |
|                                     99.99th percentile latency |                                          400s-in-range |             |         |
|                                       100th percentile latency |                                          400s-in-range |     8.36251 |      ms |
|                                   50th percentile service time |                                          400s-in-range |     4.43097 |      ms |
|                                   90th percentile service time |                                          400s-in-range |     5.14362 |      ms |
|                                   99th percentile service time |                                          400s-in-range |     5.47119 |      ms |
|                                 99.9th percentile service time |                                          400s-in-range |             |         |
|                                99.99th percentile service time |                                          400s-in-range |             |         |
|                                  100th percentile service time |                                          400s-in-range |     7.16229 |      ms |
|                                                     error rate |                                          400s-in-range |           0 |       % |
|                                                 Min Throughput |                                             hourly_agg |         0.2 |   ops/s |
|                                                Mean Throughput |                                             hourly_agg |         0.2 |   ops/s |
|                                              Median Throughput |                                             hourly_agg |         0.2 |   ops/s |
|                                                 Max Throughput |                                             hourly_agg |         0.2 |   ops/s |
|                                        50th percentile latency |                                             hourly_agg |     3360.17 |      ms |
|                                        90th percentile latency |                                             hourly_agg |     3421.77 |      ms |
|                                        99th percentile latency |                                             hourly_agg |     3473.77 |      ms |
|                                      99.9th percentile latency |                                             hourly_agg |             |         |
|                                     99.99th percentile latency |                                             hourly_agg |             |         |
|                                       100th percentile latency |                                             hourly_agg |      3477.8 |      ms |
|                                   50th percentile service time |                                             hourly_agg |     3357.88 |      ms |
|                                   90th percentile service time |                                             hourly_agg |      3419.1 |      ms |
|                                   99th percentile service time |                                             hourly_agg |     3471.17 |      ms |
|                                 99.9th percentile service time |                                             hourly_agg |             |         |
|                                99.99th percentile service time |                                             hourly_agg |             |         |
|                                  100th percentile service time |                                             hourly_agg |     3476.61 |      ms |
|                                                     error rate |                                             hourly_agg |           0 |       % |
|                                                 Min Throughput |                                                 scroll |       25.03 | pages/s |
|                                                Mean Throughput |                                                 scroll |       25.05 | pages/s |
|                                              Median Throughput |                                                 scroll |       25.05 | pages/s |
|                                                 Max Throughput |                                                 scroll |        25.1 | pages/s |
|                                        50th percentile latency |                                                 scroll |     561.454 |      ms |
|                                        90th percentile latency |                                                 scroll |     602.525 |      ms |
|                                        99th percentile latency |                                                 scroll |     847.018 |      ms |
|                                      99.9th percentile latency |                                                 scroll |             |         |
|                                     99.99th percentile latency |                                                 scroll |             |         |
|                                       100th percentile latency |                                                 scroll |     903.396 |      ms |
|                                   50th percentile service time |                                                 scroll |     559.712 |      ms |
|                                   90th percentile service time |                                                 scroll |     600.726 |      ms |
|                                   99th percentile service time |                                                 scroll |     845.016 |      ms |
|                                 99.9th percentile service time |                                                 scroll |             |         |
|                                99.99th percentile service time |                                                 scroll |             |         |
|                                  100th percentile service time |                                                 scroll |     902.295 |      ms |
|                                                     error rate |                                                 scroll |           0 |       % |
|                                                 Min Throughput |                                    desc_sort_timestamp |         0.5 |   ops/s |
|                                                Mean Throughput |                                    desc_sort_timestamp |         0.5 |   ops/s |
|                                              Median Throughput |                                    desc_sort_timestamp |         0.5 |   ops/s |
|                                                 Max Throughput |                                    desc_sort_timestamp |         0.5 |   ops/s |
|                                        50th percentile latency |                                    desc_sort_timestamp |     112.602 |      ms |
|                                        90th percentile latency |                                    desc_sort_timestamp |     161.829 |      ms |
|                                        99th percentile latency |                                    desc_sort_timestamp |      185.02 |      ms |
|                                      99.9th percentile latency |                                    desc_sort_timestamp |             |         |
|                                     99.99th percentile latency |                                    desc_sort_timestamp |             |         |
|                                       100th percentile latency |                                    desc_sort_timestamp |     198.042 |      ms |
|                                   50th percentile service time |                                    desc_sort_timestamp |     108.725 |      ms |
|                                   90th percentile service time |                                    desc_sort_timestamp |     158.756 |      ms |
|                                   99th percentile service time |                                    desc_sort_timestamp |     182.424 |      ms |
|                                 99.9th percentile service time |                                    desc_sort_timestamp |             |         |
|                                99.99th percentile service time |                                    desc_sort_timestamp |             |         |
|                                  100th percentile service time |                                    desc_sort_timestamp |     195.011 |      ms |
|                                                     error rate |                                    desc_sort_timestamp |           0 |       % |
|                                                 Min Throughput |                                     asc_sort_timestamp |         0.5 |   ops/s |
|                                                Mean Throughput |                                     asc_sort_timestamp |         0.5 |   ops/s |
|                                              Median Throughput |                                     asc_sort_timestamp |         0.5 |   ops/s |
|                                                 Max Throughput |                                     asc_sort_timestamp |         0.5 |   ops/s |
|                                        50th percentile latency |                                     asc_sort_timestamp |     9.59216 |      ms |
|                                        90th percentile latency |                                     asc_sort_timestamp |     11.2697 |      ms |
|                                        99th percentile latency |                                     asc_sort_timestamp |     15.3794 |      ms |
|                                      99.9th percentile latency |                                     asc_sort_timestamp |             |         |
|                                     99.99th percentile latency |                                     asc_sort_timestamp |             |         |
|                                       100th percentile latency |                                     asc_sort_timestamp |     328.095 |      ms |
|                                   50th percentile service time |                                     asc_sort_timestamp |     7.39149 |      ms |
|                                   90th percentile service time |                                     asc_sort_timestamp |     8.64702 |      ms |
|                                   99th percentile service time |                                     asc_sort_timestamp |     12.3307 |      ms |
|                                 99.9th percentile service time |                                     asc_sort_timestamp |             |         |
|                                99.99th percentile service time |                                     asc_sort_timestamp |             |         |
|                                  100th percentile service time |                                     asc_sort_timestamp |     325.086 |      ms |
|                                                     error rate |                                     asc_sort_timestamp |           0 |       % |
|                                                 Min Throughput |                         desc_sort_with_after_timestamp |         0.5 |   ops/s |
|                                                Mean Throughput |                         desc_sort_with_after_timestamp |        0.51 |   ops/s |
|                                              Median Throughput |                         desc_sort_with_after_timestamp |        0.51 |   ops/s |
|                                                 Max Throughput |                         desc_sort_with_after_timestamp |        0.54 |   ops/s |
|                                        50th percentile latency |                         desc_sort_with_after_timestamp |     199.694 |      ms |
|                                        90th percentile latency |                         desc_sort_with_after_timestamp |     232.433 |      ms |
|                                        99th percentile latency |                         desc_sort_with_after_timestamp |     251.766 |      ms |
|                                      99.9th percentile latency |                         desc_sort_with_after_timestamp |             |         |
|                                     99.99th percentile latency |                         desc_sort_with_after_timestamp |             |         |
|                                       100th percentile latency |                         desc_sort_with_after_timestamp |     253.875 |      ms |
|                                   50th percentile service time |                         desc_sort_with_after_timestamp |     196.399 |      ms |
|                                   90th percentile service time |                         desc_sort_with_after_timestamp |     231.613 |      ms |
|                                   99th percentile service time |                         desc_sort_with_after_timestamp |     250.976 |      ms |
|                                 99.9th percentile service time |                         desc_sort_with_after_timestamp |             |         |
|                                99.99th percentile service time |                         desc_sort_with_after_timestamp |             |         |
|                                  100th percentile service time |                         desc_sort_with_after_timestamp |     251.597 |      ms |
|                                                     error rate |                         desc_sort_with_after_timestamp |           0 |       % |
|                                                 Min Throughput |                          asc_sort_with_after_timestamp |         0.5 |   ops/s |
|                                                Mean Throughput |                          asc_sort_with_after_timestamp |        0.51 |   ops/s |
|                                              Median Throughput |                          asc_sort_with_after_timestamp |        0.51 |   ops/s |
|                                                 Max Throughput |                          asc_sort_with_after_timestamp |        0.55 |   ops/s |
|                                        50th percentile latency |                          asc_sort_with_after_timestamp |     10.7918 |      ms |
|                                        90th percentile latency |                          asc_sort_with_after_timestamp |     11.9547 |      ms |
|                                        99th percentile latency |                          asc_sort_with_after_timestamp |     13.6805 |      ms |
|                                      99.9th percentile latency |                          asc_sort_with_after_timestamp |             |         |
|                                     99.99th percentile latency |                          asc_sort_with_after_timestamp |             |         |
|                                       100th percentile latency |                          asc_sort_with_after_timestamp |     16.2472 |      ms |
|                                   50th percentile service time |                          asc_sort_with_after_timestamp |      8.4639 |      ms |
|                                   90th percentile service time |                          asc_sort_with_after_timestamp |     9.48912 |      ms |
|                                   99th percentile service time |                          asc_sort_with_after_timestamp |     10.6422 |      ms |
|                                 99.9th percentile service time |                          asc_sort_with_after_timestamp |             |         |
|                                99.99th percentile service time |                          asc_sort_with_after_timestamp |             |         |
|                                  100th percentile service time |                          asc_sort_with_after_timestamp |      14.444 |      ms |
|                                                     error rate |                          asc_sort_with_after_timestamp |           0 |       % |
|                                                 Min Throughput |                         wait-until-merges-1-seg-finish |       52.14 |   ops/s |
|                                                Mean Throughput |                         wait-until-merges-1-seg-finish |       52.14 |   ops/s |
|                                              Median Throughput |                         wait-until-merges-1-seg-finish |       52.14 |   ops/s |
|                                                 Max Throughput |                         wait-until-merges-1-seg-finish |       52.14 |   ops/s |
|                                        50th percentile latency |                         wait-until-merges-1-seg-finish |             |         |
|                                        90th percentile latency |                         wait-until-merges-1-seg-finish |             |         |
|                                        99th percentile latency |                         wait-until-merges-1-seg-finish |             |         |
|                                      99.9th percentile latency |                         wait-until-merges-1-seg-finish |             |         |
|                                     99.99th percentile latency |                         wait-until-merges-1-seg-finish |             |         |
|                                       100th percentile latency |                         wait-until-merges-1-seg-finish |     18.5744 |      ms |
|                                   50th percentile service time |                         wait-until-merges-1-seg-finish |             |         |
|                                   90th percentile service time |                         wait-until-merges-1-seg-finish |             |         |
|                                   99th percentile service time |                         wait-until-merges-1-seg-finish |             |         |
|                                 99.9th percentile service time |                         wait-until-merges-1-seg-finish |             |         |
|                                99.99th percentile service time |                         wait-until-merges-1-seg-finish |             |         |
|                                  100th percentile service time |                         wait-until-merges-1-seg-finish |     18.5744 |      ms |
|                                                     error rate |                         wait-until-merges-1-seg-finish |           0 |       % |
|                                                 Min Throughput |            desc-sort-timestamp-after-force-merge-1-seg |           2 |   ops/s |
|                                                Mean Throughput |            desc-sort-timestamp-after-force-merge-1-seg |           2 |   ops/s |
|                                              Median Throughput |            desc-sort-timestamp-after-force-merge-1-seg |           2 |   ops/s |
|                                                 Max Throughput |            desc-sort-timestamp-after-force-merge-1-seg |           2 |   ops/s |
|                                        50th percentile latency |            desc-sort-timestamp-after-force-merge-1-seg |     235.927 |      ms |
|                                        90th percentile latency |            desc-sort-timestamp-after-force-merge-1-seg |     297.068 |      ms |
|                                        99th percentile latency |            desc-sort-timestamp-after-force-merge-1-seg |     306.319 |      ms |
|                                      99.9th percentile latency |            desc-sort-timestamp-after-force-merge-1-seg |             |         |
|                                     99.99th percentile latency |            desc-sort-timestamp-after-force-merge-1-seg |             |         |
|                                       100th percentile latency |            desc-sort-timestamp-after-force-merge-1-seg |     311.942 |      ms |
|                                   50th percentile service time |            desc-sort-timestamp-after-force-merge-1-seg |     234.762 |      ms |
|                                   90th percentile service time |            desc-sort-timestamp-after-force-merge-1-seg |     296.117 |      ms |
|                                   99th percentile service time |            desc-sort-timestamp-after-force-merge-1-seg |     304.592 |      ms |
|                                 99.9th percentile service time |            desc-sort-timestamp-after-force-merge-1-seg |             |         |
|                                99.99th percentile service time |            desc-sort-timestamp-after-force-merge-1-seg |             |         |
|                                  100th percentile service time |            desc-sort-timestamp-after-force-merge-1-seg |     310.987 |      ms |
|                                                     error rate |            desc-sort-timestamp-after-force-merge-1-seg |           0 |       % |
|                                                 Min Throughput |             asc-sort-timestamp-after-force-merge-1-seg |        2.01 |   ops/s |
|                                                Mean Throughput |             asc-sort-timestamp-after-force-merge-1-seg |        2.01 |   ops/s |
|                                              Median Throughput |             asc-sort-timestamp-after-force-merge-1-seg |        2.01 |   ops/s |
|                                                 Max Throughput |             asc-sort-timestamp-after-force-merge-1-seg |        2.01 |   ops/s |
|                                        50th percentile latency |             asc-sort-timestamp-after-force-merge-1-seg |     12.1118 |      ms |
|                                        90th percentile latency |             asc-sort-timestamp-after-force-merge-1-seg |     13.6598 |      ms |
|                                        99th percentile latency |             asc-sort-timestamp-after-force-merge-1-seg |     15.8833 |      ms |
|                                      99.9th percentile latency |             asc-sort-timestamp-after-force-merge-1-seg |             |         |
|                                     99.99th percentile latency |             asc-sort-timestamp-after-force-merge-1-seg |             |         |
|                                       100th percentile latency |             asc-sort-timestamp-after-force-merge-1-seg |     16.5428 |      ms |
|                                   50th percentile service time |             asc-sort-timestamp-after-force-merge-1-seg |     10.9654 |      ms |
|                                   90th percentile service time |             asc-sort-timestamp-after-force-merge-1-seg |      12.292 |      ms |
|                                   99th percentile service time |             asc-sort-timestamp-after-force-merge-1-seg |     14.5254 |      ms |
|                                 99.9th percentile service time |             asc-sort-timestamp-after-force-merge-1-seg |             |         |
|                                99.99th percentile service time |             asc-sort-timestamp-after-force-merge-1-seg |             |         |
|                                  100th percentile service time |             asc-sort-timestamp-after-force-merge-1-seg |     15.6817 |      ms |
|                                                     error rate |             asc-sort-timestamp-after-force-merge-1-seg |           0 |       % |
|                                                 Min Throughput | desc-sort-with-after-timestamp-after-force-merge-1-seg |         0.5 |   ops/s |
|                                                Mean Throughput | desc-sort-with-after-timestamp-after-force-merge-1-seg |        0.51 |   ops/s |
|                                              Median Throughput | desc-sort-with-after-timestamp-after-force-merge-1-seg |        0.51 |   ops/s |
|                                                 Max Throughput | desc-sort-with-after-timestamp-after-force-merge-1-seg |        0.53 |   ops/s |
|                                        50th percentile latency | desc-sort-with-after-timestamp-after-force-merge-1-seg |      181.06 |      ms |
|                                        90th percentile latency | desc-sort-with-after-timestamp-after-force-merge-1-seg |     196.762 |      ms |
|                                        99th percentile latency | desc-sort-with-after-timestamp-after-force-merge-1-seg |     211.176 |      ms |
|                                      99.9th percentile latency | desc-sort-with-after-timestamp-after-force-merge-1-seg |             |         |
|                                     99.99th percentile latency | desc-sort-with-after-timestamp-after-force-merge-1-seg |             |         |
|                                       100th percentile latency | desc-sort-with-after-timestamp-after-force-merge-1-seg |     255.604 |      ms |
|                                   50th percentile service time | desc-sort-with-after-timestamp-after-force-merge-1-seg |     179.353 |      ms |
|                                   90th percentile service time | desc-sort-with-after-timestamp-after-force-merge-1-seg |     194.426 |      ms |
|                                   99th percentile service time | desc-sort-with-after-timestamp-after-force-merge-1-seg |     209.644 |      ms |
|                                 99.9th percentile service time | desc-sort-with-after-timestamp-after-force-merge-1-seg |             |         |
|                                99.99th percentile service time | desc-sort-with-after-timestamp-after-force-merge-1-seg |             |         |
|                                  100th percentile service time | desc-sort-with-after-timestamp-after-force-merge-1-seg |      253.24 |      ms |
|                                                     error rate | desc-sort-with-after-timestamp-after-force-merge-1-seg |           0 |       % |
|                                                 Min Throughput |  asc-sort-with-after-timestamp-after-force-merge-1-seg |         0.5 |   ops/s |
|                                                Mean Throughput |  asc-sort-with-after-timestamp-after-force-merge-1-seg |        0.51 |   ops/s |
|                                              Median Throughput |  asc-sort-with-after-timestamp-after-force-merge-1-seg |        0.51 |   ops/s |
|                                                 Max Throughput |  asc-sort-with-after-timestamp-after-force-merge-1-seg |        0.55 |   ops/s |
|                                        50th percentile latency |  asc-sort-with-after-timestamp-after-force-merge-1-seg |     41.6164 |      ms |
|                                        90th percentile latency |  asc-sort-with-after-timestamp-after-force-merge-1-seg |      56.693 |      ms |
|                                        99th percentile latency |  asc-sort-with-after-timestamp-after-force-merge-1-seg |     67.5295 |      ms |
|                                      99.9th percentile latency |  asc-sort-with-after-timestamp-after-force-merge-1-seg |             |         |
|                                     99.99th percentile latency |  asc-sort-with-after-timestamp-after-force-merge-1-seg |             |         |
|                                       100th percentile latency |  asc-sort-with-after-timestamp-after-force-merge-1-seg |     68.5078 |      ms |
|                                   50th percentile service time |  asc-sort-with-after-timestamp-after-force-merge-1-seg |     38.7932 |      ms |
|                                   90th percentile service time |  asc-sort-with-after-timestamp-after-force-merge-1-seg |     54.5373 |      ms |
|                                   99th percentile service time |  asc-sort-with-after-timestamp-after-force-merge-1-seg |     65.3345 |      ms |
|                                 99.9th percentile service time |  asc-sort-with-after-timestamp-after-force-merge-1-seg |             |         |
|                                99.99th percentile service time |  asc-sort-with-after-timestamp-after-force-merge-1-seg |             |         |
|                                  100th percentile service time |  asc-sort-with-after-timestamp-after-force-merge-1-seg |     66.5449 |      ms |
|                                                     error rate |  asc-sort-with-after-timestamp-after-force-merge-1-seg |           0 |       % |


----------------------------------
[INFO] SUCCESS (took 8716 seconds)
----------------------------------

http_logs,remote,bic-24,ri-10s,bri-650ms (Run 1)

|                                                         Metric |                                                   Task |       Value |    Unit |
|---------------------------------------------------------------:|-------------------------------------------------------:|------------:|--------:|
|                     Cumulative indexing time of primary shards |                                                        |     243.899 |     min |
|             Min cumulative indexing time across primary shards |                                                        |           0 |     min |
|          Median cumulative indexing time across primary shards |                                                        |     1.71882 |     min |
|             Max cumulative indexing time across primary shards |                                                        |     34.3129 |     min |
|            Cumulative indexing throttle time of primary shards |                                                        |           0 |     min |
|    Min cumulative indexing throttle time across primary shards |                                                        |           0 |     min |
| Median cumulative indexing throttle time across primary shards |                                                        |           0 |     min |
|    Max cumulative indexing throttle time across primary shards |                                                        |           0 |     min |
|                        Cumulative merge time of primary shards |                                                        |     90.4196 |     min |
|                       Cumulative merge count of primary shards |                                                        |         300 |         |
|                Min cumulative merge time across primary shards |                                                        |           0 |     min |
|             Median cumulative merge time across primary shards |                                                        |    0.365033 |     min |
|                Max cumulative merge time across primary shards |                                                        |     13.5249 |     min |
|               Cumulative merge throttle time of primary shards |                                                        |      32.217 |     min |
|       Min cumulative merge throttle time across primary shards |                                                        |           0 |     min |
|    Median cumulative merge throttle time across primary shards |                                                        |    0.044325 |     min |
|       Max cumulative merge throttle time across primary shards |                                                        |     5.45342 |     min |
|                      Cumulative refresh time of primary shards |                                                        |     25.5317 |     min |
|                     Cumulative refresh count of primary shards |                                                        |        2012 |         |
|              Min cumulative refresh time across primary shards |                                                        |           0 |     min |
|           Median cumulative refresh time across primary shards |                                                        |     0.17295 |     min |
|              Max cumulative refresh time across primary shards |                                                        |     3.27083 |     min |
|                        Cumulative flush time of primary shards |                                                        |     1.87372 |     min |
|                       Cumulative flush count of primary shards |                                                        |         126 |         |
|                Min cumulative flush time across primary shards |                                                        | 0.000183333 |     min |
|             Median cumulative flush time across primary shards |                                                        |    0.004025 |     min |
|                Max cumulative flush time across primary shards |                                                        |      0.3854 |     min |
|                                        Total Young Gen GC time |                                                        |      36.579 |       s |
|                                       Total Young Gen GC count |                                                        |         995 |         |
|                                          Total Old Gen GC time |                                                        |           0 |       s |
|                                         Total Old Gen GC count |                                                        |           0 |         |
|                                                     Store size |                                                        |     35.2043 |      GB |
|                                                  Translog size |                                                        | 4.91738e-06 |      GB |
|                                         Heap used for segments |                                                        |           0 |      MB |
|                                       Heap used for doc values |                                                        |           0 |      MB |
|                                            Heap used for terms |                                                        |           0 |      MB |
|                                            Heap used for norms |                                                        |           0 |      MB |
|                                           Heap used for points |                                                        |           0 |      MB |
|                                    Heap used for stored fields |                                                        |           0 |      MB |
|                                                  Segment count |                                                        |          42 |         |
|                                                 Min Throughput |                                           index-append |     75413.9 |  docs/s |
|                                                Mean Throughput |                                           index-append |     83155.7 |  docs/s |
|                                              Median Throughput |                                           index-append |     84021.1 |  docs/s |
|                                                 Max Throughput |                                           index-append |       88412 |  docs/s |
|                                        50th percentile latency |                                           index-append |     1379.71 |      ms |
|                                        90th percentile latency |                                           index-append |     2089.09 |      ms |
|                                        99th percentile latency |                                           index-append |     2887.72 |      ms |
|                                      99.9th percentile latency |                                           index-append |     3474.38 |      ms |
|                                     99.99th percentile latency |                                           index-append |     3913.71 |      ms |
|                                       100th percentile latency |                                           index-append |     4538.23 |      ms |
|                                   50th percentile service time |                                           index-append |     1379.71 |      ms |
|                                   90th percentile service time |                                           index-append |     2089.09 |      ms |
|                                   99th percentile service time |                                           index-append |     2887.72 |      ms |
|                                 99.9th percentile service time |                                           index-append |     3474.38 |      ms |
|                                99.99th percentile service time |                                           index-append |     3913.71 |      ms |
|                                  100th percentile service time |                                           index-append |     4538.23 |      ms |
|                                                     error rate |                                           index-append |           0 |       % |
|                                                 Min Throughput |                               wait-until-merges-finish |       12.66 |   ops/s |
|                                                Mean Throughput |                               wait-until-merges-finish |       12.66 |   ops/s |
|                                              Median Throughput |                               wait-until-merges-finish |       12.66 |   ops/s |
|                                                 Max Throughput |                               wait-until-merges-finish |       12.66 |   ops/s |
|                                        50th percentile latency |                               wait-until-merges-finish |             |         |
|                                        90th percentile latency |                               wait-until-merges-finish |             |         |
|                                        99th percentile latency |                               wait-until-merges-finish |             |         |
|                                      99.9th percentile latency |                               wait-until-merges-finish |             |         |
|                                     99.99th percentile latency |                               wait-until-merges-finish |             |         |
|                                       100th percentile latency |                               wait-until-merges-finish |     78.5112 |      ms |
|                                   50th percentile service time |                               wait-until-merges-finish |             |         |
|                                   90th percentile service time |                               wait-until-merges-finish |             |         |
|                                   99th percentile service time |                               wait-until-merges-finish |             |         |
|                                 99.9th percentile service time |                               wait-until-merges-finish |             |         |
|                                99.99th percentile service time |                               wait-until-merges-finish |             |         |
|                                  100th percentile service time |                               wait-until-merges-finish |     78.5112 |      ms |
|                                                     error rate |                               wait-until-merges-finish |           0 |       % |
|                                                 Min Throughput |                                                default |        7.99 |   ops/s |
|                                                Mean Throughput |                                                default |        7.99 |   ops/s |
|                                              Median Throughput |                                                default |        7.99 |   ops/s |
|                                                 Max Throughput |                                                default |        7.99 |   ops/s |
|                                        50th percentile latency |                                                default |     7.30949 |      ms |
|                                        90th percentile latency |                                                default |     8.70003 |      ms |
|                                        99th percentile latency |                                                default |     11.4446 |      ms |
|                                      99.9th percentile latency |                                                default |             |         |
|                                     99.99th percentile latency |                                                default |             |         |
|                                       100th percentile latency |                                                default |     13.5198 |      ms |
|                                   50th percentile service time |                                                default |     5.74037 |      ms |
|                                   90th percentile service time |                                                default |     7.03842 |      ms |
|                                   99th percentile service time |                                                default |     9.10095 |      ms |
|                                 99.9th percentile service time |                                                default |             |         |
|                                99.99th percentile service time |                                                default |             |         |
|                                  100th percentile service time |                                                default |      11.275 |      ms |
|                                                     error rate |                                                default |           0 |       % |
|                                                 Min Throughput |                                                   term |       49.41 |   ops/s |
|                                                Mean Throughput |                                                   term |       49.44 |   ops/s |
|                                              Median Throughput |                                                   term |       49.44 |   ops/s |
|                                                 Max Throughput |                                                   term |       49.46 |   ops/s |
|                                        50th percentile latency |                                                   term |     9.21367 |      ms |
|                                        90th percentile latency |                                                   term |      10.794 |      ms |
|                                        99th percentile latency |                                                   term |     14.0087 |      ms |
|                                      99.9th percentile latency |                                                   term |             |         |
|                                     99.99th percentile latency |                                                   term |             |         |
|                                       100th percentile latency |                                                   term |     14.0468 |      ms |
|                                   50th percentile service time |                                                   term |     8.37353 |      ms |
|                                   90th percentile service time |                                                   term |     10.0078 |      ms |
|                                   99th percentile service time |                                                   term |     12.8309 |      ms |
|                                 99.9th percentile service time |                                                   term |             |         |
|                                99.99th percentile service time |                                                   term |             |         |
|                                  100th percentile service time |                                                   term |     13.7534 |      ms |
|                                                     error rate |                                                   term |           0 |       % |
|                                                 Min Throughput |                                                  range |           1 |   ops/s |
|                                                Mean Throughput |                                                  range |        1.01 |   ops/s |
|                                              Median Throughput |                                                  range |        1.01 |   ops/s |
|                                                 Max Throughput |                                                  range |        1.01 |   ops/s |
|                                        50th percentile latency |                                                  range |     13.4384 |      ms |
|                                        90th percentile latency |                                                  range |     15.7938 |      ms |
|                                        99th percentile latency |                                                  range |     18.5861 |      ms |
|                                      99.9th percentile latency |                                                  range |             |         |
|                                     99.99th percentile latency |                                                  range |             |         |
|                                       100th percentile latency |                                                  range |     19.4285 |      ms |
|                                   50th percentile service time |                                                  range |     11.5017 |      ms |
|                                   90th percentile service time |                                                  range |     14.1488 |      ms |
|                                   99th percentile service time |                                                  range |     16.8445 |      ms |
|                                 99.9th percentile service time |                                                  range |             |         |
|                                99.99th percentile service time |                                                  range |             |         |
|                                  100th percentile service time |                                                  range |     17.7068 |      ms |
|                                                     error rate |                                                  range |           0 |       % |
|                                                 Min Throughput |                                          200s-in-range |       32.84 |   ops/s |
|                                                Mean Throughput |                                          200s-in-range |       32.85 |   ops/s |
|                                              Median Throughput |                                          200s-in-range |       32.85 |   ops/s |
|                                                 Max Throughput |                                          200s-in-range |       32.86 |   ops/s |
|                                        50th percentile latency |                                          200s-in-range |     9.23236 |      ms |
|                                        90th percentile latency |                                          200s-in-range |     12.4212 |      ms |
|                                        99th percentile latency |                                          200s-in-range |     15.8735 |      ms |
|                                      99.9th percentile latency |                                          200s-in-range |             |         |
|                                     99.99th percentile latency |                                          200s-in-range |             |         |
|                                       100th percentile latency |                                          200s-in-range |     18.6052 |      ms |
|                                   50th percentile service time |                                          200s-in-range |     8.18362 |      ms |
|                                   90th percentile service time |                                          200s-in-range |     11.2612 |      ms |
|                                   99th percentile service time |                                          200s-in-range |     15.1192 |      ms |
|                                 99.9th percentile service time |                                          200s-in-range |             |         |
|                                99.99th percentile service time |                                          200s-in-range |             |         |
|                                  100th percentile service time |                                          200s-in-range |     17.6669 |      ms |
|                                                     error rate |                                          200s-in-range |           0 |       % |
|                                                 Min Throughput |                                          400s-in-range |       49.95 |   ops/s |
|                                                Mean Throughput |                                          400s-in-range |       49.96 |   ops/s |
|                                              Median Throughput |                                          400s-in-range |       49.96 |   ops/s |
|                                                 Max Throughput |                                          400s-in-range |       49.96 |   ops/s |
|                                        50th percentile latency |                                          400s-in-range |     5.78699 |      ms |
|                                        90th percentile latency |                                          400s-in-range |     6.98783 |      ms |
|                                        99th percentile latency |                                          400s-in-range |     8.08313 |      ms |
|                                      99.9th percentile latency |                                          400s-in-range |             |         |
|                                     99.99th percentile latency |                                          400s-in-range |             |         |
|                                       100th percentile latency |                                          400s-in-range |     12.0686 |      ms |
|                                   50th percentile service time |                                          400s-in-range |     4.96752 |      ms |
|                                   90th percentile service time |                                          400s-in-range |     6.03033 |      ms |
|                                   99th percentile service time |                                          400s-in-range |     7.14605 |      ms |
|                                 99.9th percentile service time |                                          400s-in-range |             |         |
|                                99.99th percentile service time |                                          400s-in-range |             |         |
|                                  100th percentile service time |                                          400s-in-range |     10.8169 |      ms |
|                                                     error rate |                                          400s-in-range |           0 |       % |
|                                                 Min Throughput |                                             hourly_agg |         0.2 |   ops/s |
|                                                Mean Throughput |                                             hourly_agg |         0.2 |   ops/s |
|                                              Median Throughput |                                             hourly_agg |         0.2 |   ops/s |
|                                                 Max Throughput |                                             hourly_agg |         0.2 |   ops/s |
|                                        50th percentile latency |                                             hourly_agg |     3788.76 |      ms |
|                                        90th percentile latency |                                             hourly_agg |     4579.05 |      ms |
|                                        99th percentile latency |                                             hourly_agg |     4947.89 |      ms |
|                                      99.9th percentile latency |                                             hourly_agg |             |         |
|                                     99.99th percentile latency |                                             hourly_agg |             |         |
|                                       100th percentile latency |                                             hourly_agg |     4959.17 |      ms |
|                                   50th percentile service time |                                             hourly_agg |     3786.84 |      ms |
|                                   90th percentile service time |                                             hourly_agg |     4577.12 |      ms |
|                                   99th percentile service time |                                             hourly_agg |     4946.41 |      ms |
|                                 99.9th percentile service time |                                             hourly_agg |             |         |
|                                99.99th percentile service time |                                             hourly_agg |             |         |
|                                  100th percentile service time |                                             hourly_agg |     4958.02 |      ms |
|                                                     error rate |                                             hourly_agg |           0 |       % |
|                                                 Min Throughput |                                                 scroll |       24.84 | pages/s |
|                                                Mean Throughput |                                                 scroll |       24.96 | pages/s |
|                                              Median Throughput |                                                 scroll |       24.96 | pages/s |
|                                                 Max Throughput |                                                 scroll |       24.98 | pages/s |
|                                        50th percentile latency |                                                 scroll |     556.857 |      ms |
|                                        90th percentile latency |                                                 scroll |     620.327 |      ms |
|                                        99th percentile latency |                                                 scroll |     1211.94 |      ms |
|                                      99.9th percentile latency |                                                 scroll |             |         |
|                                     99.99th percentile latency |                                                 scroll |             |         |
|                                       100th percentile latency |                                                 scroll |     2107.16 |      ms |
|                                   50th percentile service time |                                                 scroll |     555.176 |      ms |
|                                   90th percentile service time |                                                 scroll |     617.416 |      ms |
|                                   99th percentile service time |                                                 scroll |     639.998 |      ms |
|                                 99.9th percentile service time |                                                 scroll |             |         |
|                                99.99th percentile service time |                                                 scroll |             |         |
|                                  100th percentile service time |                                                 scroll |     2105.14 |      ms |
|                                                     error rate |                                                 scroll |           0 |       % |
|                                                 Min Throughput |                                    desc_sort_timestamp |         0.5 |   ops/s |
|                                                Mean Throughput |                                    desc_sort_timestamp |         0.5 |   ops/s |
|                                              Median Throughput |                                    desc_sort_timestamp |         0.5 |   ops/s |
|                                                 Max Throughput |                                    desc_sort_timestamp |         0.5 |   ops/s |
|                                        50th percentile latency |                                    desc_sort_timestamp |     142.155 |      ms |
|                                        90th percentile latency |                                    desc_sort_timestamp |     237.314 |      ms |
|                                        99th percentile latency |                                    desc_sort_timestamp |     262.572 |      ms |
|                                      99.9th percentile latency |                                    desc_sort_timestamp |             |         |
|                                     99.99th percentile latency |                                    desc_sort_timestamp |             |         |
|                                       100th percentile latency |                                    desc_sort_timestamp |     262.688 |      ms |
|                                   50th percentile service time |                                    desc_sort_timestamp |     140.794 |      ms |
|                                   90th percentile service time |                                    desc_sort_timestamp |     235.548 |      ms |
|                                   99th percentile service time |                                    desc_sort_timestamp |      260.34 |      ms |
|                                 99.9th percentile service time |                                    desc_sort_timestamp |             |         |
|                                99.99th percentile service time |                                    desc_sort_timestamp |             |         |
|                                  100th percentile service time |                                    desc_sort_timestamp |     261.254 |      ms |
|                                                     error rate |                                    desc_sort_timestamp |           0 |       % |
|                                                 Min Throughput |                                     asc_sort_timestamp |         0.5 |   ops/s |
|                                                Mean Throughput |                                     asc_sort_timestamp |         0.5 |   ops/s |
|                                              Median Throughput |                                     asc_sort_timestamp |         0.5 |   ops/s |
|                                                 Max Throughput |                                     asc_sort_timestamp |         0.5 |   ops/s |
|                                        50th percentile latency |                                     asc_sort_timestamp |     11.3973 |      ms |
|                                        90th percentile latency |                                     asc_sort_timestamp |     14.6765 |      ms |
|                                        99th percentile latency |                                     asc_sort_timestamp |     27.1948 |      ms |
|                                      99.9th percentile latency |                                     asc_sort_timestamp |             |         |
|                                     99.99th percentile latency |                                     asc_sort_timestamp |             |         |
|                                       100th percentile latency |                                     asc_sort_timestamp |     52.8753 |      ms |
|                                   50th percentile service time |                                     asc_sort_timestamp |     8.94394 |      ms |
|                                   90th percentile service time |                                     asc_sort_timestamp |     12.2394 |      ms |
|                                   99th percentile service time |                                     asc_sort_timestamp |     25.2115 |      ms |
|                                 99.9th percentile service time |                                     asc_sort_timestamp |             |         |
|                                99.99th percentile service time |                                     asc_sort_timestamp |             |         |
|                                  100th percentile service time |                                     asc_sort_timestamp |     51.2788 |      ms |
|                                                     error rate |                                     asc_sort_timestamp |           0 |       % |
|                                                 Min Throughput |                         desc_sort_with_after_timestamp |         0.5 |   ops/s |
|                                                Mean Throughput |                         desc_sort_with_after_timestamp |        0.51 |   ops/s |
|                                              Median Throughput |                         desc_sort_with_after_timestamp |        0.51 |   ops/s |
|                                                 Max Throughput |                         desc_sort_with_after_timestamp |        0.55 |   ops/s |
|                                        50th percentile latency |                         desc_sort_with_after_timestamp |     108.331 |      ms |
|                                        90th percentile latency |                         desc_sort_with_after_timestamp |     168.142 |      ms |
|                                        99th percentile latency |                         desc_sort_with_after_timestamp |     171.741 |      ms |
|                                      99.9th percentile latency |                         desc_sort_with_after_timestamp |             |         |
|                                     99.99th percentile latency |                         desc_sort_with_after_timestamp |             |         |
|                                       100th percentile latency |                         desc_sort_with_after_timestamp |     172.975 |      ms |
|                                   50th percentile service time |                         desc_sort_with_after_timestamp |     105.471 |      ms |
|                                   90th percentile service time |                         desc_sort_with_after_timestamp |     166.378 |      ms |
|                                   99th percentile service time |                         desc_sort_with_after_timestamp |     168.904 |      ms |
|                                 99.9th percentile service time |                         desc_sort_with_after_timestamp |             |         |
|                                99.99th percentile service time |                         desc_sort_with_after_timestamp |             |         |
|                                  100th percentile service time |                         desc_sort_with_after_timestamp |     169.316 |      ms |
|                                                     error rate |                         desc_sort_with_after_timestamp |           0 |       % |
|                                                 Min Throughput |                          asc_sort_with_after_timestamp |         0.5 |   ops/s |
|                                                Mean Throughput |                          asc_sort_with_after_timestamp |        0.51 |   ops/s |
|                                              Median Throughput |                          asc_sort_with_after_timestamp |        0.51 |   ops/s |
|                                                 Max Throughput |                          asc_sort_with_after_timestamp |        0.55 |   ops/s |
|                                        50th percentile latency |                          asc_sort_with_after_timestamp |     42.8702 |      ms |
|                                        90th percentile latency |                          asc_sort_with_after_timestamp |     51.6736 |      ms |
|                                        99th percentile latency |                          asc_sort_with_after_timestamp |     60.5976 |      ms |
|                                      99.9th percentile latency |                          asc_sort_with_after_timestamp |             |         |
|                                     99.99th percentile latency |                          asc_sort_with_after_timestamp |             |         |
|                                       100th percentile latency |                          asc_sort_with_after_timestamp |     78.0527 |      ms |
|                                   50th percentile service time |                          asc_sort_with_after_timestamp |     40.8591 |      ms |
|                                   90th percentile service time |                          asc_sort_with_after_timestamp |     49.3633 |      ms |
|                                   99th percentile service time |                          asc_sort_with_after_timestamp |     57.3054 |      ms |
|                                 99.9th percentile service time |                          asc_sort_with_after_timestamp |             |         |
|                                99.99th percentile service time |                          asc_sort_with_after_timestamp |             |         |
|                                  100th percentile service time |                          asc_sort_with_after_timestamp |     75.1534 |      ms |
|                                                     error rate |                          asc_sort_with_after_timestamp |           0 |       % |
|                                                 Min Throughput |                         wait-until-merges-1-seg-finish |       52.07 |   ops/s |
|                                                Mean Throughput |                         wait-until-merges-1-seg-finish |       52.07 |   ops/s |
|                                              Median Throughput |                         wait-until-merges-1-seg-finish |       52.07 |   ops/s |
|                                                 Max Throughput |                         wait-until-merges-1-seg-finish |       52.07 |   ops/s |
|                                        50th percentile latency |                         wait-until-merges-1-seg-finish |             |         |
|                                        90th percentile latency |                         wait-until-merges-1-seg-finish |             |         |
|                                        99th percentile latency |                         wait-until-merges-1-seg-finish |             |         |
|                                      99.9th percentile latency |                         wait-until-merges-1-seg-finish |             |         |
|                                     99.99th percentile latency |                         wait-until-merges-1-seg-finish |             |         |
|                                       100th percentile latency |                         wait-until-merges-1-seg-finish |     18.6197 |      ms |
|                                   50th percentile service time |                         wait-until-merges-1-seg-finish |             |         |
|                                   90th percentile service time |                         wait-until-merges-1-seg-finish |             |         |
|                                   99th percentile service time |                         wait-until-merges-1-seg-finish |             |         |
|                                 99.9th percentile service time |                         wait-until-merges-1-seg-finish |             |         |
|                                99.99th percentile service time |                         wait-until-merges-1-seg-finish |             |         |
|                                  100th percentile service time |                         wait-until-merges-1-seg-finish |     18.6197 |      ms |
|                                                     error rate |                         wait-until-merges-1-seg-finish |           0 |       % |
|                                                 Min Throughput |            desc-sort-timestamp-after-force-merge-1-seg |           2 |   ops/s |
|                                                Mean Throughput |            desc-sort-timestamp-after-force-merge-1-seg |           2 |   ops/s |
|                                              Median Throughput |            desc-sort-timestamp-after-force-merge-1-seg |           2 |   ops/s |
|                                                 Max Throughput |            desc-sort-timestamp-after-force-merge-1-seg |           2 |   ops/s |
|                                        50th percentile latency |            desc-sort-timestamp-after-force-merge-1-seg |     51.8613 |      ms |
|                                        90th percentile latency |            desc-sort-timestamp-after-force-merge-1-seg |      67.898 |      ms |
|                                        99th percentile latency |            desc-sort-timestamp-after-force-merge-1-seg |     68.7873 |      ms |
|                                      99.9th percentile latency |            desc-sort-timestamp-after-force-merge-1-seg |             |         |
|                                     99.99th percentile latency |            desc-sort-timestamp-after-force-merge-1-seg |             |         |
|                                       100th percentile latency |            desc-sort-timestamp-after-force-merge-1-seg |     71.3603 |      ms |
|                                   50th percentile service time |            desc-sort-timestamp-after-force-merge-1-seg |     50.7784 |      ms |
|                                   90th percentile service time |            desc-sort-timestamp-after-force-merge-1-seg |     66.7115 |      ms |
|                                   99th percentile service time |            desc-sort-timestamp-after-force-merge-1-seg |     67.1848 |      ms |
|                                 99.9th percentile service time |            desc-sort-timestamp-after-force-merge-1-seg |             |         |
|                                99.99th percentile service time |            desc-sort-timestamp-after-force-merge-1-seg |             |         |
|                                  100th percentile service time |            desc-sort-timestamp-after-force-merge-1-seg |      70.141 |      ms |
|                                                     error rate |            desc-sort-timestamp-after-force-merge-1-seg |           0 |       % |
|                                                 Min Throughput |             asc-sort-timestamp-after-force-merge-1-seg |        2.01 |   ops/s |
|                                                Mean Throughput |             asc-sort-timestamp-after-force-merge-1-seg |        2.01 |   ops/s |
|                                              Median Throughput |             asc-sort-timestamp-after-force-merge-1-seg |        2.01 |   ops/s |
|                                                 Max Throughput |             asc-sort-timestamp-after-force-merge-1-seg |        2.01 |   ops/s |
|                                        50th percentile latency |             asc-sort-timestamp-after-force-merge-1-seg |     10.7656 |      ms |
|                                        90th percentile latency |             asc-sort-timestamp-after-force-merge-1-seg |     12.9686 |      ms |
|                                        99th percentile latency |             asc-sort-timestamp-after-force-merge-1-seg |     16.6664 |      ms |
|                                      99.9th percentile latency |             asc-sort-timestamp-after-force-merge-1-seg |             |         |
|                                     99.99th percentile latency |             asc-sort-timestamp-after-force-merge-1-seg |             |         |
|                                       100th percentile latency |             asc-sort-timestamp-after-force-merge-1-seg |     17.2395 |      ms |
|                                   50th percentile service time |             asc-sort-timestamp-after-force-merge-1-seg |      9.3976 |      ms |
|                                   90th percentile service time |             asc-sort-timestamp-after-force-merge-1-seg |     11.4979 |      ms |
|                                   99th percentile service time |             asc-sort-timestamp-after-force-merge-1-seg |     15.7794 |      ms |
|                                 99.9th percentile service time |             asc-sort-timestamp-after-force-merge-1-seg |             |         |
|                                99.99th percentile service time |             asc-sort-timestamp-after-force-merge-1-seg |             |         |
|                                  100th percentile service time |             asc-sort-timestamp-after-force-merge-1-seg |     16.0006 |      ms |
|                                                     error rate |             asc-sort-timestamp-after-force-merge-1-seg |           0 |       % |
|                                                 Min Throughput | desc-sort-with-after-timestamp-after-force-merge-1-seg |         0.5 |   ops/s |
|                                                Mean Throughput | desc-sort-with-after-timestamp-after-force-merge-1-seg |        0.51 |   ops/s |
|                                              Median Throughput | desc-sort-with-after-timestamp-after-force-merge-1-seg |        0.51 |   ops/s |
|                                                 Max Throughput | desc-sort-with-after-timestamp-after-force-merge-1-seg |        0.55 |   ops/s |
|                                        50th percentile latency | desc-sort-with-after-timestamp-after-force-merge-1-seg |     75.5013 |      ms |
|                                        90th percentile latency | desc-sort-with-after-timestamp-after-force-merge-1-seg |     89.6105 |      ms |
|                                        99th percentile latency | desc-sort-with-after-timestamp-after-force-merge-1-seg |     105.408 |      ms |
|                                      99.9th percentile latency | desc-sort-with-after-timestamp-after-force-merge-1-seg |             |         |
|                                     99.99th percentile latency | desc-sort-with-after-timestamp-after-force-merge-1-seg |             |         |
|                                       100th percentile latency | desc-sort-with-after-timestamp-after-force-merge-1-seg |      113.01 |      ms |
|                                   50th percentile service time | desc-sort-with-after-timestamp-after-force-merge-1-seg |     73.1106 |      ms |
|                                   90th percentile service time | desc-sort-with-after-timestamp-after-force-merge-1-seg |      87.042 |      ms |
|                                   99th percentile service time | desc-sort-with-after-timestamp-after-force-merge-1-seg |     103.848 |      ms |
|                                 99.9th percentile service time | desc-sort-with-after-timestamp-after-force-merge-1-seg |             |         |
|                                99.99th percentile service time | desc-sort-with-after-timestamp-after-force-merge-1-seg |             |         |
|                                  100th percentile service time | desc-sort-with-after-timestamp-after-force-merge-1-seg |     110.167 |      ms |
|                                                     error rate | desc-sort-with-after-timestamp-after-force-merge-1-seg |           0 |       % |
|                                                 Min Throughput |  asc-sort-with-after-timestamp-after-force-merge-1-seg |         0.5 |   ops/s |
|                                                Mean Throughput |  asc-sort-with-after-timestamp-after-force-merge-1-seg |        0.51 |   ops/s |
|                                              Median Throughput |  asc-sort-with-after-timestamp-after-force-merge-1-seg |        0.51 |   ops/s |
|                                                 Max Throughput |  asc-sort-with-after-timestamp-after-force-merge-1-seg |        0.55 |   ops/s |
|                                        50th percentile latency |  asc-sort-with-after-timestamp-after-force-merge-1-seg |     66.4225 |      ms |
|                                        90th percentile latency |  asc-sort-with-after-timestamp-after-force-merge-1-seg |     86.6332 |      ms |
|                                        99th percentile latency |  asc-sort-with-after-timestamp-after-force-merge-1-seg |      88.748 |      ms |
|                                      99.9th percentile latency |  asc-sort-with-after-timestamp-after-force-merge-1-seg |             |         |
|                                     99.99th percentile latency |  asc-sort-with-after-timestamp-after-force-merge-1-seg |             |         |
|                                       100th percentile latency |  asc-sort-with-after-timestamp-after-force-merge-1-seg |     105.402 |      ms |
|                                   50th percentile service time |  asc-sort-with-after-timestamp-after-force-merge-1-seg |     63.6731 |      ms |
|                                   90th percentile service time |  asc-sort-with-after-timestamp-after-force-merge-1-seg |      83.861 |      ms |
|                                   99th percentile service time |  asc-sort-with-after-timestamp-after-force-merge-1-seg |     85.8659 |      ms |
|                                 99.9th percentile service time |  asc-sort-with-after-timestamp-after-force-merge-1-seg |             |         |
|                                99.99th percentile service time |  asc-sort-with-after-timestamp-after-force-merge-1-seg |             |         |
|                                  100th percentile service time |  asc-sort-with-after-timestamp-after-force-merge-1-seg |     102.788 |      ms |
|                                                     error rate |  asc-sort-with-after-timestamp-after-force-merge-1-seg |           0 |       % |


----------------------------------
[INFO] SUCCESS (took 7586 seconds)
----------------------------------

http_logs,remote,bic-24,ri-10s,bri-650ms (Run 2)

|                                                         Metric |                                                   Task |       Value |    Unit |
|---------------------------------------------------------------:|-------------------------------------------------------:|------------:|--------:|
|                     Cumulative indexing time of primary shards |                                                        |     253.029 |     min |
|             Min cumulative indexing time across primary shards |                                                        |           0 |     min |
|          Median cumulative indexing time across primary shards |                                                        |     1.72462 |     min |
|             Max cumulative indexing time across primary shards |                                                        |     37.9658 |     min |
|            Cumulative indexing throttle time of primary shards |                                                        |           0 |     min |
|    Min cumulative indexing throttle time across primary shards |                                                        |           0 |     min |
| Median cumulative indexing throttle time across primary shards |                                                        |           0 |     min |
|    Max cumulative indexing throttle time across primary shards |                                                        |           0 |     min |
|                        Cumulative merge time of primary shards |                                                        |     90.3634 |     min |
|                       Cumulative merge count of primary shards |                                                        |         312 |         |
|                Min cumulative merge time across primary shards |                                                        |           0 |     min |
|             Median cumulative merge time across primary shards |                                                        |    0.336083 |     min |
|                Max cumulative merge time across primary shards |                                                        |     14.3837 |     min |
|               Cumulative merge throttle time of primary shards |                                                        |     33.4595 |     min |
|       Min cumulative merge throttle time across primary shards |                                                        |           0 |     min |
|    Median cumulative merge throttle time across primary shards |                                                        |   0.0413667 |     min |
|       Max cumulative merge throttle time across primary shards |                                                        |     5.56843 |     min |
|                      Cumulative refresh time of primary shards |                                                        |     24.9825 |     min |
|                     Cumulative refresh count of primary shards |                                                        |        2062 |         |
|              Min cumulative refresh time across primary shards |                                                        |           0 |     min |
|           Median cumulative refresh time across primary shards |                                                        |    0.186167 |     min |
|              Max cumulative refresh time across primary shards |                                                        |     3.34678 |     min |
|                        Cumulative flush time of primary shards |                                                        |     1.62975 |     min |
|                       Cumulative flush count of primary shards |                                                        |         126 |         |
|                Min cumulative flush time across primary shards |                                                        |     0.00015 |     min |
|             Median cumulative flush time across primary shards |                                                        |  0.00371667 |     min |
|                Max cumulative flush time across primary shards |                                                        |    0.311217 |     min |
|                                        Total Young Gen GC time |                                                        |      35.567 |       s |
|                                       Total Young Gen GC count |                                                        |         992 |         |
|                                          Total Old Gen GC time |                                                        |           0 |       s |
|                                         Total Old Gen GC count |                                                        |           0 |         |
|                                                     Store size |                                                        |     37.4547 |      GB |
|                                                  Translog size |                                                        | 4.91738e-06 |      GB |
|                                         Heap used for segments |                                                        |           0 |      MB |
|                                       Heap used for doc values |                                                        |           0 |      MB |
|                                            Heap used for terms |                                                        |           0 |      MB |
|                                            Heap used for norms |                                                        |           0 |      MB |
|                                           Heap used for points |                                                        |           0 |      MB |
|                                    Heap used for stored fields |                                                        |           0 |      MB |
|                                                  Segment count |                                                        |          42 |         |
|                                                 Min Throughput |                                           index-append |       66840 |  docs/s |
|                                                Mean Throughput |                                           index-append |     78207.1 |  docs/s |
|                                              Median Throughput |                                           index-append |     79566.2 |  docs/s |
|                                                 Max Throughput |                                           index-append |     81848.9 |  docs/s |
|                                        50th percentile latency |                                           index-append |     1422.92 |      ms |
|                                        90th percentile latency |                                           index-append |     2104.93 |      ms |
|                                        99th percentile latency |                                           index-append |     2918.75 |      ms |
|                                      99.9th percentile latency |                                           index-append |     3445.95 |      ms |
|                                     99.99th percentile latency |                                           index-append |     4194.41 |      ms |
|                                       100th percentile latency |                                           index-append |      4580.5 |      ms |
|                                   50th percentile service time |                                           index-append |     1422.92 |      ms |
|                                   90th percentile service time |                                           index-append |     2104.93 |      ms |
|                                   99th percentile service time |                                           index-append |     2918.75 |      ms |
|                                 99.9th percentile service time |                                           index-append |     3445.95 |      ms |
|                                99.99th percentile service time |                                           index-append |     4194.41 |      ms |
|                                  100th percentile service time |                                           index-append |      4580.5 |      ms |
|                                                     error rate |                                           index-append |        0.03 |       % |
|                                                 Min Throughput |                               wait-until-merges-finish |        0.21 |   ops/s |
|                                                Mean Throughput |                               wait-until-merges-finish |        0.21 |   ops/s |
|                                              Median Throughput |                               wait-until-merges-finish |        0.21 |   ops/s |
|                                                 Max Throughput |                               wait-until-merges-finish |        0.21 |   ops/s |
|                                        50th percentile latency |                               wait-until-merges-finish |             |         |
|                                        90th percentile latency |                               wait-until-merges-finish |             |         |
|                                        99th percentile latency |                               wait-until-merges-finish |             |         |
|                                      99.9th percentile latency |                               wait-until-merges-finish |             |         |
|                                     99.99th percentile latency |                               wait-until-merges-finish |             |         |
|                                       100th percentile latency |                               wait-until-merges-finish |     4752.64 |      ms |
|                                   50th percentile service time |                               wait-until-merges-finish |             |         |
|                                   90th percentile service time |                               wait-until-merges-finish |             |         |
|                                   99th percentile service time |                               wait-until-merges-finish |             |         |
|                                 99.9th percentile service time |                               wait-until-merges-finish |             |         |
|                                99.99th percentile service time |                               wait-until-merges-finish |             |         |
|                                  100th percentile service time |                               wait-until-merges-finish |     4752.64 |      ms |
|                                                     error rate |                               wait-until-merges-finish |           0 |       % |
|                                                 Min Throughput |                                                default |        7.99 |   ops/s |
|                                                Mean Throughput |                                                default |        7.99 |   ops/s |
|                                              Median Throughput |                                                default |        7.99 |   ops/s |
|                                                 Max Throughput |                                                default |        7.99 |   ops/s |
|                                        50th percentile latency |                                                default |     7.24466 |      ms |
|                                        90th percentile latency |                                                default |     8.68874 |      ms |
|                                        99th percentile latency |                                                default |     10.3467 |      ms |
|                                      99.9th percentile latency |                                                default |             |         |
|                                     99.99th percentile latency |                                                default |             |         |
|                                       100th percentile latency |                                                default |     12.3058 |      ms |
|                                   50th percentile service time |                                                default |      5.5013 |      ms |
|                                   90th percentile service time |                                                default |     7.12103 |      ms |
|                                   99th percentile service time |                                                default |     8.77897 |      ms |
|                                 99.9th percentile service time |                                                default |             |         |
|                                99.99th percentile service time |                                                default |             |         |
|                                  100th percentile service time |                                                default |     10.0805 |      ms |
|                                                     error rate |                                                default |           0 |       % |
|                                                 Min Throughput |                                                   term |        49.4 |   ops/s |
|                                                Mean Throughput |                                                   term |       49.43 |   ops/s |
|                                              Median Throughput |                                                   term |       49.43 |   ops/s |
|                                                 Max Throughput |                                                   term |       49.46 |   ops/s |
|                                        50th percentile latency |                                                   term |     9.42077 |      ms |
|                                        90th percentile latency |                                                   term |      11.531 |      ms |
|                                        99th percentile latency |                                                   term |     15.8405 |      ms |
|                                      99.9th percentile latency |                                                   term |             |         |
|                                     99.99th percentile latency |                                                   term |             |         |
|                                       100th percentile latency |                                                   term |      16.876 |      ms |
|                                   50th percentile service time |                                                   term |      8.5068 |      ms |
|                                   90th percentile service time |                                                   term |     10.5351 |      ms |
|                                   99th percentile service time |                                                   term |     14.6379 |      ms |
|                                 99.9th percentile service time |                                                   term |             |         |
|                                99.99th percentile service time |                                                   term |             |         |
|                                  100th percentile service time |                                                   term |     15.8331 |      ms |
|                                                     error rate |                                                   term |           0 |       % |
|                                                 Min Throughput |                                                  range |           1 |   ops/s |
|                                                Mean Throughput |                                                  range |        1.01 |   ops/s |
|                                              Median Throughput |                                                  range |        1.01 |   ops/s |
|                                                 Max Throughput |                                                  range |        1.01 |   ops/s |
|                                        50th percentile latency |                                                  range |     13.7066 |      ms |
|                                        90th percentile latency |                                                  range |     16.1937 |      ms |
|                                        99th percentile latency |                                                  range |     19.3377 |      ms |
|                                      99.9th percentile latency |                                                  range |             |         |
|                                     99.99th percentile latency |                                                  range |             |         |
|                                       100th percentile latency |                                                  range |     20.4043 |      ms |
|                                   50th percentile service time |                                                  range |     11.9024 |      ms |
|                                   90th percentile service time |                                                  range |      14.559 |      ms |
|                                   99th percentile service time |                                                  range |     17.4504 |      ms |
|                                 99.9th percentile service time |                                                  range |             |         |
|                                99.99th percentile service time |                                                  range |             |         |
|                                  100th percentile service time |                                                  range |     18.5184 |      ms |
|                                                     error rate |                                                  range |           0 |       % |
|                                                 Min Throughput |                                          200s-in-range |       32.81 |   ops/s |
|                                                Mean Throughput |                                          200s-in-range |       32.82 |   ops/s |
|                                              Median Throughput |                                          200s-in-range |       32.82 |   ops/s |
|                                                 Max Throughput |                                          200s-in-range |       32.83 |   ops/s |
|                                        50th percentile latency |                                          200s-in-range |     8.82019 |      ms |
|                                        90th percentile latency |                                          200s-in-range |     12.5052 |      ms |
|                                        99th percentile latency |                                          200s-in-range |     15.0411 |      ms |
|                                      99.9th percentile latency |                                          200s-in-range |             |         |
|                                     99.99th percentile latency |                                          200s-in-range |             |         |
|                                       100th percentile latency |                                          200s-in-range |     15.5218 |      ms |
|                                   50th percentile service time |                                          200s-in-range |     7.83755 |      ms |
|                                   90th percentile service time |                                          200s-in-range |     11.1852 |      ms |
|                                   99th percentile service time |                                          200s-in-range |     14.2989 |      ms |
|                                 99.9th percentile service time |                                          200s-in-range |             |         |
|                                99.99th percentile service time |                                          200s-in-range |             |         |
|                                  100th percentile service time |                                          200s-in-range |      15.207 |      ms |
|                                                     error rate |                                          200s-in-range |           0 |       % |
|                                                 Min Throughput |                                          400s-in-range |       49.95 |   ops/s |
|                                                Mean Throughput |                                          400s-in-range |       49.96 |   ops/s |
|                                              Median Throughput |                                          400s-in-range |       49.96 |   ops/s |
|                                                 Max Throughput |                                          400s-in-range |       49.96 |   ops/s |
|                                        50th percentile latency |                                          400s-in-range |     6.44957 |      ms |
|                                        90th percentile latency |                                          400s-in-range |     7.65983 |      ms |
|                                        99th percentile latency |                                          400s-in-range |     9.19059 |      ms |
|                                      99.9th percentile latency |                                          400s-in-range |             |         |
|                                     99.99th percentile latency |                                          400s-in-range |             |         |
|                                       100th percentile latency |                                          400s-in-range |     9.90542 |      ms |
|                                   50th percentile service time |                                          400s-in-range |     5.25072 |      ms |
|                                   90th percentile service time |                                          400s-in-range |     6.48652 |      ms |
|                                   99th percentile service time |                                          400s-in-range |     8.22346 |      ms |
|                                 99.9th percentile service time |                                          400s-in-range |             |         |
|                                99.99th percentile service time |                                          400s-in-range |             |         |
|                                  100th percentile service time |                                          400s-in-range |     8.75973 |      ms |
|                                                     error rate |                                          400s-in-range |           0 |       % |
|                                                 Min Throughput |                                             hourly_agg |         0.2 |   ops/s |
|                                                Mean Throughput |                                             hourly_agg |         0.2 |   ops/s |
|                                              Median Throughput |                                             hourly_agg |         0.2 |   ops/s |
|                                                 Max Throughput |                                             hourly_agg |         0.2 |   ops/s |
|                                        50th percentile latency |                                             hourly_agg |     3509.57 |      ms |
|                                        90th percentile latency |                                             hourly_agg |     4450.17 |      ms |
|                                        99th percentile latency |                                             hourly_agg |     4518.42 |      ms |
|                                      99.9th percentile latency |                                             hourly_agg |             |         |
|                                     99.99th percentile latency |                                             hourly_agg |             |         |
|                                       100th percentile latency |                                             hourly_agg |     4559.32 |      ms |
|                                   50th percentile service time |                                             hourly_agg |     3507.06 |      ms |
|                                   90th percentile service time |                                             hourly_agg |     4448.16 |      ms |
|                                   99th percentile service time |                                             hourly_agg |     4515.82 |      ms |
|                                 99.9th percentile service time |                                             hourly_agg |             |         |
|                                99.99th percentile service time |                                             hourly_agg |             |         |
|                                  100th percentile service time |                                             hourly_agg |     4558.32 |      ms |
|                                                     error rate |                                             hourly_agg |           0 |       % |
|                                                 Min Throughput |                                                 scroll |       24.93 | pages/s |
|                                                Mean Throughput |                                                 scroll |       24.96 | pages/s |
|                                              Median Throughput |                                                 scroll |       24.96 | pages/s |
|                                                 Max Throughput |                                                 scroll |       24.98 | pages/s |
|                                        50th percentile latency |                                                 scroll |     559.314 |      ms |
|                                        90th percentile latency |                                                 scroll |     619.081 |      ms |
|                                        99th percentile latency |                                                 scroll |     638.354 |      ms |
|                                      99.9th percentile latency |                                                 scroll |             |         |
|                                     99.99th percentile latency |                                                 scroll |             |         |
|                                       100th percentile latency |                                                 scroll |     667.492 |      ms |
|                                   50th percentile service time |                                                 scroll |     557.775 |      ms |
|                                   90th percentile service time |                                                 scroll |     617.446 |      ms |
|                                   99th percentile service time |                                                 scroll |     636.889 |      ms |
|                                 99.9th percentile service time |                                                 scroll |             |         |
|                                99.99th percentile service time |                                                 scroll |             |         |
|                                  100th percentile service time |                                                 scroll |     666.419 |      ms |
|                                                     error rate |                                                 scroll |           0 |       % |
|                                                 Min Throughput |                                    desc_sort_timestamp |         0.5 |   ops/s |
|                                                Mean Throughput |                                    desc_sort_timestamp |         0.5 |   ops/s |
|                                              Median Throughput |                                    desc_sort_timestamp |         0.5 |   ops/s |
|                                                 Max Throughput |                                    desc_sort_timestamp |         0.5 |   ops/s |
|                                        50th percentile latency |                                    desc_sort_timestamp |     161.967 |      ms |
|                                        90th percentile latency |                                    desc_sort_timestamp |      204.78 |      ms |
|                                        99th percentile latency |                                    desc_sort_timestamp |     231.204 |      ms |
|                                      99.9th percentile latency |                                    desc_sort_timestamp |             |         |
|                                     99.99th percentile latency |                                    desc_sort_timestamp |             |         |
|                                       100th percentile latency |                                    desc_sort_timestamp |     231.245 |      ms |
|                                   50th percentile service time |                                    desc_sort_timestamp |     159.094 |      ms |
|                                   90th percentile service time |                                    desc_sort_timestamp |     201.955 |      ms |
|                                   99th percentile service time |                                    desc_sort_timestamp |     227.752 |      ms |
|                                 99.9th percentile service time |                                    desc_sort_timestamp |             |         |
|                                99.99th percentile service time |                                    desc_sort_timestamp |             |         |
|                                  100th percentile service time |                                    desc_sort_timestamp |      228.07 |      ms |
|                                                     error rate |                                    desc_sort_timestamp |           0 |       % |
|                                                 Min Throughput |                                     asc_sort_timestamp |         0.5 |   ops/s |
|                                                Mean Throughput |                                     asc_sort_timestamp |         0.5 |   ops/s |
|                                              Median Throughput |                                     asc_sort_timestamp |         0.5 |   ops/s |
|                                                 Max Throughput |                                     asc_sort_timestamp |         0.5 |   ops/s |
|                                        50th percentile latency |                                     asc_sort_timestamp |     13.4628 |      ms |
|                                        90th percentile latency |                                     asc_sort_timestamp |     16.2605 |      ms |
|                                        99th percentile latency |                                     asc_sort_timestamp |     18.8523 |      ms |
|                                      99.9th percentile latency |                                     asc_sort_timestamp |             |         |
|                                     99.99th percentile latency |                                     asc_sort_timestamp |             |         |
|                                       100th percentile latency |                                     asc_sort_timestamp |     20.5777 |      ms |
|                                   50th percentile service time |                                     asc_sort_timestamp |     11.3385 |      ms |
|                                   90th percentile service time |                                     asc_sort_timestamp |     13.8649 |      ms |
|                                   99th percentile service time |                                     asc_sort_timestamp |     17.5974 |      ms |
|                                 99.9th percentile service time |                                     asc_sort_timestamp |             |         |
|                                99.99th percentile service time |                                     asc_sort_timestamp |             |         |
|                                  100th percentile service time |                                     asc_sort_timestamp |     17.7414 |      ms |
|                                                     error rate |                                     asc_sort_timestamp |           0 |       % |
|                                                 Min Throughput |                         desc_sort_with_after_timestamp |         0.5 |   ops/s |
|                                                Mean Throughput |                         desc_sort_with_after_timestamp |        0.51 |   ops/s |
|                                              Median Throughput |                         desc_sort_with_after_timestamp |        0.51 |   ops/s |
|                                                 Max Throughput |                         desc_sort_with_after_timestamp |        0.55 |   ops/s |
|                                        50th percentile latency |                         desc_sort_with_after_timestamp |      86.291 |      ms |
|                                        90th percentile latency |                         desc_sort_with_after_timestamp |     111.003 |      ms |
|                                        99th percentile latency |                         desc_sort_with_after_timestamp |     136.398 |      ms |
|                                      99.9th percentile latency |                         desc_sort_with_after_timestamp |             |         |
|                                     99.99th percentile latency |                         desc_sort_with_after_timestamp |             |         |
|                                       100th percentile latency |                         desc_sort_with_after_timestamp |     136.749 |      ms |
|                                   50th percentile service time |                         desc_sort_with_after_timestamp |     83.8102 |      ms |
|                                   90th percentile service time |                         desc_sort_with_after_timestamp |     107.626 |      ms |
|                                   99th percentile service time |                         desc_sort_with_after_timestamp |     133.658 |      ms |
|                                 99.9th percentile service time |                         desc_sort_with_after_timestamp |             |         |
|                                99.99th percentile service time |                         desc_sort_with_after_timestamp |             |         |
|                                  100th percentile service time |                         desc_sort_with_after_timestamp |     134.448 |      ms |
|                                                     error rate |                         desc_sort_with_after_timestamp |           0 |       % |
|                                                 Min Throughput |                          asc_sort_with_after_timestamp |         0.5 |   ops/s |
|                                                Mean Throughput |                          asc_sort_with_after_timestamp |        0.51 |   ops/s |
|                                              Median Throughput |                          asc_sort_with_after_timestamp |        0.51 |   ops/s |
|                                                 Max Throughput |                          asc_sort_with_after_timestamp |        0.55 |   ops/s |
|                                        50th percentile latency |                          asc_sort_with_after_timestamp |     65.4372 |      ms |
|                                        90th percentile latency |                          asc_sort_with_after_timestamp |     83.6314 |      ms |
|                                        99th percentile latency |                          asc_sort_with_after_timestamp |     97.1459 |      ms |
|                                      99.9th percentile latency |                          asc_sort_with_after_timestamp |             |         |
|                                     99.99th percentile latency |                          asc_sort_with_after_timestamp |             |         |
|                                       100th percentile latency |                          asc_sort_with_after_timestamp |     111.679 |      ms |
|                                   50th percentile service time |                          asc_sort_with_after_timestamp |     62.8007 |      ms |
|                                   90th percentile service time |                          asc_sort_with_after_timestamp |      80.695 |      ms |
|                                   99th percentile service time |                          asc_sort_with_after_timestamp |     94.5173 |      ms |
|                                 99.9th percentile service time |                          asc_sort_with_after_timestamp |             |         |
|                                99.99th percentile service time |                          asc_sort_with_after_timestamp |             |         |
|                                  100th percentile service time |                          asc_sort_with_after_timestamp |     109.836 |      ms |
|                                                     error rate |                          asc_sort_with_after_timestamp |           0 |       % |
|                                                 Min Throughput |                         wait-until-merges-1-seg-finish |        46.3 |   ops/s |
|                                                Mean Throughput |                         wait-until-merges-1-seg-finish |        46.3 |   ops/s |
|                                              Median Throughput |                         wait-until-merges-1-seg-finish |        46.3 |   ops/s |
|                                                 Max Throughput |                         wait-until-merges-1-seg-finish |        46.3 |   ops/s |
|                                        50th percentile latency |                         wait-until-merges-1-seg-finish |             |         |
|                                        90th percentile latency |                         wait-until-merges-1-seg-finish |             |         |
|                                        99th percentile latency |                         wait-until-merges-1-seg-finish |             |         |
|                                      99.9th percentile latency |                         wait-until-merges-1-seg-finish |             |         |
|                                     99.99th percentile latency |                         wait-until-merges-1-seg-finish |             |         |
|                                       100th percentile latency |                         wait-until-merges-1-seg-finish |     20.9963 |      ms |
|                                   50th percentile service time |                         wait-until-merges-1-seg-finish |             |         |
|                                   90th percentile service time |                         wait-until-merges-1-seg-finish |             |         |
|                                   99th percentile service time |                         wait-until-merges-1-seg-finish |             |         |
|                                 99.9th percentile service time |                         wait-until-merges-1-seg-finish |             |         |
|                                99.99th percentile service time |                         wait-until-merges-1-seg-finish |             |         |
|                                  100th percentile service time |                         wait-until-merges-1-seg-finish |     20.9963 |      ms |
|                                                     error rate |                         wait-until-merges-1-seg-finish |           0 |       % |
|                                                 Min Throughput |            desc-sort-timestamp-after-force-merge-1-seg |        1.99 |   ops/s |
|                                                Mean Throughput |            desc-sort-timestamp-after-force-merge-1-seg |        1.99 |   ops/s |
|                                              Median Throughput |            desc-sort-timestamp-after-force-merge-1-seg |        1.99 |   ops/s |
|                                                 Max Throughput |            desc-sort-timestamp-after-force-merge-1-seg |        1.99 |   ops/s |
|                                        50th percentile latency |            desc-sort-timestamp-after-force-merge-1-seg |     152.119 |      ms |
|                                        90th percentile latency |            desc-sort-timestamp-after-force-merge-1-seg |     209.974 |      ms |
|                                        99th percentile latency |            desc-sort-timestamp-after-force-merge-1-seg |     213.835 |      ms |
|                                      99.9th percentile latency |            desc-sort-timestamp-after-force-merge-1-seg |             |         |
|                                     99.99th percentile latency |            desc-sort-timestamp-after-force-merge-1-seg |             |         |
|                                       100th percentile latency |            desc-sort-timestamp-after-force-merge-1-seg |     214.403 |      ms |
|                                   50th percentile service time |            desc-sort-timestamp-after-force-merge-1-seg |     150.703 |      ms |
|                                   90th percentile service time |            desc-sort-timestamp-after-force-merge-1-seg |     207.992 |      ms |
|                                   99th percentile service time |            desc-sort-timestamp-after-force-merge-1-seg |     213.048 |      ms |
|                                 99.9th percentile service time |            desc-sort-timestamp-after-force-merge-1-seg |             |         |
|                                99.99th percentile service time |            desc-sort-timestamp-after-force-merge-1-seg |             |         |
|                                  100th percentile service time |            desc-sort-timestamp-after-force-merge-1-seg |     213.643 |      ms |
|                                                     error rate |            desc-sort-timestamp-after-force-merge-1-seg |           0 |       % |
|                                                 Min Throughput |             asc-sort-timestamp-after-force-merge-1-seg |        2.01 |   ops/s |
|                                                Mean Throughput |             asc-sort-timestamp-after-force-merge-1-seg |        2.01 |   ops/s |
|                                              Median Throughput |             asc-sort-timestamp-after-force-merge-1-seg |        2.01 |   ops/s |
|                                                 Max Throughput |             asc-sort-timestamp-after-force-merge-1-seg |        2.01 |   ops/s |
|                                        50th percentile latency |             asc-sort-timestamp-after-force-merge-1-seg |     10.6566 |      ms |
|                                        90th percentile latency |             asc-sort-timestamp-after-force-merge-1-seg |     15.4564 |      ms |
|                                        99th percentile latency |             asc-sort-timestamp-after-force-merge-1-seg |     20.5502 |      ms |
|                                      99.9th percentile latency |             asc-sort-timestamp-after-force-merge-1-seg |             |         |
|                                     99.99th percentile latency |             asc-sort-timestamp-after-force-merge-1-seg |             |         |
|                                       100th percentile latency |             asc-sort-timestamp-after-force-merge-1-seg |      22.716 |      ms |
|                                   50th percentile service time |             asc-sort-timestamp-after-force-merge-1-seg |     9.30599 |      ms |
|                                   90th percentile service time |             asc-sort-timestamp-after-force-merge-1-seg |     14.0236 |      ms |
|                                   99th percentile service time |             asc-sort-timestamp-after-force-merge-1-seg |     19.3795 |      ms |
|                                 99.9th percentile service time |             asc-sort-timestamp-after-force-merge-1-seg |             |         |
|                                99.99th percentile service time |             asc-sort-timestamp-after-force-merge-1-seg |             |         |
|                                  100th percentile service time |             asc-sort-timestamp-after-force-merge-1-seg |     21.3379 |      ms |
|                                                     error rate |             asc-sort-timestamp-after-force-merge-1-seg |           0 |       % |
|                                                 Min Throughput | desc-sort-with-after-timestamp-after-force-merge-1-seg |         0.5 |   ops/s |
|                                                Mean Throughput | desc-sort-with-after-timestamp-after-force-merge-1-seg |        0.51 |   ops/s |
|                                              Median Throughput | desc-sort-with-after-timestamp-after-force-merge-1-seg |        0.51 |   ops/s |
|                                                 Max Throughput | desc-sort-with-after-timestamp-after-force-merge-1-seg |        0.55 |   ops/s |
|                                        50th percentile latency | desc-sort-with-after-timestamp-after-force-merge-1-seg |     112.291 |      ms |
|                                        90th percentile latency | desc-sort-with-after-timestamp-after-force-merge-1-seg |     135.875 |      ms |
|                                        99th percentile latency | desc-sort-with-after-timestamp-after-force-merge-1-seg |     146.156 |      ms |
|                                      99.9th percentile latency | desc-sort-with-after-timestamp-after-force-merge-1-seg |             |         |
|                                     99.99th percentile latency | desc-sort-with-after-timestamp-after-force-merge-1-seg |             |         |
|                                       100th percentile latency | desc-sort-with-after-timestamp-after-force-merge-1-seg |     154.858 |      ms |
|                                   50th percentile service time | desc-sort-with-after-timestamp-after-force-merge-1-seg |     109.708 |      ms |
|                                   90th percentile service time | desc-sort-with-after-timestamp-after-force-merge-1-seg |     133.451 |      ms |
|                                   99th percentile service time | desc-sort-with-after-timestamp-after-force-merge-1-seg |     143.804 |      ms |
|                                 99.9th percentile service time | desc-sort-with-after-timestamp-after-force-merge-1-seg |             |         |
|                                99.99th percentile service time | desc-sort-with-after-timestamp-after-force-merge-1-seg |             |         |
|                                  100th percentile service time | desc-sort-with-after-timestamp-after-force-merge-1-seg |     151.782 |      ms |
|                                                     error rate | desc-sort-with-after-timestamp-after-force-merge-1-seg |           0 |       % |
|                                                 Min Throughput |  asc-sort-with-after-timestamp-after-force-merge-1-seg |         0.5 |   ops/s |
|                                                Mean Throughput |  asc-sort-with-after-timestamp-after-force-merge-1-seg |        0.51 |   ops/s |
|                                              Median Throughput |  asc-sort-with-after-timestamp-after-force-merge-1-seg |        0.51 |   ops/s |
|                                                 Max Throughput |  asc-sort-with-after-timestamp-after-force-merge-1-seg |        0.54 |   ops/s |
|                                        50th percentile latency |  asc-sort-with-after-timestamp-after-force-merge-1-seg |     121.301 |      ms |
|                                        90th percentile latency |  asc-sort-with-after-timestamp-after-force-merge-1-seg |     170.563 |      ms |
|                                        99th percentile latency |  asc-sort-with-after-timestamp-after-force-merge-1-seg |      173.07 |      ms |
|                                      99.9th percentile latency |  asc-sort-with-after-timestamp-after-force-merge-1-seg |             |         |
|                                     99.99th percentile latency |  asc-sort-with-after-timestamp-after-force-merge-1-seg |             |         |
|                                       100th percentile latency |  asc-sort-with-after-timestamp-after-force-merge-1-seg |     174.864 |      ms |
|                                   50th percentile service time |  asc-sort-with-after-timestamp-after-force-merge-1-seg |     118.575 |      ms |
|                                   90th percentile service time |  asc-sort-with-after-timestamp-after-force-merge-1-seg |     168.367 |      ms |
|                                   99th percentile service time |  asc-sort-with-after-timestamp-after-force-merge-1-seg |     170.682 |      ms |
|                                 99.9th percentile service time |  asc-sort-with-after-timestamp-after-force-merge-1-seg |             |         |
|                                99.99th percentile service time |  asc-sort-with-after-timestamp-after-force-merge-1-seg |             |         |
|                                  100th percentile service time |  asc-sort-with-after-timestamp-after-force-merge-1-seg |     171.801 |      ms |
|                                                     error rate |  asc-sort-with-after-timestamp-after-force-merge-1-seg |           0 |       % |

[WARNING] Error rate is 0.03 for operation 'index-append'. Please check the logs.

----------------------------------
[INFO] SUCCESS (took 7686 seconds)
----------------------------------
@ashking94 ashking94 added enhancement Enhancement or improvement to existing feature or request untriaged labels Sep 6, 2023
@ashking94 ashking94 changed the title Remote store performance run for different workloads, cluster configurations and test configuration Remote store performance run for different workloads, cluster configurations and test configurations Sep 6, 2023
@ashking94 ashking94 added discuss Issues intended to help drive brainstorming and decision making Storage:Durability Issues and PRs related to the durability framework v2.10.0 and removed untriaged labels Sep 6, 2023
@sachinpkale sachinpkale mentioned this issue Sep 6, 2023
Closed
@ashking94
Copy link
Member Author

ashking94 commented Sep 6, 2023 •
edited
Loading

nyc_taxis,docrep,bic-24 (Run 1)

|                                                         Metric |                     Task |       Value |   Unit |
|---------------------------------------------------------------:|-------------------------:|------------:|-------:|
|                     Cumulative indexing time of primary shards |                          |     188.269 |    min |
|             Min cumulative indexing time across primary shards |                          |     30.6438 |    min |
|          Median cumulative indexing time across primary shards |                          |     31.2289 |    min |
|             Max cumulative indexing time across primary shards |                          |     32.4311 |    min |
|            Cumulative indexing throttle time of primary shards |                          |           0 |    min |
|    Min cumulative indexing throttle time across primary shards |                          |           0 |    min |
| Median cumulative indexing throttle time across primary shards |                          |           0 |    min |
|    Max cumulative indexing throttle time across primary shards |                          |           0 |    min |
|                        Cumulative merge time of primary shards |                          |     87.3212 |    min |
|                       Cumulative merge count of primary shards |                          |         124 |        |
|                Min cumulative merge time across primary shards |                          |     11.0788 |    min |
|             Median cumulative merge time across primary shards |                          |     11.6272 |    min |
|                Max cumulative merge time across primary shards |                          |     21.9857 |    min |
|               Cumulative merge throttle time of primary shards |                          |     30.8318 |    min |
|       Min cumulative merge throttle time across primary shards |                          |     2.73943 |    min |
|    Median cumulative merge throttle time across primary shards |                          |     3.17162 |    min |
|       Max cumulative merge throttle time across primary shards |                          |     9.97448 |    min |
|                      Cumulative refresh time of primary shards |                          |     42.0707 |    min |
|                     Cumulative refresh count of primary shards |                          |         708 |        |
|              Min cumulative refresh time across primary shards |                          |     6.72905 |    min |
|           Median cumulative refresh time across primary shards |                          |     6.95863 |    min |
|              Max cumulative refresh time across primary shards |                          |     7.39108 |    min |
|                        Cumulative flush time of primary shards |                          |    0.835433 |    min |
|                       Cumulative flush count of primary shards |                          |          24 |        |
|                Min cumulative flush time across primary shards |                          |   0.0741833 |    min |
|             Median cumulative flush time across primary shards |                          |    0.125967 |    min |
|                Max cumulative flush time across primary shards |                          |     0.27065 |    min |
|                                        Total Young Gen GC time |                          |     232.276 |      s |
|                                       Total Young Gen GC count |                          |        2169 |        |
|                                          Total Old Gen GC time |                          |           0 |      s |
|                                         Total Old Gen GC count |                          |           0 |        |
|                                                     Store size |                          |     44.9925 |     GB |
|                                                  Translog size |                          | 6.14673e-07 |     GB |
|                                         Heap used for segments |                          |           0 |     MB |
|                                       Heap used for doc values |                          |           0 |     MB |
|                                            Heap used for terms |                          |           0 |     MB |
|                                            Heap used for norms |                          |           0 |     MB |
|                                           Heap used for points |                          |           0 |     MB |
|                                    Heap used for stored fields |                          |           0 |     MB |
|                                                  Segment count |                          |         145 |        |
|                                                 Min Throughput |                    index |     34136.3 | docs/s |
|                                                Mean Throughput |                    index |     35023.6 | docs/s |
|                                              Median Throughput |                    index |     34574.6 | docs/s |
|                                                 Max Throughput |                    index |       38480 | docs/s |
|                                        50th percentile latency |                    index |     6578.69 |     ms |
|                                        90th percentile latency |                    index |     9385.54 |     ms |
|                                        99th percentile latency |                    index |     13579.4 |     ms |
|                                      99.9th percentile latency |                    index |     17274.2 |     ms |
|                                     99.99th percentile latency |                    index |     18549.2 |     ms |
|                                       100th percentile latency |                    index |     18704.7 |     ms |
|                                   50th percentile service time |                    index |     6578.69 |     ms |
|                                   90th percentile service time |                    index |     9385.54 |     ms |
|                                   99th percentile service time |                    index |     13579.4 |     ms |
|                                 99.9th percentile service time |                    index |     17274.2 |     ms |
|                                99.99th percentile service time |                    index |     18549.2 |     ms |
|                                  100th percentile service time |                    index |     18704.7 |     ms |
|                                                     error rate |                    index |           0 |      % |
|                                                 Min Throughput | wait-until-merges-finish |           0 |  ops/s |
|                                                Mean Throughput | wait-until-merges-finish |           0 |  ops/s |
|                                              Median Throughput | wait-until-merges-finish |           0 |  ops/s |
|                                                 Max Throughput | wait-until-merges-finish |           0 |  ops/s |
|                                        50th percentile latency | wait-until-merges-finish |             |        |
|                                        90th percentile latency | wait-until-merges-finish |             |        |
|                                        99th percentile latency | wait-until-merges-finish |             |        |
|                                      99.9th percentile latency | wait-until-merges-finish |             |        |
|                                     99.99th percentile latency | wait-until-merges-finish |             |        |
|                                       100th percentile latency | wait-until-merges-finish |      597316 |     ms |
|                                   50th percentile service time | wait-until-merges-finish |             |        |
|                                   90th percentile service time | wait-until-merges-finish |             |        |
|                                   99th percentile service time | wait-until-merges-finish |             |        |
|                                 99.9th percentile service time | wait-until-merges-finish |             |        |
|                                99.99th percentile service time | wait-until-merges-finish |             |        |
|                                  100th percentile service time | wait-until-merges-finish |      597316 |     ms |
|                                                     error rate | wait-until-merges-finish |           0 |      % |
|                                                 Min Throughput |                  default |        2.99 |  ops/s |
|                                                Mean Throughput |                  default |           3 |  ops/s |
|                                              Median Throughput |                  default |           3 |  ops/s |
|                                                 Max Throughput |                  default |           3 |  ops/s |
|                                        50th percentile latency |                  default |     8.66236 |     ms |
|                                        90th percentile latency |                  default |     9.60014 |     ms |
|                                        99th percentile latency |                  default |     11.3552 |     ms |
|                                      99.9th percentile latency |                  default |             |        |
|                                     99.99th percentile latency |                  default |             |        |
|                                       100th percentile latency |                  default |     12.3228 |     ms |
|                                   50th percentile service time |                  default |     7.51809 |     ms |
|                                   90th percentile service time |                  default |     8.51518 |     ms |
|                                   99th percentile service time |                  default |     10.3214 |     ms |
|                                 99.9th percentile service time |                  default |             |        |
|                                99.99th percentile service time |                  default |             |        |
|                                  100th percentile service time |                  default |     11.4025 |     ms |
|                                                     error rate |                  default |           0 |      % |
|                                                 Min Throughput |                    range |         0.7 |  ops/s |
|                                                Mean Throughput |                    range |        0.71 |  ops/s |
|                                              Median Throughput |                    range |        0.71 |  ops/s |
|                                                 Max Throughput |                    range |        0.71 |  ops/s |
|                                        50th percentile latency |                    range |     66.0273 |     ms |
|                                        90th percentile latency |                    range |     72.1584 |     ms |
|                                        99th percentile latency |                    range |     79.8454 |     ms |
|                                      99.9th percentile latency |                    range |             |        |
|                                     99.99th percentile latency |                    range |             |        |
|                                       100th percentile latency |                    range |     82.0371 |     ms |
|                                   50th percentile service time |                    range |     63.8542 |     ms |
|                                   90th percentile service time |                    range |     70.1203 |     ms |
|                                   99th percentile service time |                    range |     77.4869 |     ms |
|                                 99.9th percentile service time |                    range |             |        |
|                                99.99th percentile service time |                    range |             |        |
|                                  100th percentile service time |                    range |     80.7107 |     ms |
|                                                     error rate |                    range |           0 |      % |
|                                                 Min Throughput |      distance_amount_agg |        2.01 |  ops/s |
|                                                Mean Throughput |      distance_amount_agg |        2.01 |  ops/s |
|                                              Median Throughput |      distance_amount_agg |        2.01 |  ops/s |
|                                                 Max Throughput |      distance_amount_agg |        2.03 |  ops/s |
|                                        50th percentile latency |      distance_amount_agg |     5.84951 |     ms |
|                                        90th percentile latency |      distance_amount_agg |     6.88927 |     ms |
|                                        99th percentile latency |      distance_amount_agg |     8.25107 |     ms |
|                                      99.9th percentile latency |      distance_amount_agg |             |        |
|                                     99.99th percentile latency |      distance_amount_agg |             |        |
|                                       100th percentile latency |      distance_amount_agg |     8.27259 |     ms |
|                                   50th percentile service time |      distance_amount_agg |     4.51806 |     ms |
|                                   90th percentile service time |      distance_amount_agg |     5.59804 |     ms |
|                                   99th percentile service time |      distance_amount_agg |     6.87607 |     ms |
|                                 99.9th percentile service time |      distance_amount_agg |             |        |
|                                99.99th percentile service time |      distance_amount_agg |             |        |
|                                  100th percentile service time |      distance_amount_agg |     7.16406 |     ms |
|                                                     error rate |      distance_amount_agg |           0 |      % |
|                                                 Min Throughput |            autohisto_agg |        1.47 |  ops/s |
|                                                Mean Throughput |            autohisto_agg |        1.48 |  ops/s |
|                                              Median Throughput |            autohisto_agg |        1.49 |  ops/s |
|                                                 Max Throughput |            autohisto_agg |        1.49 |  ops/s |
|                                        50th percentile latency |            autohisto_agg |      261.75 |     ms |
|                                        90th percentile latency |            autohisto_agg |     283.603 |     ms |
|                                        99th percentile latency |            autohisto_agg |     309.948 |     ms |
|                                      99.9th percentile latency |            autohisto_agg |             |        |
|                                     99.99th percentile latency |            autohisto_agg |             |        |
|                                       100th percentile latency |            autohisto_agg |     313.159 |     ms |
|                                   50th percentile service time |            autohisto_agg |     260.334 |     ms |
|                                   90th percentile service time |            autohisto_agg |      282.25 |     ms |
|                                   99th percentile service time |            autohisto_agg |      308.45 |     ms |
|                                 99.9th percentile service time |            autohisto_agg |             |        |
|                                99.99th percentile service time |            autohisto_agg |             |        |
|                                  100th percentile service time |            autohisto_agg |     311.594 |     ms |
|                                                     error rate |            autohisto_agg |           0 |      % |
|                                                 Min Throughput |       date_histogram_agg |         1.5 |  ops/s |
|                                                Mean Throughput |       date_histogram_agg |        1.51 |  ops/s |
|                                              Median Throughput |       date_histogram_agg |        1.51 |  ops/s |
|                                                 Max Throughput |       date_histogram_agg |        1.51 |  ops/s |
|                                        50th percentile latency |       date_histogram_agg |     291.134 |     ms |
|                                        90th percentile latency |       date_histogram_agg |     297.517 |     ms |
|                                        99th percentile latency |       date_histogram_agg |     317.197 |     ms |
|                                      99.9th percentile latency |       date_histogram_agg |             |        |
|                                     99.99th percentile latency |       date_histogram_agg |             |        |
|                                       100th percentile latency |       date_histogram_agg |     356.847 |     ms |
|                                   50th percentile service time |       date_histogram_agg |     290.051 |     ms |
|                                   90th percentile service time |       date_histogram_agg |     295.876 |     ms |
|                                   99th percentile service time |       date_histogram_agg |      316.31 |     ms |
|                                 99.9th percentile service time |       date_histogram_agg |             |        |
|                                99.99th percentile service time |       date_histogram_agg |             |        |
|                                  100th percentile service time |       date_histogram_agg |     355.998 |     ms |
|                                                     error rate |       date_histogram_agg |           0 |      % |


----------------------------------
[INFO] SUCCESS (took 6021 seconds)
----------------------------------

nyc_taxis,docrep,bic-24 (Run 2)

|                                                         Metric |                     Task |       Value |   Unit |
|---------------------------------------------------------------:|-------------------------:|------------:|-------:|
|                     Cumulative indexing time of primary shards |                          |     190.485 |    min |
|             Min cumulative indexing time across primary shards |                          |      31.387 |    min |
|          Median cumulative indexing time across primary shards |                          |     31.6078 |    min |
|             Max cumulative indexing time across primary shards |                          |     32.2603 |    min |
|            Cumulative indexing throttle time of primary shards |                          |           0 |    min |
|    Min cumulative indexing throttle time across primary shards |                          |           0 |    min |
| Median cumulative indexing throttle time across primary shards |                          |           0 |    min |
|    Max cumulative indexing throttle time across primary shards |                          |           0 |    min |
|                        Cumulative merge time of primary shards |                          |     76.3307 |    min |
|                       Cumulative merge count of primary shards |                          |         122 |        |
|                Min cumulative merge time across primary shards |                          |     12.2478 |    min |
|             Median cumulative merge time across primary shards |                          |     12.6342 |    min |
|                Max cumulative merge time across primary shards |                          |      13.285 |    min |
|               Cumulative merge throttle time of primary shards |                          |     18.3246 |    min |
|       Min cumulative merge throttle time across primary shards |                          |     3.00018 |    min |
|    Median cumulative merge throttle time across primary shards |                          |     3.06019 |    min |
|       Max cumulative merge throttle time across primary shards |                          |     3.08752 |    min |
|                      Cumulative refresh time of primary shards |                          |     42.4724 |    min |
|                     Cumulative refresh count of primary shards |                          |         690 |        |
|              Min cumulative refresh time across primary shards |                          |     6.88983 |    min |
|           Median cumulative refresh time across primary shards |                          |     7.12233 |    min |
|              Max cumulative refresh time across primary shards |                          |      7.2091 |    min |
|                        Cumulative flush time of primary shards |                          |    0.725667 |    min |
|                       Cumulative flush count of primary shards |                          |          24 |        |
|                Min cumulative flush time across primary shards |                          |     0.07085 |    min |
|             Median cumulative flush time across primary shards |                          |    0.125592 |    min |
|                Max cumulative flush time across primary shards |                          |      0.1621 |    min |
|                                        Total Young Gen GC time |                          |     239.395 |      s |
|                                       Total Young Gen GC count |                          |        2213 |        |
|                                          Total Old Gen GC time |                          |           0 |      s |
|                                         Total Old Gen GC count |                          |           0 |        |
|                                                     Store size |                          |     44.8945 |     GB |
|                                                  Translog size |                          | 6.14673e-07 |     GB |
|                                         Heap used for segments |                          |           0 |     MB |
|                                       Heap used for doc values |                          |           0 |     MB |
|                                            Heap used for terms |                          |           0 |     MB |
|                                            Heap used for norms |                          |           0 |     MB |
|                                           Heap used for points |                          |           0 |     MB |
|                                    Heap used for stored fields |                          |           0 |     MB |                                                                                                                                                                                                          |                                                  Segment count |                          |         172 |        |
|                                                 Min Throughput |                    index |     34527.6 | docs/s |
|                                                Mean Throughput |                    index |     36522.6 | docs/s |
|                                              Median Throughput |                    index |     35788.6 | docs/s |                                                                                                                                                                                                          |                                                 Max Throughput |                    index |     40870.1 | docs/s |
|                                        50th percentile latency |                    index |     6501.54 |     ms |
|                                        90th percentile latency |                    index |     9628.57 |     ms |
|                                        99th percentile latency |                    index |     14054.9 |     ms |
|                                      99.9th percentile latency |                    index |     17452.2 |     ms |
|                                     99.99th percentile latency |                    index |     17983.3 |     ms |
|                                       100th percentile latency |                    index |     18041.7 |     ms |
|                                   50th percentile service time |                    index |     6501.54 |     ms |
|                                   90th percentile service time |                    index |     9628.57 |     ms |
|                                   99th percentile service time |                    index |     14054.9 |     ms |
|                                 99.9th percentile service time |                    index |     17452.2 |     ms |
|                                99.99th percentile service time |                    index |     17983.3 |     ms |
|                                  100th percentile service time |                    index |     18041.7 |     ms |
|                                                     error rate |                    index |           0 |      % |
|                                                 Min Throughput | wait-until-merges-finish |           0 |  ops/s |
|                                                Mean Throughput | wait-until-merges-finish |           0 |  ops/s |
|                                              Median Throughput | wait-until-merges-finish |           0 |  ops/s |
|                                                 Max Throughput | wait-until-merges-finish |           0 |  ops/s |
|                                        50th percentile latency | wait-until-merges-finish |             |        |
|                                        90th percentile latency | wait-until-merges-finish |             |        |
|                                        99th percentile latency | wait-until-merges-finish |             |        |
|                                      99.9th percentile latency | wait-until-merges-finish |             |        |
|                                     99.99th percentile latency | wait-until-merges-finish |             |        |
|                                       100th percentile latency | wait-until-merges-finish |      553498 |     ms |
|                                   50th percentile service time | wait-until-merges-finish |             |        |
|                                   90th percentile service time | wait-until-merges-finish |             |        |
|                                   99th percentile service time | wait-until-merges-finish |             |        |
|                                 99.9th percentile service time | wait-until-merges-finish |             |        |
|                                99.99th percentile service time | wait-until-merges-finish |             |        |
|                                  100th percentile service time | wait-until-merges-finish |      553498 |     ms |
|                                                     error rate | wait-until-merges-finish |           0 |      % |
|                                                 Min Throughput |                  default |        3.02 |  ops/s |
|                                                Mean Throughput |                  default |        3.03 |  ops/s |
|                                              Median Throughput |                  default |        3.03 |  ops/s |
|                                                 Max Throughput |                  default |        3.06 |  ops/s |
|                                        50th percentile latency |                  default |     6.62506 |     ms |
|                                        90th percentile latency |                  default |     7.40712 |     ms |
|                                        99th percentile latency |                  default |     9.21353 |     ms |
|                                      99.9th percentile latency |                  default |             |        |
|                                     99.99th percentile latency |                  default |             |        |
|                                       100th percentile latency |                  default |     9.33771 |     ms |
|                                   50th percentile service time |                  default |     5.35205 |     ms |
|                                   90th percentile service time |                  default |     6.11092 |     ms |
|                                   99th percentile service time |                  default |     7.67397 |     ms |
|                                 99.9th percentile service time |                  default |             |        |
|                                99.99th percentile service time |                  default |             |        |
|                                  100th percentile service time |                  default |     8.08596 |     ms |
|                                                     error rate |                  default |           0 |      % |
|                                                 Min Throughput |                    range |         0.7 |  ops/s |
|                                                Mean Throughput |                    range |        0.71 |  ops/s |
|                                              Median Throughput |                    range |        0.71 |  ops/s |
|                                                 Max Throughput |                    range |        0.71 |  ops/s |
|                                        50th percentile latency |                    range |     100.864 |     ms |
|                                        90th percentile latency |                    range |     106.452 |     ms |
|                                        99th percentile latency |                    range |     119.776 |     ms |
|                                      99.9th percentile latency |                    range |             |        |
|                                     99.99th percentile latency |                    range |             |        |
|                                       100th percentile latency |                    range |     128.894 |     ms |
|                                   50th percentile service time |                    range |      98.808 |     ms |
|                                   90th percentile service time |                    range |     104.423 |     ms |
|                                   99th percentile service time |                    range |     118.069 |     ms |
|                                 99.9th percentile service time |                    range |             |        |
|                                99.99th percentile service time |                    range |             |        |
|                                  100th percentile service time |                    range |     127.147 |     ms |
|                                                     error rate |                    range |           0 |      % |
|                                                 Min Throughput |      distance_amount_agg |           2 |  ops/s |
|                                                Mean Throughput |      distance_amount_agg |        2.01 |  ops/s |
|                                              Median Throughput |      distance_amount_agg |           2 |  ops/s |
|                                                 Max Throughput |      distance_amount_agg |        2.01 |  ops/s |                                                                                                                                                                                                          |                                        50th percentile latency |      distance_amount_agg |     337.559 |     ms |
|                                        90th percentile latency |      distance_amount_agg |       409.4 |     ms |
|                                        99th percentile latency |      distance_amount_agg |     504.112 |     ms |
|                                      99.9th percentile latency |      distance_amount_agg |             |        |                                                                                                                                                                                                          |                                     99.99th percentile latency |      distance_amount_agg |             |        |
|                                       100th percentile latency |      distance_amount_agg |     589.238 |     ms |
|                                   50th percentile service time |      distance_amount_agg |     336.267 |     ms |
|                                   90th percentile service time |      distance_amount_agg |     408.199 |     ms |
|                                   99th percentile service time |      distance_amount_agg |     503.191 |     ms |
|                                 99.9th percentile service time |      distance_amount_agg |             |        |
|                                99.99th percentile service time |      distance_amount_agg |             |        |
|                                  100th percentile service time |      distance_amount_agg |      588.85 |     ms |
|                                                     error rate |      distance_amount_agg |           0 |      % |
|                                                 Min Throughput |            autohisto_agg |        1.49 |  ops/s |
|                                                Mean Throughput |            autohisto_agg |         1.5 |  ops/s |
|                                              Median Throughput |            autohisto_agg |         1.5 |  ops/s |
|                                                 Max Throughput |            autohisto_agg |         1.5 |  ops/s |
|                                        50th percentile latency |            autohisto_agg |     270.465 |     ms |
|                                        90th percentile latency |            autohisto_agg |     294.458 |     ms |
|                                        99th percentile latency |            autohisto_agg |     306.263 |     ms |
|                                      99.9th percentile latency |            autohisto_agg |             |        |
|                                     99.99th percentile latency |            autohisto_agg |             |        |
|                                       100th percentile latency |            autohisto_agg |     309.891 |     ms |
|                                   50th percentile service time |            autohisto_agg |     269.051 |     ms |
|                                   90th percentile service time |            autohisto_agg |     292.976 |     ms |
|                                   99th percentile service time |            autohisto_agg |     305.466 |     ms |
|                                 99.9th percentile service time |            autohisto_agg |             |        |
|                                99.99th percentile service time |            autohisto_agg |             |        |
|                                  100th percentile service time |            autohisto_agg |      308.22 |     ms |
|                                                     error rate |            autohisto_agg |           0 |      % |
|                                                 Min Throughput |       date_histogram_agg |        1.51 |  ops/s |
|                                                Mean Throughput |       date_histogram_agg |        1.51 |  ops/s |
|                                              Median Throughput |       date_histogram_agg |        1.51 |  ops/s |
|                                                 Max Throughput |       date_histogram_agg |        1.52 |  ops/s |
|                                        50th percentile latency |       date_histogram_agg |     281.321 |     ms |
|                                        90th percentile latency |       date_histogram_agg |     299.991 |     ms |
|                                        99th percentile latency |       date_histogram_agg |     308.296 |     ms |
|                                      99.9th percentile latency |       date_histogram_agg |             |        |
|                                     99.99th percentile latency |       date_histogram_agg |             |        |
|                                       100th percentile latency |       date_histogram_agg |     363.391 |     ms |
|                                   50th percentile service time |       date_histogram_agg |     280.148 |     ms |
|                                   90th percentile service time |       date_histogram_agg |      298.71 |     ms |
|                                   99th percentile service time |       date_histogram_agg |     306.676 |     ms |
|                                 99.9th percentile service time |       date_histogram_agg |             |        |
|                                99.99th percentile service time |       date_histogram_agg |             |        |
|                                  100th percentile service time |       date_histogram_agg |     361.968 |     ms |
|                                                     error rate |       date_histogram_agg |           0 |      % |


----------------------------------
[INFO] SUCCESS (took 5941 seconds)
----------------------------------

nyc_taxis,remote,bic-24,bri-650ms (Run 1)

|                                                         Metric |                     Task |       Value |   Unit |
|---------------------------------------------------------------:|-------------------------:|------------:|-------:|
|                     Cumulative indexing time of primary shards |                          |     276.563 |    min |
|             Min cumulative indexing time across primary shards |                          |     42.9478 |    min |
|          Median cumulative indexing time across primary shards |                          |     46.5705 |    min |
|             Max cumulative indexing time across primary shards |                          |     48.8391 |    min |
|            Cumulative indexing throttle time of primary shards |                          |           0 |    min |
|    Min cumulative indexing throttle time across primary shards |                          |           0 |    min |
| Median cumulative indexing throttle time across primary shards |                          |           0 |    min |
|    Max cumulative indexing throttle time across primary shards |                          |           0 |    min |
|                        Cumulative merge time of primary shards |                          |      58.979 |    min |
|                       Cumulative merge count of primary shards |                          |          89 |        |
|                Min cumulative merge time across primary shards |                          |     9.29222 |    min |
|             Median cumulative merge time across primary shards |                          |     9.61835 |    min |
|                Max cumulative merge time across primary shards |                          |     10.8504 |    min |
|               Cumulative merge throttle time of primary shards |                          |     21.5633 |    min |
|       Min cumulative merge throttle time across primary shards |                          |     3.00762 |    min |
|    Median cumulative merge throttle time across primary shards |                          |     3.28845 |    min |
|       Max cumulative merge throttle time across primary shards |                          |     4.69952 |    min |
|                      Cumulative refresh time of primary shards |                          |      30.964 |    min |
|                     Cumulative refresh count of primary shards |                          |         525 |        |
|              Min cumulative refresh time across primary shards |                          |     4.89545 |    min |
|           Median cumulative refresh time across primary shards |                          |     5.21795 |    min |
|              Max cumulative refresh time across primary shards |                          |       5.339 |    min |
|                        Cumulative flush time of primary shards |                          |    0.375967 |    min |
|                       Cumulative flush count of primary shards |                          |          14 |        |
|                Min cumulative flush time across primary shards |                          | 0.000183333 |    min |
|             Median cumulative flush time across primary shards |                          |     0.00395 |    min |
|                Max cumulative flush time across primary shards |                          |    0.194283 |    min |
|                                        Total Young Gen GC time |                          |      54.126 |      s |
|                                       Total Young Gen GC count |                          |         793 |        |
|                                          Total Old Gen GC time |                          |           0 |      s |
|                                         Total Old Gen GC count |                          |           0 |        |
|                                                     Store size |                          |      45.126 |     GB |
|                                                  Translog size |                          | 6.14673e-07 |     GB |
|                                         Heap used for segments |                          |           0 |     MB |
|                                       Heap used for doc values |                          |           0 |     MB |
|                                            Heap used for terms |                          |           0 |     MB |
|                                            Heap used for norms |                          |           0 |     MB |
|                                           Heap used for points |                          |           0 |     MB |
|                                    Heap used for stored fields |                          |           0 |     MB |
|                                                  Segment count |                          |         160 |        |
|                                                 Min Throughput |                    index |     51258.2 | docs/s |
|                                                Mean Throughput |                    index |     53046.1 | docs/s |
|                                              Median Throughput |                    index |     52422.5 | docs/s |
|                                                 Max Throughput |                    index |     58038.7 | docs/s |
|                                        50th percentile latency |                    index |     4491.87 |     ms |
|                                        90th percentile latency |                    index |     5978.72 |     ms |
|                                        99th percentile latency |                    index |     7746.94 |     ms |
|                                      99.9th percentile latency |                    index |     9583.43 |     ms |
|                                     99.99th percentile latency |                    index |     11227.6 |     ms |
|                                       100th percentile latency |                    index |     12505.3 |     ms |
|                                   50th percentile service time |                    index |     4491.87 |     ms |
|                                   90th percentile service time |                    index |     5978.72 |     ms |
|                                   99th percentile service time |                    index |     7746.94 |     ms |
|                                 99.9th percentile service time |                    index |     9583.43 |     ms |
|                                99.99th percentile service time |                    index |     11227.6 |     ms |
|                                  100th percentile service time |                    index |     12505.3 |     ms |
|                                                     error rate |                    index |           0 |      % |
|                                                 Min Throughput | wait-until-merges-finish |        0.02 |  ops/s |
|                                                Mean Throughput | wait-until-merges-finish |        0.02 |  ops/s |
|                                              Median Throughput | wait-until-merges-finish |        0.02 |  ops/s |
|                                                 Max Throughput | wait-until-merges-finish |        0.02 |  ops/s |
|                                        50th percentile latency | wait-until-merges-finish |             |        |
|                                        90th percentile latency | wait-until-merges-finish |             |        |
|                                        99th percentile latency | wait-until-merges-finish |             |        |
|                                      99.9th percentile latency | wait-until-merges-finish |             |        |
|                                     99.99th percentile latency | wait-until-merges-finish |             |        |
|                                       100th percentile latency | wait-until-merges-finish |     64352.6 |     ms |
|                                   50th percentile service time | wait-until-merges-finish |             |        |
|                                   90th percentile service time | wait-until-merges-finish |             |        |
|                                   99th percentile service time | wait-until-merges-finish |             |        |
|                                 99.9th percentile service time | wait-until-merges-finish |             |        |
|                                99.99th percentile service time | wait-until-merges-finish |             |        |
|                                  100th percentile service time | wait-until-merges-finish |     64352.6 |     ms |
|                                                     error rate | wait-until-merges-finish |           0 |      % |
|                                                 Min Throughput |                  default |        3.01 |  ops/s |
|                                                Mean Throughput |                  default |        3.01 |  ops/s |
|                                              Median Throughput |                  default |        3.01 |  ops/s |
|                                                 Max Throughput |                  default |        3.03 |  ops/s |
|                                        50th percentile latency |                  default |       9.052 |     ms |
|                                        90th percentile latency |                  default |     11.6778 |     ms |
|                                        99th percentile latency |                  default |     15.9755 |     ms |
|                                      99.9th percentile latency |                  default |             |        |
|                                     99.99th percentile latency |                  default |             |        |
|                                       100th percentile latency |                  default |     16.2653 |     ms |
|                                   50th percentile service time |                  default |     7.79832 |     ms |
|                                   90th percentile service time |                  default |     10.2434 |     ms |
|                                   99th percentile service time |                  default |     14.7158 |     ms |
|                                 99.9th percentile service time |                  default |             |        |
|                                99.99th percentile service time |                  default |             |        |
|                                  100th percentile service time |                  default |      15.258 |     ms |
|                                                     error rate |                  default |           0 |      % |
|                                                 Min Throughput |                    range |         0.7 |  ops/s |
|                                                Mean Throughput |                    range |         0.7 |  ops/s |
|                                              Median Throughput |                    range |         0.7 |  ops/s |
|                                                 Max Throughput |                    range |        0.71 |  ops/s |
|                                        50th percentile latency |                    range |     61.3812 |     ms |
|                                        90th percentile latency |                    range |     69.5744 |     ms |
|                                        99th percentile latency |                    range |     91.9409 |     ms |
|                                      99.9th percentile latency |                    range |             |        |
|                                     99.99th percentile latency |                    range |             |        |
|                                       100th percentile latency |                    range |     106.867 |     ms |
|                                   50th percentile service time |                    range |      59.737 |     ms |
|                                   90th percentile service time |                    range |     67.1947 |     ms |
|                                   99th percentile service time |                    range |     90.0922 |     ms |
|                                 99.9th percentile service time |                    range |             |        |
|                                99.99th percentile service time |                    range |             |        |
|                                  100th percentile service time |                    range |     104.475 |     ms |
|                                                     error rate |                    range |           0 |      % |
|                                                 Min Throughput |      distance_amount_agg |        2.01 |  ops/s |
|                                                Mean Throughput |      distance_amount_agg |        2.02 |  ops/s |
|                                              Median Throughput |      distance_amount_agg |        2.02 |  ops/s |
|                                                 Max Throughput |      distance_amount_agg |        2.03 |  ops/s |
|                                        50th percentile latency |      distance_amount_agg |     5.53086 |     ms |
|                                        90th percentile latency |      distance_amount_agg |     6.36609 |     ms |
|                                        99th percentile latency |      distance_amount_agg |     7.28813 |     ms |
|                                      99.9th percentile latency |      distance_amount_agg |             |        |
|                                     99.99th percentile latency |      distance_amount_agg |             |        |
|                                       100th percentile latency |      distance_amount_agg |     7.42918 |     ms |
|                                   50th percentile service time |      distance_amount_agg |      4.3222 |     ms |
|                                   90th percentile service time |      distance_amount_agg |     5.13565 |     ms |
|                                   99th percentile service time |      distance_amount_agg |      5.7593 |     ms |
|                                 99.9th percentile service time |      distance_amount_agg |             |        |
|                                99.99th percentile service time |      distance_amount_agg |             |        |
|                                  100th percentile service time |      distance_amount_agg |     5.90737 |     ms |
|                                                     error rate |      distance_amount_agg |           0 |      % |
|                                                 Min Throughput |            autohisto_agg |        1.48 |  ops/s |
|                                                Mean Throughput |            autohisto_agg |        1.49 |  ops/s |
|                                              Median Throughput |            autohisto_agg |        1.49 |  ops/s |
|                                                 Max Throughput |            autohisto_agg |        1.49 |  ops/s |
|                                        50th percentile latency |            autohisto_agg |     271.002 |     ms |
|                                        90th percentile latency |            autohisto_agg |     279.394 |     ms |
|                                        99th percentile latency |            autohisto_agg |     286.017 |     ms |
|                                      99.9th percentile latency |            autohisto_agg |             |        |
|                                     99.99th percentile latency |            autohisto_agg |             |        |
|                                       100th percentile latency |            autohisto_agg |     289.541 |     ms |
|                                   50th percentile service time |            autohisto_agg |     269.829 |     ms |
|                                   90th percentile service time |            autohisto_agg |     278.741 |     ms |
|                                   99th percentile service time |            autohisto_agg |     284.522 |     ms |
|                                 99.9th percentile service time |            autohisto_agg |             |        |
|                                99.99th percentile service time |            autohisto_agg |             |        |
|                                  100th percentile service time |            autohisto_agg |     287.794 |     ms |
|                                                     error rate |            autohisto_agg |           0 |      % |
|                                                 Min Throughput |       date_histogram_agg |         1.5 |  ops/s |
|                                                Mean Throughput |       date_histogram_agg |        1.51 |  ops/s |
|                                              Median Throughput |       date_histogram_agg |        1.51 |  ops/s |
|                                                 Max Throughput |       date_histogram_agg |        1.51 |  ops/s |
|                                        50th percentile latency |       date_histogram_agg |     278.112 |     ms |
|                                        90th percentile latency |       date_histogram_agg |     285.181 |     ms |
|                                        99th percentile latency |       date_histogram_agg |     288.734 |     ms |
|                                      99.9th percentile latency |       date_histogram_agg |             |        |
|                                     99.99th percentile latency |       date_histogram_agg |             |        |
|                                       100th percentile latency |       date_histogram_agg |       290.8 |     ms |
|                                   50th percentile service time |       date_histogram_agg |     277.017 |     ms |
|                                   90th percentile service time |       date_histogram_agg |     283.837 |     ms |
|                                   99th percentile service time |       date_histogram_agg |     287.739 |     ms |
|                                 99.9th percentile service time |       date_histogram_agg |             |        |
|                                99.99th percentile service time |       date_histogram_agg |             |        |
|                                  100th percentile service time |       date_histogram_agg |     289.129 |     ms |
|                                                     error rate |       date_histogram_agg |           0 |      % |

nyc_taxis,remote,bic-24,bri-650ms (Run 2)

|                                                         Metric |                     Task |       Value |   Unit |
|---------------------------------------------------------------:|-------------------------:|------------:|-------:|
|                     Cumulative indexing time of primary shards |                          |      282.99 |    min |
|             Min cumulative indexing time across primary shards |                          |     43.6253 |    min |
|          Median cumulative indexing time across primary shards |                          |     47.9653 |    min |
|             Max cumulative indexing time across primary shards |                          |     50.3314 |    min |
|            Cumulative indexing throttle time of primary shards |                          |           0 |    min |
|    Min cumulative indexing throttle time across primary shards |                          |           0 |    min |
| Median cumulative indexing throttle time across primary shards |                          |           0 |    min |
|    Max cumulative indexing throttle time across primary shards |                          |           0 |    min |
|                        Cumulative merge time of primary shards |                          |     59.2423 |    min |
|                       Cumulative merge count of primary shards |                          |          91 |        |
|                Min cumulative merge time across primary shards |                          |     8.87152 |    min |
|             Median cumulative merge time across primary shards |                          |      10.107 |    min |
|                Max cumulative merge time across primary shards |                          |     10.6029 |    min |
|               Cumulative merge throttle time of primary shards |                          |     19.2004 |    min |
|       Min cumulative merge throttle time across primary shards |                          |     2.69392 |    min |
|    Median cumulative merge throttle time across primary shards |                          |     3.19002 |    min |
|       Max cumulative merge throttle time across primary shards |                          |     3.55828 |    min |
|                      Cumulative refresh time of primary shards |                          |     30.6112 |    min |
|                     Cumulative refresh count of primary shards |                          |         525 |        |
|              Min cumulative refresh time across primary shards |                          |     4.74668 |    min |
|           Median cumulative refresh time across primary shards |                          |     5.17784 |    min |
|              Max cumulative refresh time across primary shards |                          |     5.33273 |    min |
|                        Cumulative flush time of primary shards |                          |      0.4253 |    min |
|                       Cumulative flush count of primary shards |                          |          15 |        |
|                Min cumulative flush time across primary shards |                          |     0.00015 |    min |
|             Median cumulative flush time across primary shards |                          |     0.06515 |    min |
|                Max cumulative flush time across primary shards |                          |    0.150483 |    min |
|                                        Total Young Gen GC time |                          |      53.495 |      s |
|                                       Total Young Gen GC count |                          |         788 |        |
|                                          Total Old Gen GC time |                          |           0 |      s |
|                                         Total Old Gen GC count |                          |           0 |        |
|                                                     Store size |                          |     45.1358 |     GB |
|                                                  Translog size |                          | 6.14673e-07 |     GB |
|                                         Heap used for segments |                          |           0 |     MB |
|                                       Heap used for doc values |                          |           0 |     MB |
|                                            Heap used for terms |                          |           0 |     MB |
|                                            Heap used for norms |                          |           0 |     MB |
|                                           Heap used for points |                          |           0 |     MB |
|                                    Heap used for stored fields |                          |           0 |     MB |
|                                                  Segment count |                          |         176 |        |
|                                                 Min Throughput |                    index |     50527.9 | docs/s |
|                                                Mean Throughput |                    index |     51803.2 | docs/s |
|                                              Median Throughput |                    index |     51571.8 | docs/s |
|                                                 Max Throughput |                    index |     55232.6 | docs/s |
|                                        50th percentile latency |                    index |     4468.81 |     ms |
|                                        90th percentile latency |                    index |     6319.26 |     ms |
|                                        99th percentile latency |                    index |     8607.29 |     ms |
|                                      99.9th percentile latency |                    index |     9971.36 |     ms |
|                                     99.99th percentile latency |                    index |     11782.6 |     ms |
|                                       100th percentile latency |                    index |       12325 |     ms |
|                                   50th percentile service time |                    index |     4468.81 |     ms |
|                                   90th percentile service time |                    index |     6319.26 |     ms |
|                                   99th percentile service time |                    index |     8607.29 |     ms |
|                                 99.9th percentile service time |                    index |     9971.36 |     ms |
|                                99.99th percentile service time |                    index |     11782.6 |     ms |
|                                  100th percentile service time |                    index |       12325 |     ms |
|                                                     error rate |                    index |           0 |      % |
|                                                 Min Throughput | wait-until-merges-finish |        45.5 |  ops/s |
|                                                Mean Throughput | wait-until-merges-finish |        45.5 |  ops/s |
|                                              Median Throughput | wait-until-merges-finish |        45.5 |  ops/s |
|                                                 Max Throughput | wait-until-merges-finish |        45.5 |  ops/s |
|                                        50th percentile latency | wait-until-merges-finish |             |        |
|                                        90th percentile latency | wait-until-merges-finish |             |        |
|                                        99th percentile latency | wait-until-merges-finish |             |        |
|                                      99.9th percentile latency | wait-until-merges-finish |             |        |
|                                     99.99th percentile latency | wait-until-merges-finish |             |        |
|                                       100th percentile latency | wait-until-merges-finish |     21.3812 |     ms |
|                                   50th percentile service time | wait-until-merges-finish |             |        |
|                                   90th percentile service time | wait-until-merges-finish |             |        |
|                                   99th percentile service time | wait-until-merges-finish |             |        |
|                                 99.9th percentile service time | wait-until-merges-finish |             |        |
|                                99.99th percentile service time | wait-until-merges-finish |             |        |
|                                  100th percentile service time | wait-until-merges-finish |     21.3812 |     ms |
|                                                     error rate | wait-until-merges-finish |           0 |      % |
|                                                 Min Throughput |                  default |        3.02 |  ops/s |
|                                                Mean Throughput |                  default |        3.03 |  ops/s |
|                                              Median Throughput |                  default |        3.03 |  ops/s |
|                                                 Max Throughput |                  default |        3.05 |  ops/s |
|                                        50th percentile latency |                  default |     6.20838 |     ms |
|                                        90th percentile latency |                  default |      6.9671 |     ms |
|                                        99th percentile latency |                  default |     9.16464 |     ms |
|                                      99.9th percentile latency |                  default |             |        |
|                                     99.99th percentile latency |                  default |             |        |
|                                       100th percentile latency |                  default |     9.27164 |     ms |
|                                   50th percentile service time |                  default |     5.16427 |     ms |
|                                   90th percentile service time |                  default |     5.89277 |     ms |
|                                   99th percentile service time |                  default |     7.91947 |     ms |
|                                 99.9th percentile service time |                  default |             |        |
|                                99.99th percentile service time |                  default |             |        |
|                                  100th percentile service time |                  default |     7.96544 |     ms |
|                                                     error rate |                  default |           0 |      % |
|                                                 Min Throughput |                    range |         0.7 |  ops/s |
|                                                Mean Throughput |                    range |        0.71 |  ops/s |
|                                              Median Throughput |                    range |        0.71 |  ops/s |
|                                                 Max Throughput |                    range |        0.71 |  ops/s |
|                                        50th percentile latency |                    range |     98.1811 |     ms |
|                                        90th percentile latency |                    range |     109.408 |     ms |
|                                        99th percentile latency |                    range |     112.257 |     ms |
|                                      99.9th percentile latency |                    range |             |        |
|                                     99.99th percentile latency |                    range |             |        |
|                                       100th percentile latency |                    range |     115.531 |     ms |
|                                   50th percentile service time |                    range |      95.964 |     ms |
|                                   90th percentile service time |                    range |     107.312 |     ms |
|                                   99th percentile service time |                    range |     111.411 |     ms |
|                                 99.9th percentile service time |                    range |             |        |
|                                99.99th percentile service time |                    range |             |        |
|                                  100th percentile service time |                    range |     113.422 |     ms |
|                                                     error rate |                    range |           0 |      % |
|                                                 Min Throughput |      distance_amount_agg |        2.01 |  ops/s |
|                                                Mean Throughput |      distance_amount_agg |        2.02 |  ops/s |
|                                              Median Throughput |      distance_amount_agg |        2.02 |  ops/s |
|                                                 Max Throughput |      distance_amount_agg |        2.04 |  ops/s |
|                                        50th percentile latency |      distance_amount_agg |      4.8311 |     ms |
|                                        90th percentile latency |      distance_amount_agg |      5.5215 |     ms |
|                                        99th percentile latency |      distance_amount_agg |     5.89463 |     ms |
|                                      99.9th percentile latency |      distance_amount_agg |             |        |
|                                     99.99th percentile latency |      distance_amount_agg |             |        |
|                                       100th percentile latency |      distance_amount_agg |     6.38617 |     ms |
|                                   50th percentile service time |      distance_amount_agg |     3.53278 |     ms |
|                                   90th percentile service time |      distance_amount_agg |     4.16955 |     ms |
|                                   99th percentile service time |      distance_amount_agg |     4.86623 |     ms |
|                                 99.9th percentile service time |      distance_amount_agg |             |        |
|                                99.99th percentile service time |      distance_amount_agg |             |        |
|                                  100th percentile service time |      distance_amount_agg |     4.96181 |     ms |
|                                                     error rate |      distance_amount_agg |           0 |      % |
|                                                 Min Throughput |            autohisto_agg |         1.5 |  ops/s |
|                                                Mean Throughput |            autohisto_agg |         1.5 |  ops/s |
|                                              Median Throughput |            autohisto_agg |         1.5 |  ops/s |
|                                                 Max Throughput |            autohisto_agg |         1.5 |  ops/s |
|                                        50th percentile latency |            autohisto_agg |     270.136 |     ms |
|                                        90th percentile latency |            autohisto_agg |     279.703 |     ms |
|                                        99th percentile latency |            autohisto_agg |     292.342 |     ms |
|                                      99.9th percentile latency |            autohisto_agg |             |        |
|                                     99.99th percentile latency |            autohisto_agg |             |        |
|                                       100th percentile latency |            autohisto_agg |     293.018 |     ms |
|                                   50th percentile service time |            autohisto_agg |      269.14 |     ms |
|                                   90th percentile service time |            autohisto_agg |     278.351 |     ms |
|                                   99th percentile service time |            autohisto_agg |      290.81 |     ms |
|                                 99.9th percentile service time |            autohisto_agg |             |        |
|                                99.99th percentile service time |            autohisto_agg |             |        |
|                                  100th percentile service time |            autohisto_agg |     291.936 |     ms |
|                                                     error rate |            autohisto_agg |           0 |      % |
|                                                 Min Throughput |       date_histogram_agg |        1.51 |  ops/s |
|                                                Mean Throughput |       date_histogram_agg |        1.51 |  ops/s |
|                                              Median Throughput |       date_histogram_agg |        1.51 |  ops/s |
|                                                 Max Throughput |       date_histogram_agg |        1.52 |  ops/s |
|                                        50th percentile latency |       date_histogram_agg |     275.191 |     ms |
|                                        90th percentile latency |       date_histogram_agg |     279.299 |     ms |
|                                        99th percentile latency |       date_histogram_agg |     286.499 |     ms |
|                                      99.9th percentile latency |       date_histogram_agg |             |        |
|                                     99.99th percentile latency |       date_histogram_agg |             |        |
|                                       100th percentile latency |       date_histogram_agg |     330.028 |     ms |
|                                   50th percentile service time |       date_histogram_agg |     274.169 |     ms |
|                                   90th percentile service time |       date_histogram_agg |     278.161 |     ms |
|                                   99th percentile service time |       date_histogram_agg |      285.23 |     ms |
|                                 99.9th percentile service time |       date_histogram_agg |             |        |
|                                99.99th percentile service time |       date_histogram_agg |             |        |
|                                  100th percentile service time |       date_histogram_agg |     329.405 |     ms |
|                                                     error rate |       date_histogram_agg |           0 |      % |


----------------------------------
[INFO] SUCCESS (took 3879 seconds)
----------------------------------

Configuration 2 - Using r6g.xlarge, 3 shards 1 replica

  • 3 Shards 1 Replica
  • Cluster Configuration - 3 Nodes with data, ingest, master role
  • Instance Details - AWS EC2 r6g.xlarge
  • OSB host - Single AWS EC2 m5.2xlarge

Run Summary

Workload Index type Refresh interval Remote translog Buffer Interval Bulk client / Bulk Size Mean Indexing Throughput p50 Indexing Throughput Indexing Latency in ms (P90) Workload-params
so docrep 1s - Default (4/5K) 29427.8 28992.2 1787.08 {"number_of_shards":"3","number_of_replicas":"1"}
so remote 10s Default (650ms) 8/10K 35103.6 35067.3 3530.24 {"bulk_indexing_clients":"8","bulk_size":"10000","number_of_shards":"3","number_of_replicas":"1","index_settings":{"index":{"refresh_interval":"10s","remote_store":{"translog":{"buffer_interval":"650ms"}}}}}
http_logs docrep 1s - Default (8/5K) 144157 143762 326.216 {"number_of_shards":"3","number_of_replicas":"1"}
http_logs remote 10s 850ms 16/20K 177108 176442 2461.17 {"bulk_indexing_clients":"16","bulk_size":"20000","number_of_shards":"3","number_of_replicas":"1","index_settings":{"index":{"refresh_interval":"10s","remote_store":{"translog":{"buffer_interval":"850ms"}}}}}
nyc_taxis docrep - - Default (8/10K) 91828.7 91287.5 1035.17 {"number_of_shards":"3","number_of_replicas":"1"}
nyc_taxis remote 10s 850ms 16/20K 103620 103635 3912.72 {"bulk_indexing_clients":"16","bulk_size":"20000","number_of_shards":"3","number_of_replicas":"1","index_settings":{"index":{"refresh_interval":"10s","remote_store":{"translog":{"buffer_interval":"850ms"}}}}}

Results for configuration 2

so, docrep (Run 1)

|                                                         Metric |                     Task |       Value |   Unit |
|---------------------------------------------------------------:|-------------------------:|------------:|-------:|
|                     Cumulative indexing time of primary shards |                          |     91.4263 |    min |
|             Min cumulative indexing time across primary shards |                          |     30.3391 |    min |
|          Median cumulative indexing time across primary shards |                          |     30.4993 |    min |
|             Max cumulative indexing time across primary shards |                          |     30.5878 |    min |
|            Cumulative indexing throttle time of primary shards |                          |           0 |    min |
|    Min cumulative indexing throttle time across primary shards |                          |           0 |    min |
| Median cumulative indexing throttle time across primary shards |                          |           0 |    min |
|    Max cumulative indexing throttle time across primary shards |                          |           0 |    min |
|                        Cumulative merge time of primary shards |                          |     75.3811 |    min |
|                       Cumulative merge count of primary shards |                          |          60 |        |
|                Min cumulative merge time across primary shards |                          |     22.5734 |    min |
|             Median cumulative merge time across primary shards |                          |     24.9858 |    min |
|                Max cumulative merge time across primary shards |                          |     27.8219 |    min |
|               Cumulative merge throttle time of primary shards |                          |     42.7349 |    min |
|       Min cumulative merge throttle time across primary shards |                          |     12.6102 |    min |
|    Median cumulative merge throttle time across primary shards |                          |     13.3487 |    min |
|       Max cumulative merge throttle time across primary shards |                          |     16.7759 |    min |
|                      Cumulative refresh time of primary shards |                          |     4.49755 |    min |
|                     Cumulative refresh count of primary shards |                          |         124 |        |
|              Min cumulative refresh time across primary shards |                          |     1.38728 |    min |
|           Median cumulative refresh time across primary shards |                          |     1.42688 |    min |
|              Max cumulative refresh time across primary shards |                          |     1.68338 |    min |
|                        Cumulative flush time of primary shards |                          |     21.4162 |    min |
|                       Cumulative flush count of primary shards |                          |          78 |        |
|                Min cumulative flush time across primary shards |                          |       6.801 |    min |
|             Median cumulative flush time across primary shards |                          |        6.96 |    min |
|                Max cumulative flush time across primary shards |                          |     7.65518 |    min |
|                                        Total Young Gen GC time |                          |       5.027 |      s |
|                                       Total Young Gen GC count |                          |         287 |        |
|                                          Total Old Gen GC time |                          |           0 |      s |
|                                         Total Old Gen GC count |                          |           0 |        |
|                                                     Store size |                          |     61.1943 |     GB |
|                                                  Translog size |                          | 3.07336e-07 |     GB |
|                                         Heap used for segments |                          |           0 |     MB |
|                                       Heap used for doc values |                          |           0 |     MB |
|                                            Heap used for terms |                          |           0 |     MB |
|                                            Heap used for norms |                          |           0 |     MB |
|                                           Heap used for points |                          |           0 |     MB |
|                                    Heap used for stored fields |                          |           0 |     MB |
|                                                  Segment count |                          |          79 |        |
|                                                 Min Throughput |             index-append |     27543.2 | docs/s |
|                                                Mean Throughput |             index-append |     29427.8 | docs/s |
|                                              Median Throughput |             index-append |     28992.2 | docs/s |
|                                                 Max Throughput |             index-append |       34262 | docs/s |
|                                        50th percentile latency |             index-append |      1137.7 |     ms |
|                                        90th percentile latency |             index-append |     1787.08 |     ms |
|                                        99th percentile latency |             index-append |      9302.6 |     ms |
|                                      99.9th percentile latency |             index-append |     13792.6 |     ms |
|                                     99.99th percentile latency |             index-append |             |        |
|                                       100th percentile latency |             index-append |     16878.8 |     ms |
|                                   50th percentile service time |             index-append |      1137.7 |     ms |
|                                   90th percentile service time |             index-append |     1787.08 |     ms |
|                                   99th percentile service time |             index-append |      9302.6 |     ms |
|                                 99.9th percentile service time |             index-append |     13792.6 |     ms |
|                                99.99th percentile service time |             index-append |             |        |
|                                  100th percentile service time |             index-append |     16878.8 |     ms |
|                                                     error rate |             index-append |           0 |      % |
|                                                 Min Throughput | wait-until-merges-finish |           0 |  ops/s |
|                                                Mean Throughput | wait-until-merges-finish |           0 |  ops/s |
|                                              Median Throughput | wait-until-merges-finish |           0 |  ops/s |
|                                                 Max Throughput | wait-until-merges-finish |           0 |  ops/s |
|                                        50th percentile latency | wait-until-merges-finish |             |        |
|                                        90th percentile latency | wait-until-merges-finish |             |        |
|                                        99th percentile latency | wait-until-merges-finish |             |        |
|                                      99.9th percentile latency | wait-until-merges-finish |             |        |
|                                     99.99th percentile latency | wait-until-merges-finish |             |        |
|                                       100th percentile latency | wait-until-merges-finish |      461263 |     ms |
|                                   50th percentile service time | wait-until-merges-finish |             |        |
|                                   90th percentile service time | wait-until-merges-finish |             |        |
|                                   99th percentile service time | wait-until-merges-finish |             |        |
|                                 99.9th percentile service time | wait-until-merges-finish |             |        |
|                                99.99th percentile service time | wait-until-merges-finish |             |        |
|                                  100th percentile service time | wait-until-merges-finish |      461263 |     ms |
|                                                     error rate | wait-until-merges-finish |           0 |      % |

so,remote,bic-8,bs-10k,ri-10s,bri-650ms (Run 1)

|                                                         Metric |                     Task |       Value |   Unit |
|---------------------------------------------------------------:|-------------------------:|------------:|-------:|
|                     Cumulative indexing time of primary shards |                          |     164.616 |    min |
|             Min cumulative indexing time across primary shards |                          |      51.549 |    min |
|          Median cumulative indexing time across primary shards |                          |     52.1491 |    min |
|             Max cumulative indexing time across primary shards |                          |     60.9176 |    min |
|            Cumulative indexing throttle time of primary shards |                          |           0 |    min |
|    Min cumulative indexing throttle time across primary shards |                          |           0 |    min |
| Median cumulative indexing throttle time across primary shards |                          |           0 |    min |
|    Max cumulative indexing throttle time across primary shards |                          |           0 |    min |
|                        Cumulative merge time of primary shards |                          |      77.522 |    min |
|                       Cumulative merge count of primary shards |                          |          92 |        |
|                Min cumulative merge time across primary shards |                          |      24.094 |    min |
|             Median cumulative merge time across primary shards |                          |     24.3741 |    min |
|                Max cumulative merge time across primary shards |                          |     29.0539 |    min |
|               Cumulative merge throttle time of primary shards |                          |      42.778 |    min |
|       Min cumulative merge throttle time across primary shards |                          |     12.5285 |    min |
|    Median cumulative merge throttle time across primary shards |                          |     13.3249 |    min |
|       Max cumulative merge throttle time across primary shards |                          |     16.9246 |    min |
|                      Cumulative refresh time of primary shards |                          |     9.39013 |    min |
|                     Cumulative refresh count of primary shards |                          |         242 |        |
|              Min cumulative refresh time across primary shards |                          |      3.0897 |    min |
|           Median cumulative refresh time across primary shards |                          |     3.10545 |    min |
|              Max cumulative refresh time across primary shards |                          |     3.19498 |    min |
|                        Cumulative flush time of primary shards |                          |     3.85867 |    min |
|                       Cumulative flush count of primary shards |                          |          15 |        |
|                Min cumulative flush time across primary shards |                          |    0.889417 |    min |
|             Median cumulative flush time across primary shards |                          |     1.47823 |    min |
|                Max cumulative flush time across primary shards |                          |     1.49102 |    min |
|                                        Total Young Gen GC time |                          |       2.852 |      s |
|                                       Total Young Gen GC count |                          |         188 |        |
|                                          Total Old Gen GC time |                          |           0 |      s |
|                                         Total Old Gen GC count |                          |           0 |        |
|                                                     Store size |                          |      83.411 |     GB |
|                                                  Translog size |                          | 3.07336e-07 |     GB |
|                                         Heap used for segments |                          |           0 |     MB |
|                                       Heap used for doc values |                          |           0 |     MB |
|                                            Heap used for terms |                          |           0 |     MB |
|                                            Heap used for norms |                          |           0 |     MB |
|                                           Heap used for points |                          |           0 |     MB |
|                                    Heap used for stored fields |                          |           0 |     MB |
|                                                  Segment count |                          |         100 |        |
|                                                 Min Throughput |             index-append |     32952.2 | docs/s |
|                                                Mean Throughput |             index-append |     35103.6 | docs/s |
|                                              Median Throughput |             index-append |     35067.3 | docs/s |
|                                                 Max Throughput |             index-append |     38259.8 | docs/s |
|                                        50th percentile latency |             index-append |     2051.12 |     ms |
|                                        90th percentile latency |             index-append |     3530.24 |     ms |
|                                        99th percentile latency |             index-append |     5342.31 |     ms |
|                                      99.9th percentile latency |             index-append |     8397.06 |     ms |
|                                     99.99th percentile latency |             index-append |             |        |
|                                       100th percentile latency |             index-append |     8940.85 |     ms |
|                                   50th percentile service time |             index-append |     2051.12 |     ms |
|                                   90th percentile service time |             index-append |     3530.24 |     ms |
|                                   99th percentile service time |             index-append |     5342.31 |     ms |
|                                 99.9th percentile service time |             index-append |     8397.06 |     ms |
|                                99.99th percentile service time |             index-append |             |        |
|                                  100th percentile service time |             index-append |     8940.85 |     ms |
|                                                     error rate |             index-append |           0 |      % |
|                                                 Min Throughput | wait-until-merges-finish |           0 |  ops/s |
|                                                Mean Throughput | wait-until-merges-finish |           0 |  ops/s |
|                                              Median Throughput | wait-until-merges-finish |           0 |  ops/s |
|                                                 Max Throughput | wait-until-merges-finish |           0 |  ops/s |
|                                        50th percentile latency | wait-until-merges-finish |             |        |
|                                        90th percentile latency | wait-until-merges-finish |             |        |
|                                        99th percentile latency | wait-until-merges-finish |             |        |
|                                      99.9th percentile latency | wait-until-merges-finish |             |        |
|                                     99.99th percentile latency | wait-until-merges-finish |             |        |
|                                       100th percentile latency | wait-until-merges-finish |      278646 |     ms |
|                                   50th percentile service time | wait-until-merges-finish |             |        |
|                                   90th percentile service time | wait-until-merges-finish |             |        |
|                                   99th percentile service time | wait-until-merges-finish |             |        |
|                                 99.9th percentile service time | wait-until-merges-finish |             |        |
|                                99.99th percentile service time | wait-until-merges-finish |             |        |
|                                  100th percentile service time | wait-until-merges-finish |      278646 |     ms |
|                                                     error rate | wait-until-merges-finish |           0 |      % |


----------------------------------
[INFO] SUCCESS (took 1412 seconds)
----------------------------------

http_logs,docrep (Run 1)

|                                                         Metric |                                                   Task |       Value |    Unit |                                                                                                                                                                                                                                  [263/567]
|---------------------------------------------------------------:|-------------------------------------------------------:|------------:|--------:|
|                     Cumulative indexing time of primary shards |                                                        |      120.64 |     min |
|             Min cumulative indexing time across primary shards |                                                        |           0 |     min |
|          Median cumulative indexing time across primary shards |                                                        |     1.77689 |     min |
|             Max cumulative indexing time across primary shards |                                                        |      30.843 |     min |
|            Cumulative indexing throttle time of primary shards |                                                        |           0 |     min |
|    Min cumulative indexing throttle time across primary shards |                                                        |           0 |     min |
| Median cumulative indexing throttle time across primary shards |                                                        |           0 |     min |
|    Max cumulative indexing throttle time across primary shards |                                                        |           0 |     min |
|                        Cumulative merge time of primary shards |                                                        |     32.2736 |     min |
|                       Cumulative merge count of primary shards |                                                        |          79 |         |
|                Min cumulative merge time across primary shards |                                                        |           0 |     min |
|             Median cumulative merge time across primary shards |                                                        |    0.198642 |     min |
|                Max cumulative merge time across primary shards |                                                        |     9.60348 |     min |
|               Cumulative merge throttle time of primary shards |                                                        |     8.49145 |     min |
|       Min cumulative merge throttle time across primary shards |                                                        |           0 |     min |
|    Median cumulative merge throttle time across primary shards |                                                        |           0 |     min |
|       Max cumulative merge throttle time across primary shards |                                                        |     2.94792 |     min |
|                      Cumulative refresh time of primary shards |                                                        |     3.08128 |     min |
|                     Cumulative refresh count of primary shards |                                                        |         420 |         |
|              Min cumulative refresh time across primary shards |                                                        |           0 |     min |
|           Median cumulative refresh time across primary shards |                                                        |   0.0404167 |     min |
|              Max cumulative refresh time across primary shards |                                                        |    0.795233 |     min |
|                        Cumulative flush time of primary shards |                                                        |     16.7829 |     min |
|                       Cumulative flush count of primary shards |                                                        |         121 |         |
|                Min cumulative flush time across primary shards |                                                        |           0 |     min |
|             Median cumulative flush time across primary shards |                                                        |      0.2792 |     min |
|                Max cumulative flush time across primary shards |                                                        |     4.07095 |     min |
|                                        Total Young Gen GC time |                                                        |      16.143 |       s |
|                                       Total Young Gen GC count |                                                        |         812 |         |
|                                          Total Old Gen GC time |                                                        |           0 |       s |
|                                         Total Old Gen GC count |                                                        |           0 |         |
|                                                     Store size |                                                        |     36.2593 |      GB |
|                                                  Translog size |                                                        | 2.45869e-06 |      GB |
|                                         Heap used for segments |                                                        |           0 |      MB |
|                                       Heap used for doc values |                                                        |           0 |      MB |
|                                            Heap used for terms |                                                        |           0 |      MB |
|                                            Heap used for norms |                                                        |           0 |      MB |
|                                           Heap used for points |                                                        |           0 |      MB |
|                                    Heap used for stored fields |                                                        |           0 |      MB |
|                                                  Segment count |                                                        |          21 |         |
|                                                 Min Throughput |                                           index-append |      136224 |  docs/s |
|                                                Mean Throughput |                                           index-append |      144157 |  docs/s |
|                                              Median Throughput |                                           index-append |      143762 |  docs/s |
|                                                 Max Throughput |                                           index-append |      149332 |  docs/s |
|                                        50th percentile latency |                                           index-append |     236.827 |      ms |
|                                        90th percentile latency |                                           index-append |     326.216 |      ms |
|                                        99th percentile latency |                                           index-append |     609.701 |      ms |
|                                      99.9th percentile latency |                                           index-append |     8858.75 |      ms |
|                                     99.99th percentile latency |                                           index-append |     10680.8 |      ms |
|                                       100th percentile latency |                                           index-append |     12001.9 |      ms |
|                                   50th percentile service time |                                           index-append |     236.827 |      ms |
|                                   90th percentile service time |                                           index-append |     326.216 |      ms |
|                                   99th percentile service time |                                           index-append |     609.701 |      ms |
|                                 99.9th percentile service time |                                           index-append |     8858.75 |      ms |
|                                99.99th percentile service time |                                           index-append |     10680.8 |      ms |
|                                  100th percentile service time |                                           index-append |     12001.9 |      ms |
|                                                     error rate |                                           index-append |           0 |       % |
|                                                 Min Throughput |                               wait-until-merges-finish |        0.08 |   ops/s |
|                                                Mean Throughput |                               wait-until-merges-finish |        0.08 |   ops/s |
|                                              Median Throughput |                               wait-until-merges-finish |        0.08 |   ops/s |
|                                                 Max Throughput |                               wait-until-merges-finish |        0.08 |   ops/s |
|                                        50th percentile latency |                               wait-until-merges-finish |             |         |
|                                        90th percentile latency |                               wait-until-merges-finish |             |         |
|                                        99th percentile latency |                               wait-until-merges-finish |             |         |
|                                      99.9th percentile latency |                               wait-until-merges-finish |             |         |
|                                     99.99th percentile latency |                               wait-until-merges-finish |             |         |
|                                       100th percentile latency |                               wait-until-merges-finish |       12757 |      ms |
|                                   50th percentile service time |                               wait-until-merges-finish |             |         |
|                                   90th percentile service time |                               wait-until-merges-finish |             |         |
|                                   99th percentile service time |                               wait-until-merges-finish |             |         |
|                                 99.9th percentile service time |                               wait-until-merges-finish |             |         |
|                                99.99th percentile service time |                               wait-until-merges-finish |             |         |
|                                  100th percentile service time |                               wait-until-merges-finish |       12757 |      ms |
|                                                     error rate |                               wait-until-merges-finish |           0 |       % |
|                                                 Min Throughput |                                                default |           8 |   ops/s |
|                                                Mean Throughput |                                                default |           8 |   ops/s |
|                                              Median Throughput |                                                default |           8 |   ops/s |
|                                                 Max Throughput |                                                default |           8 |   ops/s |
|                                        50th percentile latency |                                                default |     6.56933 |      ms |
|                                        90th percentile latency |                                                default |     7.10947 |      ms |
|                                        99th percentile latency |                                                default |     7.43513 |      ms |
|                                      99.9th percentile latency |                                                default |             |         |
|                                     99.99th percentile latency |                                                default |             |         |
|                                       100th percentile latency |                                                default |     7.77376 |      ms |
|                                   50th percentile service time |                                                default |     4.75668 |      ms |
|                                   90th percentile service time |                                                default |     5.17912 |      ms |
|                                   99th percentile service time |                                                default |     5.49166 |      ms |
|                                 99.9th percentile service time |                                                default |             |         |
|                                99.99th percentile service time |                                                default |             |         |
|                                  100th percentile service time |                                                default |     5.91411 |      ms |
|                                                     error rate |                                                default |           0 |       % |
|                                                 Min Throughput |                                                   term |       49.91 |   ops/s |
|                                                Mean Throughput |                                                   term |       49.91 |   ops/s |
|                                              Median Throughput |                                                   term |       49.91 |   ops/s |
|                                                 Max Throughput |                                                   term |       49.91 |   ops/s |
|                                        50th percentile latency |                                                   term |     5.56272 |      ms |
|                                        90th percentile latency |                                                   term |     6.11233 |      ms |
|                                        99th percentile latency |                                                   term |     6.97837 |      ms |
|                                      99.9th percentile latency |                                                   term |             |         |
|                                     99.99th percentile latency |                                                   term |             |         |
|                                       100th percentile latency |                                                   term |     15.4586 |      ms |
|                                   50th percentile service time |                                                   term |     4.81346 |      ms |
|                                   90th percentile service time |                                                   term |      5.1951 |      ms |
|                                   99th percentile service time |                                                   term |     5.73111 |      ms |
|                                 99.9th percentile service time |                                                   term |             |         |
|                                99.99th percentile service time |                                                   term |             |         |
|                                  100th percentile service time |                                                   term |     14.5167 |      ms |
|                                                     error rate |                                                   term |           0 |       % |
|                                                 Min Throughput |                                                  range |           1 |   ops/s |
|                                                Mean Throughput |                                                  range |        1.01 |   ops/s |
|                                              Median Throughput |                                                  range |        1.01 |   ops/s |
|                                                 Max Throughput |                                                  range |        1.01 |   ops/s |
|                                        50th percentile latency |                                                  range |      8.1586 |      ms |
|                                        90th percentile latency |                                                  range |     8.94153 |      ms |
|                                        99th percentile latency |                                                  range |     9.22804 |      ms |
|                                      99.9th percentile latency |                                                  range |             |         |
|                                     99.99th percentile latency |                                                  range |             |         |
|                                       100th percentile latency |                                                  range |      9.2603 |      ms |
|                                   50th percentile service time |                                                  range |     6.57125 |      ms |
|                                   90th percentile service time |                                                  range |     7.03155 |      ms |
|                                   99th percentile service time |                                                  range |     7.51153 |      ms |
|                                 99.9th percentile service time |                                                  range |             |         |
|                                99.99th percentile service time |                                                  range |             |         |
|                                  100th percentile service time |                                                  range |     7.97348 |      ms |
|                                                     error rate |                                                  range |           0 |       % |
|                                                 Min Throughput |                                          200s-in-range |       32.93 |   ops/s |
|                                                Mean Throughput |                                          200s-in-range |       32.93 |   ops/s |
|                                              Median Throughput |                                          200s-in-range |       32.93 |   ops/s |
|                                                 Max Throughput |                                          200s-in-range |       32.94 |   ops/s |
|                                        50th percentile latency |                                          200s-in-range |     6.23531 |      ms |
|                                        90th percentile latency |                                          200s-in-range |     7.22349 |      ms |
|                                        99th percentile latency |                                          200s-in-range |     8.76527 |      ms |
|                                      99.9th percentile latency |                                          200s-in-range |             |         |
|                                     99.99th percentile latency |                                          200s-in-range |             |         |
|                                       100th percentile latency |                                          200s-in-range |     9.98661 |      ms |
|                                   50th percentile service time |                                          200s-in-range |     5.56843 |      ms |
|                                   90th percentile service time |                                          200s-in-range |     6.20463 |      ms |
|                                   99th percentile service time |                                          200s-in-range |     7.80088 |      ms |
|                                 99.9th percentile service time |                                          200s-in-range |             |         |
|                                99.99th percentile service time |                                          200s-in-range |             |         |
|                                  100th percentile service time |                                          200s-in-range |     8.82232 |      ms |
|                                                     error rate |                                          200s-in-range |           0 |       % |
|                                                 Min Throughput |                                          400s-in-range |       49.99 |   ops/s |
|                                                Mean Throughput |                                          400s-in-range |       49.99 |   ops/s |
|                                              Median Throughput |                                          400s-in-range |       49.99 |   ops/s |
|                                                 Max Throughput |                                          400s-in-range |       49.99 |   ops/s |
|                                        50th percentile latency |                                          400s-in-range |     4.53566 |      ms |
|                                        90th percentile latency |                                          400s-in-range |     4.99374 |      ms |
|                                        99th percentile latency |                                          400s-in-range |     5.87445 |      ms |
|                                      99.9th percentile latency |                                          400s-in-range |             |         |
|                                     99.99th percentile latency |                                          400s-in-range |             |         |
|                                       100th percentile latency |                                          400s-in-range |     5.89556 |      ms |
|                                   50th percentile service time |                                          400s-in-range |      3.8117 |      ms |
|                                   90th percentile service time |                                          400s-in-range |     4.10875 |      ms |
|                                   99th percentile service time |                                          400s-in-range |     5.14016 |      ms |
|                                 99.9th percentile service time |                                          400s-in-range |             |         |
|                                99.99th percentile service time |                                          400s-in-range |             |         |
|                                  100th percentile service time |                                          400s-in-range |     5.33645 |      ms |
|                                                     error rate |                                          400s-in-range |           0 |       % |
|                                                 Min Throughput |                                             hourly_agg |         0.2 |   ops/s |
|                                                Mean Throughput |                                             hourly_agg |         0.2 |   ops/s |
|                                              Median Throughput |                                             hourly_agg |         0.2 |   ops/s |
|                                                 Max Throughput |                                             hourly_agg |         0.2 |   ops/s |
|                                        50th percentile latency |                                             hourly_agg |     3026.43 |      ms |
|                                        90th percentile latency |                                             hourly_agg |     3299.52 |      ms |
|                                        99th percentile latency |                                             hourly_agg |     3373.23 |      ms |
|                                      99.9th percentile latency |                                             hourly_agg |             |         |
|                                     99.99th percentile latency |                                             hourly_agg |             |         |
|                                       100th percentile latency |                                             hourly_agg |     3468.55 |      ms |
|                                   50th percentile service time |                                             hourly_agg |     3023.53 |      ms |
|                                   90th percentile service time |                                             hourly_agg |      3297.1 |      ms |
|                                   99th percentile service time |                                             hourly_agg |     3370.19 |      ms |
|                                 99.9th percentile service time |                                             hourly_agg |             |         |
|                                99.99th percentile service time |                                             hourly_agg |             |         |
|                                  100th percentile service time |                                             hourly_agg |     3467.77 |      ms |
|                                                     error rate |                                             hourly_agg |           0 |       % |
|                                                 Min Throughput |                                                 scroll |       25.02 | pages/s |
|                                                Mean Throughput |                                                 scroll |       25.04 | pages/s |
|                                              Median Throughput |                                                 scroll |       25.03 | pages/s |
|                                                 Max Throughput |                                                 scroll |       25.07 | pages/s |
|                                        50th percentile latency |                                                 scroll |     258.313 |      ms |
|                                        90th percentile latency |                                                 scroll |     264.147 |      ms |
|                                        99th percentile latency |                                                 scroll |     320.643 |      ms |
|                                      99.9th percentile latency |                                                 scroll |             |         |
|                                     99.99th percentile latency |                                                 scroll |             |         |
|                                       100th percentile latency |                                                 scroll |     359.574 |      ms |
|                                   50th percentile service time |                                                 scroll |     256.352 |      ms |
|                                   90th percentile service time |                                                 scroll |     262.009 |      ms |
|                                   99th percentile service time |                                                 scroll |      319.15 |      ms |
|                                 99.9th percentile service time |                                                 scroll |             |         |
|                                99.99th percentile service time |                                                 scroll |             |         |
|                                  100th percentile service time |                                                 scroll |     357.809 |      ms |
|                                                     error rate |                                                 scroll |           0 |       % |
|                                                 Min Throughput |                                    desc_sort_timestamp |         0.5 |   ops/s |
|                                                Mean Throughput |                                    desc_sort_timestamp |         0.5 |   ops/s |
|                                              Median Throughput |                                    desc_sort_timestamp |         0.5 |   ops/s |
|                                                 Max Throughput |                                    desc_sort_timestamp |         0.5 |   ops/s |
|                                        50th percentile latency |                                    desc_sort_timestamp |     34.8899 |      ms |
|                                        90th percentile latency |                                    desc_sort_timestamp |     41.1937 |      ms |
|                                        99th percentile latency |                                    desc_sort_timestamp |     77.4854 |      ms |
|                                      99.9th percentile latency |                                    desc_sort_timestamp |             |         |
|                                     99.99th percentile latency |                                    desc_sort_timestamp |             |         |
|                                       100th percentile latency |                                    desc_sort_timestamp |     83.8124 |      ms |
|                                   50th percentile service time |                                    desc_sort_timestamp |     32.6111 |      ms |
|                                   90th percentile service time |                                    desc_sort_timestamp |     38.7254 |      ms |
|                                   99th percentile service time |                                    desc_sort_timestamp |     74.3274 |      ms |
|                                 99.9th percentile service time |                                    desc_sort_timestamp |             |         |
|                                99.99th percentile service time |                                    desc_sort_timestamp |             |         |
|                                  100th percentile service time |                                    desc_sort_timestamp |     80.7269 |      ms |
|                                                     error rate |                                    desc_sort_timestamp |           0 |       % |
|                                                 Min Throughput |                                     asc_sort_timestamp |         0.5 |   ops/s |
|                                                Mean Throughput |                                     asc_sort_timestamp |         0.5 |   ops/s |
|                                              Median Throughput |                                     asc_sort_timestamp |         0.5 |   ops/s |
|                                                 Max Throughput |                                     asc_sort_timestamp |         0.5 |   ops/s |
|                                        50th percentile latency |                                     asc_sort_timestamp |     8.38397 |      ms |
|                                        90th percentile latency |                                     asc_sort_timestamp |     9.04573 |      ms |
|                                        99th percentile latency |                                     asc_sort_timestamp |     9.47717 |      ms |
|                                      99.9th percentile latency |                                     asc_sort_timestamp |             |         |
|                                     99.99th percentile latency |                                     asc_sort_timestamp |             |         |
|                                       100th percentile latency |                                     asc_sort_timestamp |     10.4871 |      ms |
|                                   50th percentile service time |                                     asc_sort_timestamp |     5.82743 |      ms |
|                                   90th percentile service time |                                     asc_sort_timestamp |     6.18149 |      ms |
|                                   99th percentile service time |                                     asc_sort_timestamp |     7.30916 |      ms |
|                                 99.9th percentile service time |                                     asc_sort_timestamp |             |         |
|                                99.99th percentile service time |                                     asc_sort_timestamp |             |         |
|                                  100th percentile service time |                                     asc_sort_timestamp |     7.63961 |      ms |
|                                                     error rate |                                     asc_sort_timestamp |           0 |       % |
|                                                 Min Throughput |                         desc_sort_with_after_timestamp |         0.5 |   ops/s |
|                                                Mean Throughput |                         desc_sort_with_after_timestamp |        0.51 |   ops/s |
|                                              Median Throughput |                         desc_sort_with_after_timestamp |        0.51 |   ops/s |
|                                                 Max Throughput |                         desc_sort_with_after_timestamp |        0.54 |   ops/s |
|                                        50th percentile latency |                         desc_sort_with_after_timestamp |     100.082 |      ms |
|                                        90th percentile latency |                         desc_sort_with_after_timestamp |     106.696 |      ms |
|                                        99th percentile latency |                         desc_sort_with_after_timestamp |     112.627 |      ms |
|                                      99.9th percentile latency |                         desc_sort_with_after_timestamp |             |         |
|                                     99.99th percentile latency |                         desc_sort_with_after_timestamp |             |         |
|                                       100th percentile latency |                         desc_sort_with_after_timestamp |     113.237 |      ms |
|                                   50th percentile service time |                         desc_sort_with_after_timestamp |     98.4671 |      ms |
|                                   90th percentile service time |                         desc_sort_with_after_timestamp |     104.658 |      ms |
|                                   99th percentile service time |                         desc_sort_with_after_timestamp |     109.929 |      ms |
|                                 99.9th percentile service time |                         desc_sort_with_after_timestamp |             |         |
|                                99.99th percentile service time |                         desc_sort_with_after_timestamp |             |         |
|                                  100th percentile service time |                         desc_sort_with_after_timestamp |     110.778 |      ms |
|                                                     error rate |                         desc_sort_with_after_timestamp |           0 |       % |
|                                                 Min Throughput |                          asc_sort_with_after_timestamp |         0.5 |   ops/s |
|                                                Mean Throughput |                          asc_sort_with_after_timestamp |        0.51 |   ops/s |
|                                              Median Throughput |                          asc_sort_with_after_timestamp |        0.51 |   ops/s |
|                                                 Max Throughput |                          asc_sort_with_after_timestamp |        0.55 |   ops/s |
|                                        50th percentile latency |                          asc_sort_with_after_timestamp |     12.2562 |      ms |
|                                        90th percentile latency |                          asc_sort_with_after_timestamp |     13.9464 |      ms |
|                                        99th percentile latency |                          asc_sort_with_after_timestamp |     14.6686 |      ms |
|                                      99.9th percentile latency |                          asc_sort_with_after_timestamp |             |         |
|                                     99.99th percentile latency |                          asc_sort_with_after_timestamp |             |         |
|                                       100th percentile latency |                          asc_sort_with_after_timestamp |      14.669 |      ms |
|                                   50th percentile service time |                          asc_sort_with_after_timestamp |     10.8539 |      ms |
|                                   90th percentile service time |                          asc_sort_with_after_timestamp |     11.2565 |      ms |
|                                   99th percentile service time |                          asc_sort_with_after_timestamp |     11.7341 |      ms |
|                                 99.9th percentile service time |                          asc_sort_with_after_timestamp |             |         |
|                                99.99th percentile service time |                          asc_sort_with_after_timestamp |             |         |
|                                  100th percentile service time |                          asc_sort_with_after_timestamp |     11.7933 |      ms |
|                                                     error rate |                          asc_sort_with_after_timestamp |           0 |       % |
|                                                 Min Throughput |                         wait-until-merges-1-seg-finish |       72.85 |   ops/s |
|                                                Mean Throughput |                         wait-until-merges-1-seg-finish |       72.85 |   ops/s |
|                                              Median Throughput |                         wait-until-merges-1-seg-finish |       72.85 |   ops/s |
|                                                 Max Throughput |                         wait-until-merges-1-seg-finish |       72.85 |   ops/s |
|                                        50th percentile latency |                         wait-until-merges-1-seg-finish |             |         |
|                                        90th percentile latency |                         wait-until-merges-1-seg-finish |             |         |
|                                        99th percentile latency |                         wait-until-merges-1-seg-finish |             |         |
|                                      99.9th percentile latency |                         wait-until-merges-1-seg-finish |             |         |
|                                     99.99th percentile latency |                         wait-until-merges-1-seg-finish |             |         |
|                                       100th percentile latency |                         wait-until-merges-1-seg-finish |     12.8974 |      ms |
|                                   50th percentile service time |                         wait-until-merges-1-seg-finish |             |         |
|                                   90th percentile service time |                         wait-until-merges-1-seg-finish |             |         |
|                                   99th percentile service time |                         wait-until-merges-1-seg-finish |             |         |
|                                 99.9th percentile service time |                         wait-until-merges-1-seg-finish |             |         |
|                                99.99th percentile service time |                         wait-until-merges-1-seg-finish |             |         |
|                                  100th percentile service time |                         wait-until-merges-1-seg-finish |     12.8974 |      ms |
|                                                     error rate |                         wait-until-merges-1-seg-finish |           0 |       % |
|                                                 Min Throughput |            desc-sort-timestamp-after-force-merge-1-seg |           2 |   ops/s |
|                                                Mean Throughput |            desc-sort-timestamp-after-force-merge-1-seg |           2 |   ops/s |
|                                              Median Throughput |            desc-sort-timestamp-after-force-merge-1-seg |           2 |   ops/s |
|                                                 Max Throughput |            desc-sort-timestamp-after-force-merge-1-seg |           2 |   ops/s |
|                                        50th percentile latency |            desc-sort-timestamp-after-force-merge-1-seg |     262.518 |      ms |
|                                        90th percentile latency |            desc-sort-timestamp-after-force-merge-1-seg |     264.164 |      ms |
|                                        99th percentile latency |            desc-sort-timestamp-after-force-merge-1-seg |     282.532 |      ms |
|                                      99.9th percentile latency |            desc-sort-timestamp-after-force-merge-1-seg |             |         |
|                                     99.99th percentile latency |            desc-sort-timestamp-after-force-merge-1-seg |             |         |
|                                       100th percentile latency |            desc-sort-timestamp-after-force-merge-1-seg |     284.066 |      ms |
|                                   50th percentile service time |            desc-sort-timestamp-after-force-merge-1-seg |     260.604 |      ms |
|                                   90th percentile service time |            desc-sort-timestamp-after-force-merge-1-seg |     261.977 |      ms |
|                                   99th percentile service time |            desc-sort-timestamp-after-force-merge-1-seg |     280.287 |      ms |
|                                 99.9th percentile service time |            desc-sort-timestamp-after-force-merge-1-seg |             |         |
|                                99.99th percentile service time |            desc-sort-timestamp-after-force-merge-1-seg |             |         |
|                                  100th percentile service time |            desc-sort-timestamp-after-force-merge-1-seg |     282.589 |      ms |
|                                                     error rate |            desc-sort-timestamp-after-force-merge-1-seg |           0 |       % |
|                                                 Min Throughput |             asc-sort-timestamp-after-force-merge-1-seg |        2.01 |   ops/s |
|                                                Mean Throughput |             asc-sort-timestamp-after-force-merge-1-seg |        2.01 |   ops/s |
|                                              Median Throughput |             asc-sort-timestamp-after-force-merge-1-seg |        2.01 |   ops/s |
|                                                 Max Throughput |             asc-sort-timestamp-after-force-merge-1-seg |        2.01 |   ops/s |
|                                        50th percentile latency |             asc-sort-timestamp-after-force-merge-1-seg |     14.2991 |      ms |
|                                        90th percentile latency |             asc-sort-timestamp-after-force-merge-1-seg |     15.8313 |      ms |
|                                        99th percentile latency |             asc-sort-timestamp-after-force-merge-1-seg |     16.4465 |      ms |
|                                      99.9th percentile latency |             asc-sort-timestamp-after-force-merge-1-seg |             |         |
|                                     99.99th percentile latency |             asc-sort-timestamp-after-force-merge-1-seg |             |         |
|                                       100th percentile latency |             asc-sort-timestamp-after-force-merge-1-seg |     16.9976 |      ms |
|                                   50th percentile service time |             asc-sort-timestamp-after-force-merge-1-seg |     12.9295 |      ms |
|                                   90th percentile service time |             asc-sort-timestamp-after-force-merge-1-seg |     14.5367 |      ms |
|                                   99th percentile service time |             asc-sort-timestamp-after-force-merge-1-seg |     15.1749 |      ms |
|                                 99.9th percentile service time |             asc-sort-timestamp-after-force-merge-1-seg |             |         |
|                                99.99th percentile service time |             asc-sort-timestamp-after-force-merge-1-seg |             |         |
|                                  100th percentile service time |             asc-sort-timestamp-after-force-merge-1-seg |     15.6512 |      ms |
|                                                     error rate |             asc-sort-timestamp-after-force-merge-1-seg |           0 |       % |
|                                                 Min Throughput | desc-sort-with-after-timestamp-after-force-merge-1-seg |         0.5 |   ops/s |
|                                                Mean Throughput | desc-sort-with-after-timestamp-after-force-merge-1-seg |        0.51 |   ops/s |
|                                              Median Throughput | desc-sort-with-after-timestamp-after-force-merge-1-seg |        0.51 |   ops/s |
|                                                 Max Throughput | desc-sort-with-after-timestamp-after-force-merge-1-seg |        0.55 |   ops/s |
|                                        50th percentile latency | desc-sort-with-after-timestamp-after-force-merge-1-seg |     129.855 |      ms |
|                                        90th percentile latency | desc-sort-with-after-timestamp-after-force-merge-1-seg |     132.698 |      ms |
|                                        99th percentile latency | desc-sort-with-after-timestamp-after-force-merge-1-seg |     140.591 |      ms |
|                                      99.9th percentile latency | desc-sort-with-after-timestamp-after-force-merge-1-seg |             |         |
|                                     99.99th percentile latency | desc-sort-with-after-timestamp-after-force-merge-1-seg |             |         |
|                                       100th percentile latency | desc-sort-with-after-timestamp-after-force-merge-1-seg |     157.212 |      ms |
|                                   50th percentile service time | desc-sort-with-after-timestamp-after-force-merge-1-seg |      128.01 |      ms |
|                                   90th percentile service time | desc-sort-with-after-timestamp-after-force-merge-1-seg |     129.481 |      ms |
|                                   99th percentile service time | desc-sort-with-after-timestamp-after-force-merge-1-seg |      136.79 |      ms |
|                                 99.9th percentile service time | desc-sort-with-after-timestamp-after-force-merge-1-seg |             |         |
|                                99.99th percentile service time | desc-sort-with-after-timestamp-after-force-merge-1-seg |             |         |
|                                  100th percentile service time | desc-sort-with-after-timestamp-after-force-merge-1-seg |     154.015 |      ms |
|                                                     error rate | desc-sort-with-after-timestamp-after-force-merge-1-seg |           0 |       % |
|                                                 Min Throughput |  asc-sort-with-after-timestamp-after-force-merge-1-seg |         0.5 |   ops/s |
|                                                Mean Throughput |  asc-sort-with-after-timestamp-after-force-merge-1-seg |        0.51 |   ops/s |
|                                              Median Throughput |  asc-sort-with-after-timestamp-after-force-merge-1-seg |        0.51 |   ops/s |
|                                                 Max Throughput |  asc-sort-with-after-timestamp-after-force-merge-1-seg |        0.55 |   ops/s |
|                                        50th percentile latency |  asc-sort-with-after-timestamp-after-force-merge-1-seg |     62.5721 |      ms |
|                                        90th percentile latency |  asc-sort-with-after-timestamp-after-force-merge-1-seg |     63.5902 |      ms |
|                                        99th percentile latency |  asc-sort-with-after-timestamp-after-force-merge-1-seg |     66.2104 |      ms |
|                                      99.9th percentile latency |  asc-sort-with-after-timestamp-after-force-merge-1-seg |             |         |
|                                     99.99th percentile latency |  asc-sort-with-after-timestamp-after-force-merge-1-seg |             |         |
|                                       100th percentile latency |  asc-sort-with-after-timestamp-after-force-merge-1-seg |     66.6432 |      ms |
|                                   50th percentile service time |  asc-sort-with-after-timestamp-after-force-merge-1-seg |     59.8306 |      ms |
|                                   90th percentile service time |  asc-sort-with-after-timestamp-after-force-merge-1-seg |     60.7677 |      ms |
|                                   99th percentile service time |  asc-sort-with-after-timestamp-after-force-merge-1-seg |     63.3555 |      ms |
|                                 99.9th percentile service time |  asc-sort-with-after-timestamp-after-force-merge-1-seg |             |         |
|                                99.99th percentile service time |  asc-sort-with-after-timestamp-after-force-merge-1-seg |             |         |
|                                  100th percentile service time |  asc-sort-with-after-timestamp-after-force-merge-1-seg |     63.4679 |      ms |
|                                                     error rate |  asc-sort-with-after-timestamp-after-force-merge-1-seg |           0 |       % |

http_logs,remote,bic-16,bs-20k,ri-10s,bri-850ms (Run 1)

|                                                         Metric |                                                   Task |       Value |    Unit |
|---------------------------------------------------------------:|-------------------------------------------------------:|------------:|--------:|
|                     Cumulative indexing time of primary shards |                                                        |      216.96 |     min |
|             Min cumulative indexing time across primary shards |                                                        |           0 |     min |
|          Median cumulative indexing time across primary shards |                                                        |      3.0571 |     min |
|             Max cumulative indexing time across primary shards |                                                        |     54.5695 |     min |
|            Cumulative indexing throttle time of primary shards |                                                        |           0 |     min |
|    Min cumulative indexing throttle time across primary shards |                                                        |           0 |     min |
| Median cumulative indexing throttle time across primary shards |                                                        |           0 |     min |
|    Max cumulative indexing throttle time across primary shards |                                                        |           0 |     min |
|                        Cumulative merge time of primary shards |                                                        |     84.1194 |     min |
|                       Cumulative merge count of primary shards |                                                        |         137 |         |
|                Min cumulative merge time across primary shards |                                                        |           0 |     min |
|             Median cumulative merge time across primary shards |                                                        |    0.436908 |     min |
|                Max cumulative merge time across primary shards |                                                        |     28.0157 |     min |
|               Cumulative merge throttle time of primary shards |                                                        |     45.5946 |     min |
|       Min cumulative merge throttle time across primary shards |                                                        |           0 |     min |
|    Median cumulative merge throttle time across primary shards |                                                        |     0.08385 |     min |
|       Max cumulative merge throttle time across primary shards |                                                        |     16.7378 |     min |
|                      Cumulative refresh time of primary shards |                                                        |     9.37072 |     min |
|                     Cumulative refresh count of primary shards |                                                        |         658 |         |
|              Min cumulative refresh time across primary shards |                                                        |           0 |     min |
|           Median cumulative refresh time across primary shards |                                                        |    0.147792 |     min |
|              Max cumulative refresh time across primary shards |                                                        |     2.26175 |     min |
|                        Cumulative flush time of primary shards |                                                        |     1.00122 |     min |
|                       Cumulative flush count of primary shards |                                                        |          54 |         |
|                Min cumulative flush time across primary shards |                                                        |      0.0002 |     min |
|             Median cumulative flush time across primary shards |                                                        |    0.003875 |     min |
|                Max cumulative flush time across primary shards |                                                        |    0.324567 |     min |
|                                        Total Young Gen GC time |                                                        |      16.054 |       s |
|                                       Total Young Gen GC count |                                                        |         528 |         |
|                                          Total Old Gen GC time |                                                        |           0 |       s |
|                                         Total Old Gen GC count |                                                        |           0 |         |
|                                                     Store size |                                                        |     38.8406 |      GB |
|                                                  Translog size |                                                        | 2.45869e-06 |      GB |
|                                         Heap used for segments |                                                        |           0 |      MB |
|                                       Heap used for doc values |                                                        |           0 |      MB |
|                                            Heap used for terms |                                                        |           0 |      MB |
|                                            Heap used for norms |                                                        |           0 |      MB |
|                                           Heap used for points |                                                        |           0 |      MB |
|                                    Heap used for stored fields |                                                        |           0 |      MB |
|                                                  Segment count |                                                        |          21 |         |
|                                                 Min Throughput |                                           index-append |      172370 |  docs/s |
|                                                Mean Throughput |                                           index-append |      177108 |  docs/s |
|                                              Median Throughput |                                           index-append |      176442 |  docs/s |
|                                                 Max Throughput |                                           index-append |      181670 |  docs/s |
|                                        50th percentile latency |                                           index-append |     1682.92 |      ms |
|                                        90th percentile latency |                                           index-append |     2461.17 |      ms |
|                                        99th percentile latency |                                           index-append |     4045.43 |      ms |
|                                      99.9th percentile latency |                                           index-append |     10773.7 |      ms |
|                                     99.99th percentile latency |                                           index-append |     12471.3 |      ms |
|                                       100th percentile latency |                                           index-append |     12476.7 |      ms |
|                                   50th percentile service time |                                           index-append |     1682.92 |      ms |
|                                   90th percentile service time |                                           index-append |     2461.17 |      ms |
|                                   99th percentile service time |                                           index-append |     4045.43 |      ms |
|                                 99.9th percentile service time |                                           index-append |     10773.7 |      ms |
|                                99.99th percentile service time |                                           index-append |     12471.3 |      ms |
|                                  100th percentile service time |                                           index-append |     12476.7 |      ms |
|                                                     error rate |                                           index-append |           0 |       % |
|                                                 Min Throughput |                               wait-until-merges-finish |           0 |   ops/s |
|                                                Mean Throughput |                               wait-until-merges-finish |           0 |   ops/s |
|                                              Median Throughput |                               wait-until-merges-finish |           0 |   ops/s |
|                                                 Max Throughput |                               wait-until-merges-finish |           0 |   ops/s |
|                                        50th percentile latency |                               wait-until-merges-finish |             |         |
|                                        90th percentile latency |                               wait-until-merges-finish |             |         |
|                                        99th percentile latency |                               wait-until-merges-finish |             |         |
|                                      99.9th percentile latency |                               wait-until-merges-finish |             |         |
|                                     99.99th percentile latency |                               wait-until-merges-finish |             |         |
|                                       100th percentile latency |                               wait-until-merges-finish |      363750 |      ms |
|                                   50th percentile service time |                               wait-until-merges-finish |             |         |
|                                   90th percentile service time |                               wait-until-merges-finish |             |         |
|                                   99th percentile service time |                               wait-until-merges-finish |             |         |
|                                 99.9th percentile service time |                               wait-until-merges-finish |             |         |
|                                99.99th percentile service time |                               wait-until-merges-finish |             |         |
|                                  100th percentile service time |                               wait-until-merges-finish |      363750 |      ms |
|                                                     error rate |                               wait-until-merges-finish |           0 |       % |
|                                                 Min Throughput |                                                default |        7.99 |   ops/s |
|                                                Mean Throughput |                                                default |        7.99 |   ops/s |
|                                              Median Throughput |                                                default |        7.99 |   ops/s |
|                                                 Max Throughput |                                                default |        7.99 |   ops/s |
|                                        50th percentile latency |                                                default |      6.5523 |      ms |
|                                        90th percentile latency |                                                default |     7.03914 |      ms |
|                                        99th percentile latency |                                                default |     7.29141 |      ms |
|                                      99.9th percentile latency |                                                default |             |         |
|                                     99.99th percentile latency |                                                default |             |         |
|                                       100th percentile latency |                                                default |     7.68097 |      ms |
|                                   50th percentile service time |                                                default |     4.79482 |      ms |
|                                   90th percentile service time |                                                default |     5.07227 |      ms |
|                                   99th percentile service time |                                                default |     5.43209 |      ms |
|                                 99.9th percentile service time |                                                default |             |         |
|                                99.99th percentile service time |                                                default |             |         |
|                                  100th percentile service time |                                                default |     5.64186 |      ms |
|                                                     error rate |                                                default |           0 |       % |
|                                                 Min Throughput |                                                   term |       49.75 |   ops/s |
|                                                Mean Throughput |                                                   term |       49.77 |   ops/s |
|                                              Median Throughput |                                                   term |       49.77 |   ops/s |
|                                                 Max Throughput |                                                   term |       49.79 |   ops/s |
|                                        50th percentile latency |                                                   term |      5.8317 |      ms |
|                                        90th percentile latency |                                                   term |     6.46945 |      ms |
|                                        99th percentile latency |                                                   term |     6.88348 |      ms |
|                                      99.9th percentile latency |                                                   term |             |         |
|                                     99.99th percentile latency |                                                   term |             |         |
|                                       100th percentile latency |                                                   term |     7.03311 |      ms |
|                                   50th percentile service time |                                                   term |     5.06488 |      ms |
|                                   90th percentile service time |                                                   term |     5.62093 |      ms |
|                                   99th percentile service time |                                                   term |     5.78312 |      ms |
|                                 99.9th percentile service time |                                                   term |             |         |
|                                99.99th percentile service time |                                                   term |             |         |
|                                  100th percentile service time |                                                   term |     5.87254 |      ms |
|                                                     error rate |                                                   term |           0 |       % |
|                                                 Min Throughput |                                                  range |           1 |   ops/s |
|                                                Mean Throughput |                                                  range |        1.01 |   ops/s |
|                                              Median Throughput |                                                  range |        1.01 |   ops/s |
|                                                 Max Throughput |                                                  range |        1.01 |   ops/s |
|                                        50th percentile latency |                                                  range |     10.3218 |      ms |
|                                        90th percentile latency |                                                  range |     11.5803 |      ms |
|                                        99th percentile latency |                                                  range |     12.2706 |      ms |
|                                      99.9th percentile latency |                                                  range |             |         |
|                                     99.99th percentile latency |                                                  range |             |         |
|                                       100th percentile latency |                                                  range |      12.711 |      ms |
|                                   50th percentile service time |                                                  range |     8.52718 |      ms |
|                                   90th percentile service time |                                                  range |     9.78191 |      ms |
|                                   99th percentile service time |                                                  range |      10.402 |      ms |
|                                 99.9th percentile service time |                                                  range |             |         |
|                                99.99th percentile service time |                                                  range |             |         |
|                                  100th percentile service time |                                                  range |     10.5836 |      ms |
|                                                     error rate |                                                  range |           0 |       % |
|                                                 Min Throughput |                                          200s-in-range |       32.89 |   ops/s |
|                                                Mean Throughput |                                          200s-in-range |       32.89 |   ops/s |
|                                              Median Throughput |                                          200s-in-range |       32.89 |   ops/s |
|                                                 Max Throughput |                                          200s-in-range |        32.9 |   ops/s |
|                                        50th percentile latency |                                          200s-in-range |     6.13901 |      ms |
|                                        90th percentile latency |                                          200s-in-range |     6.63123 |      ms |
|                                        99th percentile latency |                                          200s-in-range |     7.04891 |      ms |
|                                      99.9th percentile latency |                                          200s-in-range |             |         |
|                                     99.99th percentile latency |                                          200s-in-range |             |         |
|                                       100th percentile latency |                                          200s-in-range |      7.1373 |      ms |
|                                   50th percentile service time |                                          200s-in-range |     5.42197 |      ms |
|                                   90th percentile service time |                                          200s-in-range |     5.77903 |      ms |
|                                   99th percentile service time |                                          200s-in-range |     6.10369 |      ms |
|                                 99.9th percentile service time |                                          200s-in-range |             |         |
|                                99.99th percentile service time |                                          200s-in-range |             |         |
|                                  100th percentile service time |                                          200s-in-range |     6.13984 |      ms |
|                                                     error rate |                                          200s-in-range |           0 |       % |
|                                                 Min Throughput |                                          400s-in-range |       49.95 |   ops/s |
|                                                Mean Throughput |                                          400s-in-range |       49.95 |   ops/s |
|                                              Median Throughput |                                          400s-in-range |       49.95 |   ops/s |
|                                                 Max Throughput |                                          400s-in-range |       49.96 |   ops/s |
|                                        50th percentile latency |                                          400s-in-range |     4.77628 |      ms |
|                                        90th percentile latency |                                          400s-in-range |     5.32273 |      ms |
|                                        99th percentile latency |                                          400s-in-range |     5.95005 |      ms |
|                                      99.9th percentile latency |                                          400s-in-range |             |         |
|                                     99.99th percentile latency |                                          400s-in-range |             |         |
|                                       100th percentile latency |                                          400s-in-range |     6.59634 |      ms |
|                                   50th percentile service time |                                          400s-in-range |     4.03751 |      ms |
|                                   90th percentile service time |                                          400s-in-range |      4.4812 |      ms |
|                                   99th percentile service time |                                          400s-in-range |     4.88231 |      ms |
|                                 99.9th percentile service time |                                          400s-in-range |             |         |
|                                99.99th percentile service time |                                          400s-in-range |             |         |
|                                  100th percentile service time |                                          400s-in-range |     5.61349 |      ms |
|                                                     error rate |                                          400s-in-range |           0 |       % |
|                                                 Min Throughput |                                             hourly_agg |         0.2 |   ops/s |
|                                                Mean Throughput |                                             hourly_agg |         0.2 |   ops/s |
|                                              Median Throughput |                                             hourly_agg |         0.2 |   ops/s |
|                                                 Max Throughput |                                             hourly_agg |         0.2 |   ops/s |
|                                        50th percentile latency |                                             hourly_agg |     3187.34 |      ms |
|                                        90th percentile latency |                                             hourly_agg |     3388.46 |      ms |
|                                        99th percentile latency |                                             hourly_agg |     3468.34 |      ms |
|                                      99.9th percentile latency |                                             hourly_agg |             |         |
|                                     99.99th percentile latency |                                             hourly_agg |             |         |
|                                       100th percentile latency |                                             hourly_agg |      3488.2 |      ms |
|                                   50th percentile service time |                                             hourly_agg |      3184.3 |      ms |
|                                   90th percentile service time |                                             hourly_agg |     3386.16 |      ms |
|                                   99th percentile service time |                                             hourly_agg |     3465.84 |      ms |
|                                 99.9th percentile service time |                                             hourly_agg |             |         |
|                                99.99th percentile service time |                                             hourly_agg |             |         |
|                                  100th percentile service time |                                             hourly_agg |     3485.88 |      ms |
|                                                     error rate |                                             hourly_agg |           0 |       % |
|                                                 Min Throughput |                                                 scroll |       25.02 | pages/s |
|                                                Mean Throughput |                                                 scroll |       25.04 | pages/s |
|                                              Median Throughput |                                                 scroll |       25.03 | pages/s |
|                                                 Max Throughput |                                                 scroll |       25.07 | pages/s |
|                                        50th percentile latency |                                                 scroll |     258.728 |      ms |
|                                        90th percentile latency |                                                 scroll |     266.754 |      ms |
|                                        99th percentile latency |                                                 scroll |     325.276 |      ms |
|                                      99.9th percentile latency |                                                 scroll |             |         |
|                                     99.99th percentile latency |                                                 scroll |             |         |
|                                       100th percentile latency |                                                 scroll |     333.477 |      ms |
|                                   50th percentile service time |                                                 scroll |     256.885 |      ms |
|                                   90th percentile service time |                                                 scroll |     264.451 |      ms |
|                                   99th percentile service time |                                                 scroll |     323.383 |      ms |
|                                 99.9th percentile service time |                                                 scroll |             |         |
|                                99.99th percentile service time |                                                 scroll |             |         |
|                                  100th percentile service time |                                                 scroll |     331.722 |      ms |
|                                                     error rate |                                                 scroll |           0 |       % |
|                                                 Min Throughput |                                    desc_sort_timestamp |         0.5 |   ops/s |
|                                                Mean Throughput |                                    desc_sort_timestamp |         0.5 |   ops/s |
|                                              Median Throughput |                                    desc_sort_timestamp |         0.5 |   ops/s |
|                                                 Max Throughput |                                    desc_sort_timestamp |         0.5 |   ops/s |
|                                        50th percentile latency |                                    desc_sort_timestamp |     151.334 |      ms |
|                                        90th percentile latency |                                    desc_sort_timestamp |     171.743 |      ms |
|                                        99th percentile latency |                                    desc_sort_timestamp |     174.761 |      ms |
|                                      99.9th percentile latency |                                    desc_sort_timestamp |             |         |
|                                     99.99th percentile latency |                                    desc_sort_timestamp |             |         |
|                                       100th percentile latency |                                    desc_sort_timestamp |     179.112 |      ms |
|                                   50th percentile service time |                                    desc_sort_timestamp |     148.751 |      ms |
|                                   90th percentile service time |                                    desc_sort_timestamp |     168.999 |      ms |
|                                   99th percentile service time |                                    desc_sort_timestamp |     172.614 |      ms |
|                                 99.9th percentile service time |                                    desc_sort_timestamp |             |         |
|                                99.99th percentile service time |                                    desc_sort_timestamp |             |         |
|                                  100th percentile service time |                                    desc_sort_timestamp |     177.168 |      ms |
|                                                     error rate |                                    desc_sort_timestamp |           0 |       % |
|                                                 Min Throughput |                                     asc_sort_timestamp |         0.5 |   ops/s |
|                                                Mean Throughput |                                     asc_sort_timestamp |         0.5 |   ops/s |
|                                              Median Throughput |                                     asc_sort_timestamp |         0.5 |   ops/s |
|                                                 Max Throughput |                                     asc_sort_timestamp |         0.5 |   ops/s |
|                                        50th percentile latency |                                     asc_sort_timestamp |     9.71108 |      ms |
|                                        90th percentile latency |                                     asc_sort_timestamp |     11.1134 |      ms |
|                                        99th percentile latency |                                     asc_sort_timestamp |     12.1587 |      ms |
|                                      99.9th percentile latency |                                     asc_sort_timestamp |             |         |
|                                     99.99th percentile latency |                                     asc_sort_timestamp |             |         |
|                                       100th percentile latency |                                     asc_sort_timestamp |     13.3693 |      ms |
|                                   50th percentile service time |                                     asc_sort_timestamp |     7.41601 |      ms |
|                                   90th percentile service time |                                     asc_sort_timestamp |     8.34697 |      ms |
|                                   99th percentile service time |                                     asc_sort_timestamp |     9.42721 |      ms |
|                                 99.9th percentile service time |                                     asc_sort_timestamp |             |         |
|                                99.99th percentile service time |                                     asc_sort_timestamp |             |         |
|                                  100th percentile service time |                                     asc_sort_timestamp |     10.8433 |      ms |
|                                                     error rate |                                     asc_sort_timestamp |           0 |       % |
|                                                 Min Throughput |                         desc_sort_with_after_timestamp |         0.5 |   ops/s |
|                                                Mean Throughput |                         desc_sort_with_after_timestamp |        0.51 |   ops/s |
|                                              Median Throughput |                         desc_sort_with_after_timestamp |        0.51 |   ops/s |
|                                                 Max Throughput |                         desc_sort_with_after_timestamp |        0.55 |   ops/s |
|                                        50th percentile latency |                         desc_sort_with_after_timestamp |     66.7631 |      ms |
|                                        90th percentile latency |                         desc_sort_with_after_timestamp |     84.8835 |      ms |
|                                        99th percentile latency |                         desc_sort_with_after_timestamp |      92.659 |      ms |
|                                      99.9th percentile latency |                         desc_sort_with_after_timestamp |             |         |
|                                     99.99th percentile latency |                         desc_sort_with_after_timestamp |             |         |
|                                       100th percentile latency |                         desc_sort_with_after_timestamp |     108.608 |      ms |
|                                   50th percentile service time |                         desc_sort_with_after_timestamp |     64.0926 |      ms |
|                                   90th percentile service time |                         desc_sort_with_after_timestamp |     82.6206 |      ms |
|                                   99th percentile service time |                         desc_sort_with_after_timestamp |     90.5283 |      ms |
|                                 99.9th percentile service time |                         desc_sort_with_after_timestamp |             |         |
|                                99.99th percentile service time |                         desc_sort_with_after_timestamp |             |         |
|                                  100th percentile service time |                         desc_sort_with_after_timestamp |     105.404 |      ms |
|                                                     error rate |                         desc_sort_with_after_timestamp |           0 |       % |
|                                                 Min Throughput |                          asc_sort_with_after_timestamp |         0.5 |   ops/s |
|                                                Mean Throughput |                          asc_sort_with_after_timestamp |        0.51 |   ops/s |
|                                              Median Throughput |                          asc_sort_with_after_timestamp |        0.51 |   ops/s |
|                                                 Max Throughput |                          asc_sort_with_after_timestamp |        0.55 |   ops/s |
|                                        50th percentile latency |                          asc_sort_with_after_timestamp |     41.3591 |      ms |
|                                        90th percentile latency |                          asc_sort_with_after_timestamp |     42.0737 |      ms |
|                                        99th percentile latency |                          asc_sort_with_after_timestamp |     42.6048 |      ms |
|                                      99.9th percentile latency |                          asc_sort_with_after_timestamp |             |         |
|                                     99.99th percentile latency |                          asc_sort_with_after_timestamp |             |         |
|                                       100th percentile latency |                          asc_sort_with_after_timestamp |     42.7375 |      ms |
|                                   50th percentile service time |                          asc_sort_with_after_timestamp |     38.9095 |      ms |
|                                   90th percentile service time |                          asc_sort_with_after_timestamp |     39.2944 |      ms |
|                                   99th percentile service time |                          asc_sort_with_after_timestamp |     39.7544 |      ms |
|                                 99.9th percentile service time |                          asc_sort_with_after_timestamp |             |         |
|                                99.99th percentile service time |                          asc_sort_with_after_timestamp |             |         |
|                                  100th percentile service time |                          asc_sort_with_after_timestamp |     39.9047 |      ms |
|                                                     error rate |                          asc_sort_with_after_timestamp |           0 |       % |
|                                                 Min Throughput |                         wait-until-merges-1-seg-finish |       57.09 |   ops/s |
|                                                Mean Throughput |                         wait-until-merges-1-seg-finish |       57.09 |   ops/s |
|                                              Median Throughput |                         wait-until-merges-1-seg-finish |       57.09 |   ops/s |
|                                                 Max Throughput |                         wait-until-merges-1-seg-finish |       57.09 |   ops/s |
|                                        50th percentile latency |                         wait-until-merges-1-seg-finish |             |         |
|                                        90th percentile latency |                         wait-until-merges-1-seg-finish |             |         |
|                                        99th percentile latency |                         wait-until-merges-1-seg-finish |             |         |
|                                      99.9th percentile latency |                         wait-until-merges-1-seg-finish |             |         |
|                                     99.99th percentile latency |                         wait-until-merges-1-seg-finish |             |         |
|                                       100th percentile latency |                         wait-until-merges-1-seg-finish |     16.9214 |      ms |
|                                   50th percentile service time |                         wait-until-merges-1-seg-finish |             |         |
|                                   90th percentile service time |                         wait-until-merges-1-seg-finish |             |         |
|                                   99th percentile service time |                         wait-until-merges-1-seg-finish |             |         |
|                                 99.9th percentile service time |                         wait-until-merges-1-seg-finish |             |         |
|                                99.99th percentile service time |                         wait-until-merges-1-seg-finish |             |         |
|                                  100th percentile service time |                         wait-until-merges-1-seg-finish |     16.9214 |      ms |
|                                                     error rate |                         wait-until-merges-1-seg-finish |           0 |       % |
|                                                 Min Throughput |            desc-sort-timestamp-after-force-merge-1-seg |        2.01 |   ops/s |
|                                                Mean Throughput |            desc-sort-timestamp-after-force-merge-1-seg |        2.01 |   ops/s |
|                                              Median Throughput |            desc-sort-timestamp-after-force-merge-1-seg |        2.01 |   ops/s |
|                                                 Max Throughput |            desc-sort-timestamp-after-force-merge-1-seg |        2.01 |   ops/s |
|                                        50th percentile latency |            desc-sort-timestamp-after-force-merge-1-seg |     67.2411 |      ms |
|                                        90th percentile latency |            desc-sort-timestamp-after-force-merge-1-seg |     76.0511 |      ms |
|                                        99th percentile latency |            desc-sort-timestamp-after-force-merge-1-seg |     83.4724 |      ms |
|                                      99.9th percentile latency |            desc-sort-timestamp-after-force-merge-1-seg |             |         |
|                                     99.99th percentile latency |            desc-sort-timestamp-after-force-merge-1-seg |             |         |
|                                       100th percentile latency |            desc-sort-timestamp-after-force-merge-1-seg |     83.4983 |      ms |
|                                   50th percentile service time |            desc-sort-timestamp-after-force-merge-1-seg |     65.9276 |      ms |
|                                   90th percentile service time |            desc-sort-timestamp-after-force-merge-1-seg |     74.9196 |      ms |
|                                   99th percentile service time |            desc-sort-timestamp-after-force-merge-1-seg |       82.38 |      ms |
|                                 99.9th percentile service time |            desc-sort-timestamp-after-force-merge-1-seg |             |         |
|                                99.99th percentile service time |            desc-sort-timestamp-after-force-merge-1-seg |             |         |
|                                  100th percentile service time |            desc-sort-timestamp-after-force-merge-1-seg |     82.4486 |      ms |
|                                                     error rate |            desc-sort-timestamp-after-force-merge-1-seg |           0 |       % |
|                                                 Min Throughput |             asc-sort-timestamp-after-force-merge-1-seg |        2.01 |   ops/s |
|                                                Mean Throughput |             asc-sort-timestamp-after-force-merge-1-seg |        2.01 |   ops/s |
|                                              Median Throughput |             asc-sort-timestamp-after-force-merge-1-seg |        2.01 |   ops/s |
|                                                 Max Throughput |             asc-sort-timestamp-after-force-merge-1-seg |        2.01 |   ops/s |
|                                        50th percentile latency |             asc-sort-timestamp-after-force-merge-1-seg |     8.58301 |      ms |
|                                        90th percentile latency |             asc-sort-timestamp-after-force-merge-1-seg |     9.13727 |      ms |
|                                        99th percentile latency |             asc-sort-timestamp-after-force-merge-1-seg |     9.45005 |      ms |
|                                      99.9th percentile latency |             asc-sort-timestamp-after-force-merge-1-seg |             |         |
|                                     99.99th percentile latency |             asc-sort-timestamp-after-force-merge-1-seg |             |         |
|                                       100th percentile latency |             asc-sort-timestamp-after-force-merge-1-seg |     9.63569 |      ms |
|                                   50th percentile service time |             asc-sort-timestamp-after-force-merge-1-seg |     7.33847 |      ms |
|                                   90th percentile service time |             asc-sort-timestamp-after-force-merge-1-seg |      7.6606 |      ms |
|                                   99th percentile service time |             asc-sort-timestamp-after-force-merge-1-seg |     7.91378 |      ms |
|                                 99.9th percentile service time |             asc-sort-timestamp-after-force-merge-1-seg |             |         |
|                                99.99th percentile service time |             asc-sort-timestamp-after-force-merge-1-seg |             |         |
|                                  100th percentile service time |             asc-sort-timestamp-after-force-merge-1-seg |      7.9294 |      ms |
|                                                     error rate |             asc-sort-timestamp-after-force-merge-1-seg |           0 |       % |
|                                                 Min Throughput | desc-sort-with-after-timestamp-after-force-merge-1-seg |         0.5 |   ops/s |
|                                                Mean Throughput | desc-sort-with-after-timestamp-after-force-merge-1-seg |        0.51 |   ops/s |
|                                              Median Throughput | desc-sort-with-after-timestamp-after-force-merge-1-seg |        0.51 |   ops/s |
|                                                 Max Throughput | desc-sort-with-after-timestamp-after-force-merge-1-seg |        0.55 |   ops/s |
|                                        50th percentile latency | desc-sort-with-after-timestamp-after-force-merge-1-seg |     68.3991 |      ms |
|                                        90th percentile latency | desc-sort-with-after-timestamp-after-force-merge-1-seg |     76.3568 |      ms |
|                                        99th percentile latency | desc-sort-with-after-timestamp-after-force-merge-1-seg |     85.0849 |      ms |
|                                      99.9th percentile latency | desc-sort-with-after-timestamp-after-force-merge-1-seg |             |         |
|                                     99.99th percentile latency | desc-sort-with-after-timestamp-after-force-merge-1-seg |             |         |
|                                       100th percentile latency | desc-sort-with-after-timestamp-after-force-merge-1-seg |     88.6349 |      ms |
|                                   50th percentile service time | desc-sort-with-after-timestamp-after-force-merge-1-seg |      66.014 |      ms |
|                                   90th percentile service time | desc-sort-with-after-timestamp-after-force-merge-1-seg |     74.6418 |      ms |
|                                   99th percentile service time | desc-sort-with-after-timestamp-after-force-merge-1-seg |     83.7384 |      ms |
|                                 99.9th percentile service time | desc-sort-with-after-timestamp-after-force-merge-1-seg |             |         |
|                                99.99th percentile service time | desc-sort-with-after-timestamp-after-force-merge-1-seg |             |         |
|                                  100th percentile service time | desc-sort-with-after-timestamp-after-force-merge-1-seg |     85.4659 |      ms |
|                                                     error rate | desc-sort-with-after-timestamp-after-force-merge-1-seg |           0 |       % |
|                                                 Min Throughput |  asc-sort-with-after-timestamp-after-force-merge-1-seg |         0.5 |   ops/s |
|                                                Mean Throughput |  asc-sort-with-after-timestamp-after-force-merge-1-seg |        0.51 |   ops/s |
|                                              Median Throughput |  asc-sort-with-after-timestamp-after-force-merge-1-seg |        0.51 |   ops/s |
|                                                 Max Throughput |  asc-sort-with-after-timestamp-after-force-merge-1-seg |        0.55 |   ops/s |
|                                        50th percentile latency |  asc-sort-with-after-timestamp-after-force-merge-1-seg |     74.1798 |      ms |
|                                        90th percentile latency |  asc-sort-with-after-timestamp-after-force-merge-1-seg |      76.329 |      ms |
|                                        99th percentile latency |  asc-sort-with-after-timestamp-after-force-merge-1-seg |     76.8501 |      ms |
|                                      99.9th percentile latency |  asc-sort-with-after-timestamp-after-force-merge-1-seg |             |         |
|                                     99.99th percentile latency |  asc-sort-with-after-timestamp-after-force-merge-1-seg |             |         |
|                                       100th percentile latency |  asc-sort-with-after-timestamp-after-force-merge-1-seg |     77.1029 |      ms |
|                                   50th percentile service time |  asc-sort-with-after-timestamp-after-force-merge-1-seg |     73.1626 |      ms |
|                                   90th percentile service time |  asc-sort-with-after-timestamp-after-force-merge-1-seg |     73.6418 |      ms |
|                                   99th percentile service time |  asc-sort-with-after-timestamp-after-force-merge-1-seg |     74.4617 |      ms |
|                                 99.9th percentile service time |  asc-sort-with-after-timestamp-after-force-merge-1-seg |             |         |
|                                99.99th percentile service time |  asc-sort-with-after-timestamp-after-force-merge-1-seg |             |         |
|                                  100th percentile service time |  asc-sort-with-after-timestamp-after-force-merge-1-seg |     74.6101 |      ms |
|                                                     error rate |  asc-sort-with-after-timestamp-after-force-merge-1-seg |           0 |       % |


----------------------------------
[INFO] SUCCESS (took 6241 seconds)
----------------------------------

nyc_taxis,docrep (Run 1)

|                                                         Metric |                     Task |       Value |   Unit |
|---------------------------------------------------------------:|-------------------------:|------------:|-------:|
|                     Cumulative indexing time of primary shards |                          |       135.5 |    min |
|             Min cumulative indexing time across primary shards |                          |     43.8216 |    min |
|          Median cumulative indexing time across primary shards |                          |     45.8146 |    min |
|             Max cumulative indexing time across primary shards |                          |     45.8639 |    min |
|            Cumulative indexing throttle time of primary shards |                          |           0 |    min |
|    Min cumulative indexing throttle time across primary shards |                          |           0 |    min |
| Median cumulative indexing throttle time across primary shards |                          |           0 |    min |
|    Max cumulative indexing throttle time across primary shards |                          |           0 |    min |
|                        Cumulative merge time of primary shards |                          |     82.5749 |    min |
|                       Cumulative merge count of primary shards |                          |          55 |        |
|                Min cumulative merge time across primary shards |                          |     27.4068 |    min |
|             Median cumulative merge time across primary shards |                          |     27.4996 |    min |
|                Max cumulative merge time across primary shards |                          |     27.6685 |    min |
|               Cumulative merge throttle time of primary shards |                          |     41.9854 |    min |
|       Min cumulative merge throttle time across primary shards |                          |     13.8264 |    min |
|    Median cumulative merge throttle time across primary shards |                          |     13.8935 |    min |
|       Max cumulative merge throttle time across primary shards |                          |     14.2655 |    min |
|                      Cumulative refresh time of primary shards |                          |      21.747 |    min |
|                     Cumulative refresh count of primary shards |                          |         173 |        |
|              Min cumulative refresh time across primary shards |                          |     7.06715 |    min |
|           Median cumulative refresh time across primary shards |                          |     7.28802 |    min |
|              Max cumulative refresh time across primary shards |                          |     7.39178 |    min |
|                        Cumulative flush time of primary shards |                          |     4.96367 |    min |
|                       Cumulative flush count of primary shards |                          |          27 |        |
|                Min cumulative flush time across primary shards |                          |     1.42222 |    min |
|             Median cumulative flush time across primary shards |                          |     1.57757 |    min |
|                Max cumulative flush time across primary shards |                          |     1.96388 |    min |
|                                        Total Young Gen GC time |                          |      21.384 |      s |
|                                       Total Young Gen GC count |                          |         756 |        |
|                                          Total Old Gen GC time |                          |           0 |      s |
|                                         Total Old Gen GC count |                          |           0 |        |
|                                                     Store size |                          |     46.5918 |     GB |
|                                                  Translog size |                          | 3.07336e-07 |     GB |
|                                         Heap used for segments |                          |           0 |     MB |
|                                       Heap used for doc values |                          |           0 |     MB |
|                                            Heap used for terms |                          |           0 |     MB |
|                                            Heap used for norms |                          |           0 |     MB |
|                                           Heap used for points |                          |           0 |     MB |
|                                    Heap used for stored fields |                          |           0 |     MB |
|                                                  Segment count |                          |          85 |        |
|                                                 Min Throughput |                    index |     84825.4 | docs/s |
|                                                Mean Throughput |                    index |     91828.7 | docs/s |
|                                              Median Throughput |                    index |     91287.5 | docs/s |
|                                                 Max Throughput |                    index |      101114 | docs/s |
|                                        50th percentile latency |                    index |     745.307 |     ms |
|                                        90th percentile latency |                    index |     1035.17 |     ms |
|                                        99th percentile latency |                    index |     4060.47 |     ms |
|                                      99.9th percentile latency |                    index |     16758.7 |     ms |
|                                     99.99th percentile latency |                    index |     20086.3 |     ms |
|                                       100th percentile latency |                    index |     20923.1 |     ms |
|                                   50th percentile service time |                    index |     745.307 |     ms |
|                                   90th percentile service time |                    index |     1035.17 |     ms |
|                                   99th percentile service time |                    index |     4060.47 |     ms |
|                                 99.9th percentile service time |                    index |     16758.7 |     ms |
|                                99.99th percentile service time |                    index |     20086.3 |     ms |
|                                  100th percentile service time |                    index |     20923.1 |     ms |
|                                                     error rate |                    index |           0 |      % |
|                                                 Min Throughput | wait-until-merges-finish |           0 |  ops/s |
|                                                Mean Throughput | wait-until-merges-finish |           0 |  ops/s |
|                                              Median Throughput | wait-until-merges-finish |           0 |  ops/s |
|                                                 Max Throughput | wait-until-merges-finish |           0 |  ops/s |
|                                        50th percentile latency | wait-until-merges-finish |             |        |
|                                        90th percentile latency | wait-until-merges-finish |             |        |
|                                        99th percentile latency | wait-until-merges-finish |             |        |
|                                      99.9th percentile latency | wait-until-merges-finish |             |        |
|                                     99.99th percentile latency | wait-until-merges-finish |             |        |
|                                       100th percentile latency | wait-until-merges-finish |      461893 |     ms |
|                                   50th percentile service time | wait-until-merges-finish |             |        |
|                                   90th percentile service time | wait-until-merges-finish |             |        |
|                                   99th percentile service time | wait-until-merges-finish |             |        |
|                                 99.9th percentile service time | wait-until-merges-finish |             |        |
|                                99.99th percentile service time | wait-until-merges-finish |             |        |
|                                  100th percentile service time | wait-until-merges-finish |      461893 |     ms |
|                                                     error rate | wait-until-merges-finish |           0 |      % |
|                                                 Min Throughput |                  default |        3.01 |  ops/s |
|                                                Mean Throughput |                  default |        3.01 |  ops/s |
|                                              Median Throughput |                  default |        3.01 |  ops/s |
|                                                 Max Throughput |                  default |        3.02 |  ops/s |
|                                        50th percentile latency |                  default |     7.16909 |     ms |
|                                        90th percentile latency |                  default |     7.81019 |     ms |
|                                        99th percentile latency |                  default |     8.22108 |     ms |
|                                      99.9th percentile latency |                  default |             |        |
|                                     99.99th percentile latency |                  default |             |        |
|                                       100th percentile latency |                  default |     8.65206 |     ms |
|                                   50th percentile service time |                  default |     6.06026 |     ms |
|                                   90th percentile service time |                  default |     6.48309 |     ms |
|                                   99th percentile service time |                  default |     6.84017 |     ms |
|                                 99.9th percentile service time |                  default |             |        |
|                                99.99th percentile service time |                  default |             |        |
|                                  100th percentile service time |                  default |     7.56981 |     ms |
|                                                     error rate |                  default |           0 |      % |
|                                                 Min Throughput |                    range |         0.7 |  ops/s |
|                                                Mean Throughput |                    range |        0.71 |  ops/s |
|                                              Median Throughput |                    range |         0.7 |  ops/s |
|                                                 Max Throughput |                    range |        0.71 |  ops/s |
|                                        50th percentile latency |                    range |     86.9624 |     ms |
|                                        90th percentile latency |                    range |     87.9659 |     ms |
|                                        99th percentile latency |                    range |     91.5293 |     ms |
|                                      99.9th percentile latency |                    range |             |        |
|                                     99.99th percentile latency |                    range |             |        |
|                                       100th percentile latency |                    range |     93.7098 |     ms |
|                                   50th percentile service time |                    range |     84.8225 |     ms |
|                                   90th percentile service time |                    range |     85.8824 |     ms |
|                                   99th percentile service time |                    range |     89.2953 |     ms |
|                                 99.9th percentile service time |                    range |             |        |
|                                99.99th percentile service time |                    range |             |        |
|                                  100th percentile service time |                    range |     91.7161 |     ms |
|                                                     error rate |                    range |           0 |      % |
|                                                 Min Throughput |      distance_amount_agg |        2.01 |  ops/s |
|                                                Mean Throughput |      distance_amount_agg |        2.02 |  ops/s |
|                                              Median Throughput |      distance_amount_agg |        2.02 |  ops/s |
|                                                 Max Throughput |      distance_amount_agg |        2.03 |  ops/s |
|                                        50th percentile latency |      distance_amount_agg |     5.50442 |     ms |
|                                        90th percentile latency |      distance_amount_agg |     6.03863 |     ms |
|                                        99th percentile latency |      distance_amount_agg |     6.23015 |     ms |
|                                      99.9th percentile latency |      distance_amount_agg |             |        |
|                                     99.99th percentile latency |      distance_amount_agg |             |        |
|                                       100th percentile latency |      distance_amount_agg |     6.53021 |     ms |
|                                   50th percentile service time |      distance_amount_agg |     4.22437 |     ms |
|                                   90th percentile service time |      distance_amount_agg |     4.51316 |     ms |
|                                   99th percentile service time |      distance_amount_agg |      4.7846 |     ms |
|                                 99.9th percentile service time |      distance_amount_agg |             |        |
|                                99.99th percentile service time |      distance_amount_agg |             |        |
|                                  100th percentile service time |      distance_amount_agg |     4.87008 |     ms |
|                                                     error rate |      distance_amount_agg |           0 |      % |
|                                                 Min Throughput |            autohisto_agg |        1.47 |  ops/s |
|                                                Mean Throughput |            autohisto_agg |        1.48 |  ops/s |
|                                              Median Throughput |            autohisto_agg |        1.49 |  ops/s |
|                                                 Max Throughput |            autohisto_agg |        1.49 |  ops/s |
|                                        50th percentile latency |            autohisto_agg |      214.91 |     ms |
|                                        90th percentile latency |            autohisto_agg |      224.37 |     ms |
|                                        99th percentile latency |            autohisto_agg |      225.96 |     ms |
|                                      99.9th percentile latency |            autohisto_agg |             |        |
|                                     99.99th percentile latency |            autohisto_agg |             |        |
|                                       100th percentile latency |            autohisto_agg |     226.271 |     ms |
|                                   50th percentile service time |            autohisto_agg |     213.805 |     ms |
|                                   90th percentile service time |            autohisto_agg |     222.963 |     ms |
|                                   99th percentile service time |            autohisto_agg |     224.311 |     ms |
|                                 99.9th percentile service time |            autohisto_agg |             |        |
|                                99.99th percentile service time |            autohisto_agg |             |        |
|                                  100th percentile service time |            autohisto_agg |     224.779 |     ms |
|                                                     error rate |            autohisto_agg |           0 |      % |
|                                                 Min Throughput |       date_histogram_agg |        1.51 |  ops/s |
|                                                Mean Throughput |       date_histogram_agg |        1.51 |  ops/s |
|                                              Median Throughput |       date_histogram_agg |        1.51 |  ops/s |
|                                                 Max Throughput |       date_histogram_agg |        1.52 |  ops/s |
|                                        50th percentile latency |       date_histogram_agg |     230.974 |     ms |
|                                        90th percentile latency |       date_histogram_agg |     232.943 |     ms |
|                                        99th percentile latency |       date_histogram_agg |     238.607 |     ms |
|                                      99.9th percentile latency |       date_histogram_agg |             |        |
|                                     99.99th percentile latency |       date_histogram_agg |             |        |
|                                       100th percentile latency |       date_histogram_agg |     301.133 |     ms |
|                                   50th percentile service time |       date_histogram_agg |     229.703 |     ms |
|                                   90th percentile service time |       date_histogram_agg |     231.506 |     ms |
|                                   99th percentile service time |       date_histogram_agg |     237.386 |     ms |
|                                 99.9th percentile service time |       date_histogram_agg |             |        |
|                                99.99th percentile service time |       date_histogram_agg |             |        |
|                                  100th percentile service time |       date_histogram_agg |      299.64 |     ms |
|                                                     error rate |       date_histogram_agg |           0 |      % |

nyc_taxis,remote,bic-16,bs-20k,ri-10s,bri-850ms (Run 1)

|                                                         Metric |                     Task |       Value |   Unit |
|---------------------------------------------------------------:|-------------------------:|------------:|-------:|
|                     Cumulative indexing time of primary shards |                          |     271.067 |    min |
|             Min cumulative indexing time across primary shards |                          |     89.4051 |    min |
|          Median cumulative indexing time across primary shards |                          |     89.5424 |    min |
|             Max cumulative indexing time across primary shards |                          |     92.1193 |    min |
|            Cumulative indexing throttle time of primary shards |                          |           0 |    min |
|    Min cumulative indexing throttle time across primary shards |                          |           0 |    min |
| Median cumulative indexing throttle time across primary shards |                          |           0 |    min |
|    Max cumulative indexing throttle time across primary shards |                          |           0 |    min |
|                        Cumulative merge time of primary shards |                          |     150.192 |    min |
|                       Cumulative merge count of primary shards |                          |         248 |        |
|                Min cumulative merge time across primary shards |                          |     46.4701 |    min |
|             Median cumulative merge time across primary shards |                          |     47.8975 |    min |
|                Max cumulative merge time across primary shards |                          |     55.8244 |    min |
|               Cumulative merge throttle time of primary shards |                          |     91.8982 |    min |
|       Min cumulative merge throttle time across primary shards |                          |     27.8968 |    min |
|    Median cumulative merge throttle time across primary shards |                          |     29.6001 |    min |
|       Max cumulative merge throttle time across primary shards |                          |     34.4013 |    min |
|                      Cumulative refresh time of primary shards |                          |     11.2603 |    min |
|                     Cumulative refresh count of primary shards |                          |         493 |        |
|              Min cumulative refresh time across primary shards |                          |     3.72943 |    min |
|           Median cumulative refresh time across primary shards |                          |     3.73055 |    min |
|              Max cumulative refresh time across primary shards |                          |     3.80037 |    min |
|                        Cumulative flush time of primary shards |                          |     18.3609 |    min |
|                       Cumulative flush count of primary shards |                          |         130 |        |
|                Min cumulative flush time across primary shards |                          |     5.54427 |    min |
|             Median cumulative flush time across primary shards |                          |     6.19268 |    min |
|                Max cumulative flush time across primary shards |                          |     6.62397 |    min |
|                                        Total Young Gen GC time |                          |      15.882 |      s |
|                                       Total Young Gen GC count |                          |         506 |        |
|                                          Total Old Gen GC time |                          |           0 |      s |
|                                         Total Old Gen GC count |                          |           0 |        |
|                                                     Store size |                          |     70.3883 |     GB |
|                                                  Translog size |                          | 3.07336e-07 |     GB |
|                                         Heap used for segments |                          |           0 |     MB |
|                                       Heap used for doc values |                          |           0 |     MB |
|                                            Heap used for terms |                          |           0 |     MB |
|                                            Heap used for norms |                          |           0 |     MB |
|                                           Heap used for points |                          |           0 |     MB |
|                                    Heap used for stored fields |                          |           0 |     MB |
|                                                  Segment count |                          |          81 |        |
|                                                 Min Throughput |                    index |      101989 | docs/s |
|                                                Mean Throughput |                    index |      103620 | docs/s |
|                                              Median Throughput |                    index |      103635 | docs/s |
|                                                 Max Throughput |                    index |      105369 | docs/s |
|                                        50th percentile latency |                    index |     2923.37 |     ms |
|                                        90th percentile latency |                    index |     3912.72 |     ms |
|                                        99th percentile latency |                    index |     6306.76 |     ms |
|                                      99.9th percentile latency |                    index |     7681.07 |     ms |
|                                     99.99th percentile latency |                    index |             |        |
|                                       100th percentile latency |                    index |     10995.8 |     ms |
|                                   50th percentile service time |                    index |     2923.37 |     ms |
|                                   90th percentile service time |                    index |     3912.72 |     ms |
|                                   99th percentile service time |                    index |     6306.76 |     ms |
|                                 99.9th percentile service time |                    index |     7681.07 |     ms |
|                                99.99th percentile service time |                    index |             |        |
|                                  100th percentile service time |                    index |     10995.8 |     ms |
|                                                     error rate |                    index |           0 |      % |
|                                                 Min Throughput | wait-until-merges-finish |           0 |  ops/s |
|                                                Mean Throughput | wait-until-merges-finish |           0 |  ops/s |
|                                              Median Throughput | wait-until-merges-finish |           0 |  ops/s |
|                                                 Max Throughput | wait-until-merges-finish |           0 |  ops/s |
|                                        50th percentile latency | wait-until-merges-finish |             |        |
|                                        90th percentile latency | wait-until-merges-finish |             |        |
|                                        99th percentile latency | wait-until-merges-finish |             |        |
|                                      99.9th percentile latency | wait-until-merges-finish |             |        |
|                                     99.99th percentile latency | wait-until-merges-finish |             |        |
|                                       100th percentile latency | wait-until-merges-finish |      960657 |     ms |
|                                   50th percentile service time | wait-until-merges-finish |             |        |
|                                   90th percentile service time | wait-until-merges-finish |             |        |
|                                   99th percentile service time | wait-until-merges-finish |             |        |
|                                 99.9th percentile service time | wait-until-merges-finish |             |        |
|                                99.99th percentile service time | wait-until-merges-finish |             |        |
|                                  100th percentile service time | wait-until-merges-finish |      960657 |     ms |
|                                                     error rate | wait-until-merges-finish |           0 |      % |
|                                                 Min Throughput |                  default |        3.01 |  ops/s |
|                                                Mean Throughput |                  default |        3.01 |  ops/s |
|                                              Median Throughput |                  default |        3.01 |  ops/s |
|                                                 Max Throughput |                  default |        3.02 |  ops/s |
|                                        50th percentile latency |                  default |      6.4153 |     ms |
|                                        90th percentile latency |                  default |     7.19633 |     ms |
|                                        99th percentile latency |                  default |     7.61608 |     ms |
|                                      99.9th percentile latency |                  default |             |        |
|                                     99.99th percentile latency |                  default |             |        |
|                                       100th percentile latency |                  default |     8.37806 |     ms |
|                                   50th percentile service time |                  default |     5.29942 |     ms |
|                                   90th percentile service time |                  default |     5.93527 |     ms |
|                                   99th percentile service time |                  default |     6.46126 |     ms |
|                                 99.9th percentile service time |                  default |             |        |
|                                99.99th percentile service time |                  default |             |        |
|                                  100th percentile service time |                  default |     7.64886 |     ms |
|                                                     error rate |                  default |           0 |      % |
|                                                 Min Throughput |                    range |         0.7 |  ops/s |
|                                                Mean Throughput |                    range |        0.71 |  ops/s |
|                                              Median Throughput |                    range |         0.7 |  ops/s |
|                                                 Max Throughput |                    range |        0.71 |  ops/s |
|                                        50th percentile latency |                    range |     87.6798 |     ms |
|                                        90th percentile latency |                    range |      88.975 |     ms |
|                                        99th percentile latency |                    range |      90.768 |     ms |
|                                      99.9th percentile latency |                    range |             |        |
|                                     99.99th percentile latency |                    range |             |        |
|                                       100th percentile latency |                    range |       159.9 |     ms |
|                                   50th percentile service time |                    range |     85.6683 |     ms |
|                                   90th percentile service time |                    range |     86.7456 |     ms |
|                                   99th percentile service time |                    range |     88.6935 |     ms |
|                                 99.9th percentile service time |                    range |             |        |
|                                99.99th percentile service time |                    range |             |        |
|                                  100th percentile service time |                    range |     157.395 |     ms |
|                                                     error rate |                    range |           0 |      % |
|                                                 Min Throughput |      distance_amount_agg |        2.01 |  ops/s |
|                                                Mean Throughput |      distance_amount_agg |        2.01 |  ops/s |
|                                              Median Throughput |      distance_amount_agg |        2.01 |  ops/s |
|                                                 Max Throughput |      distance_amount_agg |        2.03 |  ops/s |
|                                        50th percentile latency |      distance_amount_agg |     5.59813 |     ms |
|                                        90th percentile latency |      distance_amount_agg |     6.14908 |     ms |
|                                        99th percentile latency |      distance_amount_agg |      6.4628 |     ms |
|                                      99.9th percentile latency |      distance_amount_agg |             |        |
|                                     99.99th percentile latency |      distance_amount_agg |             |        |
|                                       100th percentile latency |      distance_amount_agg |     6.46897 |     ms |
|                                   50th percentile service time |      distance_amount_agg |     4.32063 |     ms |
|                                   90th percentile service time |      distance_amount_agg |     4.58106 |     ms |
|                                   99th percentile service time |      distance_amount_agg |     5.05213 |     ms |
|                                 99.9th percentile service time |      distance_amount_agg |             |        |
|                                99.99th percentile service time |      distance_amount_agg |             |        |
|                                  100th percentile service time |      distance_amount_agg |     5.06485 |     ms |
|                                                     error rate |      distance_amount_agg |           0 |      % |
|                                                 Min Throughput |            autohisto_agg |        1.49 |  ops/s |
|                                                Mean Throughput |            autohisto_agg |        1.49 |  ops/s |
|                                              Median Throughput |            autohisto_agg |        1.49 |  ops/s |
|                                                 Max Throughput |            autohisto_agg |         1.5 |  ops/s |
|                                        50th percentile latency |            autohisto_agg |      188.02 |     ms |
|                                        90th percentile latency |            autohisto_agg |      189.93 |     ms |
|                                        99th percentile latency |            autohisto_agg |     196.041 |     ms |
|                                      99.9th percentile latency |            autohisto_agg |             |        |
|                                     99.99th percentile latency |            autohisto_agg |             |        |
|                                       100th percentile latency |            autohisto_agg |     196.117 |     ms |
|                                   50th percentile service time |            autohisto_agg |     186.299 |     ms |
|                                   90th percentile service time |            autohisto_agg |     187.776 |     ms |
|                                   99th percentile service time |            autohisto_agg |     194.182 |     ms |
|                                 99.9th percentile service time |            autohisto_agg |             |        |
|                                99.99th percentile service time |            autohisto_agg |             |        |
|                                  100th percentile service time |            autohisto_agg |     194.452 |     ms |
|                                                     error rate |            autohisto_agg |           0 |      % |
|                                                 Min Throughput |       date_histogram_agg |        1.51 |  ops/s |
|                                                Mean Throughput |       date_histogram_agg |        1.51 |  ops/s |
|                                              Median Throughput |       date_histogram_agg |        1.51 |  ops/s |
|                                                 Max Throughput |       date_histogram_agg |        1.52 |  ops/s |
|                                        50th percentile latency |       date_histogram_agg |     202.924 |     ms |
|                                        90th percentile latency |       date_histogram_agg |     204.751 |     ms |
|                                        99th percentile latency |       date_histogram_agg |     207.347 |     ms |
|                                      99.9th percentile latency |       date_histogram_agg |             |        |
|                                     99.99th percentile latency |       date_histogram_agg |             |        |
|                                       100th percentile latency |       date_histogram_agg |     208.849 |     ms |
|                                   50th percentile service time |       date_histogram_agg |     201.887 |     ms |
|                                   90th percentile service time |       date_histogram_agg |     202.957 |     ms |
|                                   99th percentile service time |       date_histogram_agg |     205.905 |     ms |
|                                 99.9th percentile service time |       date_histogram_agg |             |        |
|                                99.99th percentile service time |       date_histogram_agg |             |        |
|                                  100th percentile service time |       date_histogram_agg |     207.193 |     ms |
|                                                     error rate |       date_histogram_agg |           0 |      % |


----------------------------------
[INFO] SUCCESS (took 3181 seconds)
----------------------------------

@rishabh6788
Copy link
Contributor

@ashking94 Thanks for adding the results, I can confirm that I am seeing similar results with the cluster and workload configuration you have mentioned in the issue.

@rishabh6788
Copy link
Contributor

rishabh6788 commented Sep 7, 2023 •
edited
Loading

so workload indexing throughput with default workload params with 250ms buffer:
https://build.ci.opensearch.org/blue/rest/organizations/jenkins/pipelines/benchmark-test/runs/6699/nodes/37/steps/107/log/?start=0

Mean Throughput |             index-append |     19683.4 | docs/s |
Median Throughput |             index-append |     19647.6 | docs/s |
Max Throughput |             index-append |     20349.9 | docs/s |

With 650ms buffer:
https://build.ci.opensearch.org/blue/rest/organizations/jenkins/pipelines/benchmark-test/runs/6704/nodes/37/steps/107/log/?start=0

Mean Throughput |             index-append |      24638 | docs/s |
Median Throughput |             index-append |    24800.8 | docs/s |
Max Throughput |             index-append |    25189.6 | docs/s |

That is a ~20% jump.

Running the same comparison with http_logs as well. Cluster config is 3 data (r5.xlarge)+3 master (c5.xlarge), 3 shards with 1-replica. Default workload params, 8 clients/5k bulk_size

@rishabh6788
Copy link
Contributor

Running http_logs with default workload params is still showing regression on r5.xlarge/3-shard-1-replica config with 250ms and 650ms buffer interval.

250ms:
https://build.ci.opensearch.org/blue/rest/organizations/jenkins/pipelines/benchmark-test/runs/6698/nodes/37/steps/107/log/?start=0

Min Throughput |                                           index-append |     54891.5 |  docs/s |
Mean Throughput |                                           index-append |     60060.5 |  docs/s |
Median Throughput |                                           index-append |     60270.4 |  docs/s |
Max Throughput |                                           index-append |     60881.2 |  docs/s |

650ms:
https://build.ci.opensearch.org/blue/rest/organizations/jenkins/pipelines/benchmark-test/runs/6701/nodes/37/steps/107/log/?start=0

Min Throughput |                                           index-append |     55662.4 |  docs/s |
Mean Throughput |                                           index-append |     59162.3 |  docs/s |
Median Throughput |                                           index-append |     59540.4 |  docs/s |
Max Throughput |                                           index-append |     59704.1 |  docs/s |

Running new benchmark on r5.xlarge host with below workload params:
{"bulk_indexing_clients":"24","number_of_shards":"6","number_of_replicas":"1","index_settings":{"index":{"refresh_interval":"10s"}}} against 250ms and 650ms buffer intervals.
250: https://build.ci.opensearch.org/blue/organizations/jenkins/benchmark-test/detail/benchmark-test/6717/pipeline/37
650: https://build.ci.opensearch.org/blue/organizations/jenkins/benchmark-test/detail/benchmark-test/6718/pipeline

@tlfeng
Copy link
Collaborator

tlfeng commented Sep 7, 2023 •
edited
Loading

For reference, I have ran performance test for 6 different scenarios to compare "segment replication + remote store" vs "document replication" in the issue #8874 .
The test scenario with different node numbers, replica numbers and shard count, and shard size.
It can be a reference to see performance when using default cluster or index configurations.

@ashking94 ashking94 mentioned this issue Sep 20, 2023
Closed
4 tasks
@Bukhtawar Bukhtawar mentioned this issue Jun 27, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
discuss Issues intended to help drive brainstorming and decision making enhancement Enhancement or improvement to existing feature or request Storage:Durability Issues and PRs related to the durability framework v2.10.0
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@sachinpkale @ashking94 @rishabh6788 @tlfeng