Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add benchmark to measure CustomBinaryDocValuesField performance #9426

Merged
merged 1 commit into from
Aug 30, 2023

Conversation

kkmr
Copy link
Contributor

@kkmr kkmr commented Aug 17, 2023

Add benchmark to measure CustomBinaryDocValuesField.

Description

We were considering using a TreeMap in CustomBinaryDocValuesField instead of ArrayList. Benchmarks showed that ArrayList performs better. We are keeping the benchmark code for future reference.

Related Issues

Resolves #6897

Check List

  • New functionality includes testing.
    • All tests pass
  • New functionality has been documented.
    • New functionality has javadoc added
  • Commits are signed per the DCO using --signoff
  • Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@kkmr
Copy link
Contributor Author

kkmr commented Aug 27, 2023

It looks like the old code performs much better. Most likely, the performance is not dominated by the sort.

New code Performance

Benchmark                                        (entrySize)  (maximumNumberOfEntries)   Mode  Cnt     Score   Error   Units
CustomBinaryDocValuesFieldBenchmark.add                    8                         8  thrpt       4826.142          ops/ms
CustomBinaryDocValuesFieldBenchmark.add                    8                        32  thrpt       1380.164          ops/ms
CustomBinaryDocValuesFieldBenchmark.add                    8                       128  thrpt        358.869          ops/ms
CustomBinaryDocValuesFieldBenchmark.add                    8                       512  thrpt         90.848          ops/ms
CustomBinaryDocValuesFieldBenchmark.add                   32                         8  thrpt       2506.266          ops/ms
CustomBinaryDocValuesFieldBenchmark.add                   32                        32  thrpt        662.682          ops/ms
CustomBinaryDocValuesFieldBenchmark.add                   32                       128  thrpt        168.800          ops/ms
CustomBinaryDocValuesFieldBenchmark.add                   32                       512  thrpt         40.062          ops/ms
CustomBinaryDocValuesFieldBenchmark.add                  128                         8  thrpt        742.890          ops/ms
CustomBinaryDocValuesFieldBenchmark.add                  128                        32  thrpt        188.871          ops/ms
CustomBinaryDocValuesFieldBenchmark.add                  128                       128  thrpt         47.518          ops/ms
CustomBinaryDocValuesFieldBenchmark.add                  128                       512  thrpt         11.920          ops/ms
CustomBinaryDocValuesFieldBenchmark.add                  512                         8  thrpt        210.733          ops/ms
CustomBinaryDocValuesFieldBenchmark.add                  512                        32  thrpt         53.659          ops/ms
CustomBinaryDocValuesFieldBenchmark.add                  512                       128  thrpt         13.238          ops/ms
CustomBinaryDocValuesFieldBenchmark.add                  512                       512  thrpt          3.352          ops/ms
CustomBinaryDocValuesFieldBenchmark.binaryValue            8                         8  thrpt       9542.207          ops/ms
CustomBinaryDocValuesFieldBenchmark.binaryValue            8                        32  thrpt       9916.014          ops/ms
CustomBinaryDocValuesFieldBenchmark.binaryValue            8                       128  thrpt       9466.273          ops/ms
CustomBinaryDocValuesFieldBenchmark.binaryValue            8                       512  thrpt       9323.077          ops/ms
CustomBinaryDocValuesFieldBenchmark.binaryValue           32                         8  thrpt       9253.423          ops/ms
CustomBinaryDocValuesFieldBenchmark.binaryValue           32                        32  thrpt       9535.728          ops/ms
CustomBinaryDocValuesFieldBenchmark.binaryValue           32                       128  thrpt       9156.015          ops/ms
CustomBinaryDocValuesFieldBenchmark.binaryValue           32                       512  thrpt       9203.324          ops/ms
CustomBinaryDocValuesFieldBenchmark.binaryValue          128                         8  thrpt       7356.562          ops/ms
CustomBinaryDocValuesFieldBenchmark.binaryValue          128                        32  thrpt       7757.455          ops/ms
CustomBinaryDocValuesFieldBenchmark.binaryValue          128                       128  thrpt       7553.087          ops/ms
CustomBinaryDocValuesFieldBenchmark.binaryValue          128                       512  thrpt       7496.237          ops/ms
CustomBinaryDocValuesFieldBenchmark.binaryValue          512                         8  thrpt       5050.827          ops/ms
CustomBinaryDocValuesFieldBenchmark.binaryValue          512                        32  thrpt       5048.714          ops/ms
CustomBinaryDocValuesFieldBenchmark.binaryValue          512                       128  thrpt       4975.164          ops/ms
CustomBinaryDocValuesFieldBenchmark.binaryValue          512                       512  thrpt       5085.280          ops/ms

Old code performance

Benchmark                                        (entrySize)  (maximumNumberOfEntries)   Mode  Cnt      Score   Error   Units
CustomBinaryDocValuesFieldBenchmark.add                    8                         8  thrpt        9012.270          ops/ms
CustomBinaryDocValuesFieldBenchmark.add                    8                        32  thrpt        2002.192          ops/ms
CustomBinaryDocValuesFieldBenchmark.add                    8                       128  thrpt         571.450          ops/ms
CustomBinaryDocValuesFieldBenchmark.add                    8                       512  thrpt         153.650          ops/ms
CustomBinaryDocValuesFieldBenchmark.add                   32                         8  thrpt        3256.273          ops/ms
CustomBinaryDocValuesFieldBenchmark.add                   32                        32  thrpt         735.980          ops/ms
CustomBinaryDocValuesFieldBenchmark.add                   32                       128  thrpt         201.739          ops/ms
CustomBinaryDocValuesFieldBenchmark.add                   32                       512  thrpt          52.079          ops/ms
CustomBinaryDocValuesFieldBenchmark.add                  128                         8  thrpt         807.506          ops/ms
CustomBinaryDocValuesFieldBenchmark.add                  128                        32  thrpt         190.050          ops/ms
CustomBinaryDocValuesFieldBenchmark.add                  128                       128  thrpt          49.815          ops/ms
CustomBinaryDocValuesFieldBenchmark.add                  128                       512  thrpt          12.696          ops/ms
CustomBinaryDocValuesFieldBenchmark.add                  512                         8  thrpt         215.063          ops/ms
CustomBinaryDocValuesFieldBenchmark.add                  512                        32  thrpt          52.353          ops/ms
CustomBinaryDocValuesFieldBenchmark.add                  512                       128  thrpt          13.426          ops/ms
CustomBinaryDocValuesFieldBenchmark.add                  512                       512  thrpt           3.326          ops/ms
CustomBinaryDocValuesFieldBenchmark.binaryValue            8                         8  thrpt       13037.574          ops/ms
CustomBinaryDocValuesFieldBenchmark.binaryValue            8                        32  thrpt       12014.520          ops/ms
CustomBinaryDocValuesFieldBenchmark.binaryValue            8                       128  thrpt       13252.598          ops/ms
CustomBinaryDocValuesFieldBenchmark.binaryValue            8                       512  thrpt       12223.060          ops/ms
CustomBinaryDocValuesFieldBenchmark.binaryValue           32                         8  thrpt       13465.619          ops/ms
CustomBinaryDocValuesFieldBenchmark.binaryValue           32                        32  thrpt       13116.687          ops/ms
CustomBinaryDocValuesFieldBenchmark.binaryValue           32                       128  thrpt       13456.720          ops/ms
CustomBinaryDocValuesFieldBenchmark.binaryValue           32                       512  thrpt       12030.096          ops/ms
CustomBinaryDocValuesFieldBenchmark.binaryValue          128                         8  thrpt       10226.014          ops/ms
CustomBinaryDocValuesFieldBenchmark.binaryValue          128                        32  thrpt        9657.811          ops/ms
CustomBinaryDocValuesFieldBenchmark.binaryValue          128                       128  thrpt       10168.291          ops/ms
CustomBinaryDocValuesFieldBenchmark.binaryValue          128                       512  thrpt        9972.951          ops/ms
CustomBinaryDocValuesFieldBenchmark.binaryValue          512                         8  thrpt        7945.902          ops/ms
CustomBinaryDocValuesFieldBenchmark.binaryValue          512                        32  thrpt        7865.644          ops/ms
CustomBinaryDocValuesFieldBenchmark.binaryValue          512                       128  thrpt        7945.872          ops/ms
CustomBinaryDocValuesFieldBenchmark.binaryValue          512                       512  thrpt        7702.249          ops/ms

@kkmr kkmr requested a review from msfroh as a code owner August 27, 2023 16:56
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@andrross andrross changed the title [Refactor] Use TreeSet instead of ArrayList Add benchmark to measure performance of CustomBinaryDocValuesField Aug 29, 2023
@andrross andrross changed the title Add benchmark to measure performance of CustomBinaryDocValuesField Add benchmark to measure CustomBinaryDocValuesField performance Aug 29, 2023
@andrross andrross added the backport 2.x Backport to 2.x branch label Aug 29, 2023
Benchmarks show that ArrayList performs better than TreeSet. Added a comment
on where to find the results and benchmark.

Signed-off-by: Kiran Reddy <kkreddy@amazon.com>
@kkmr kkmr force-pushed the kkmr/OpenSearch branch from c4868a3 to 47056c3 Compare August 29, 2023 22:32
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

Compatibility status:

Checks if related components are compatible with change c3ff05d

Incompatible components

Incompatible components: [https://github.com/opensearch-project/cross-cluster-replication.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@codecov
Copy link

codecov bot commented Aug 29, 2023

Codecov Report

Merging #9426 (47056c3) into main (60787b8) will decrease coverage by 0.41%.
Report is 6 commits behind head on main.
The diff coverage is n/a.

@@             Coverage Diff              @@
##               main    #9426      +/-   ##
============================================
- Coverage     71.53%   71.12%   -0.41%     
+ Complexity    57897    57580     -317     
============================================
  Files          4782     4782              
  Lines        271332   271332              
  Branches      39614    39614              
============================================
- Hits         194107   192997    -1110     
- Misses        61255    62073     +818     
- Partials      15970    16262     +292     
Files Changed Coverage Δ
...org/opensearch/index/mapper/BinaryFieldMapper.java 78.04% <ø> (ø)

... and 471 files with indirect coverage changes

@github-actions
Copy link
Contributor

Compatibility status:

Checks if related components are compatible with change 47056c3

Incompatible components

Incompatible components: [https://github.com/opensearch-project/cross-cluster-replication.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@andrross
Copy link
Member

Known test failure: #9580

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@andrross
Copy link
Member

andrross commented Aug 30, 2023

Error response from daemon: login attempt to https://registry-1.docker.io/v2/ failed with status: 503 Service Unavailable

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@andrross andrross merged commit cc007e4 into opensearch-project:main Aug 30, 2023
opensearch-trigger-bot bot pushed a commit that referenced this pull request Aug 30, 2023
…9426)

Benchmarks show that ArrayList performs better than TreeSet. Added a comment
on where to find the results and benchmark.

Signed-off-by: Kiran Reddy <kkreddy@amazon.com>
(cherry picked from commit cc007e4)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
@kkmr kkmr deleted the kkmr/OpenSearch branch August 30, 2023 22:28
reta pushed a commit that referenced this pull request Aug 31, 2023
…9426) (#9649)

Benchmarks show that ArrayList performs better than TreeSet. Added a comment
on where to find the results and benchmark.


(cherry picked from commit cc007e4)

Signed-off-by: Kiran Reddy <kkreddy@amazon.com>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
kaushalmahi12 pushed a commit to kaushalmahi12/OpenSearch that referenced this pull request Sep 12, 2023
…pensearch-project#9426)

Benchmarks show that ArrayList performs better than TreeSet. Added a comment
on where to find the results and benchmark.

Signed-off-by: Kiran Reddy <kkreddy@amazon.com>
Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com>
brusic pushed a commit to brusic/OpenSearch that referenced this pull request Sep 25, 2023
…pensearch-project#9426)

Benchmarks show that ArrayList performs better than TreeSet. Added a comment
on where to find the results and benchmark.

Signed-off-by: Kiran Reddy <kkreddy@amazon.com>
Signed-off-by: Ivan Brusic <ivan.brusic@flocksafety.com>
shiv0408 pushed a commit to Gaurav614/OpenSearch that referenced this pull request Apr 25, 2024
…pensearch-project#9426)

Benchmarks show that ArrayList performs better than TreeSet. Added a comment
on where to find the results and benchmark.

Signed-off-by: Kiran Reddy <kkreddy@amazon.com>
Signed-off-by: Shivansh Arora <hishiv@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport 2.x Backport to 2.x branch skip-changelog
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Improve BinaryFieldMapper.CustomBinaryDocValuesField.bytes()
4 participants