Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement microbenchmark for FileCache #6610

Merged
merged 1 commit into from
Mar 29, 2023

Conversation

andrross
Copy link
Member

@andrross andrross commented Mar 9, 2023

The current implementation of the FileCache uses some hand-rolled data structures that would be nice to replace with library implementations. This benchmark will be useful to compare the performance of any future replacements.

Sample results (AWS EC2 Ubuntu machine with 32 vCPUs and 64GB of RAM):

Benchmark                   (concurrencyLevel)  (maximumNumberOfEntries)   Mode  Cnt      Score   Error   Units
FileCacheBenchmark.get                       1                     65536  thrpt        2243.092          ops/ms
FileCacheBenchmark.get                       1                   1048576  thrpt         950.818          ops/ms
FileCacheBenchmark.get                       8                     65536  thrpt        5651.150          ops/ms
FileCacheBenchmark.get                       8                   1048576  thrpt        2831.012          ops/ms
FileCacheBenchmark.put                       1                     65536  thrpt        2206.027          ops/ms
FileCacheBenchmark.put                       1                   1048576  thrpt         921.248          ops/ms
FileCacheBenchmark.put                       8                     65536  thrpt        4421.122          ops/ms
FileCacheBenchmark.put                       8                   1048576  thrpt        2624.550          ops/ms
FileCacheBenchmark.remove                    1                     65536  thrpt       12387.999          ops/ms
FileCacheBenchmark.remove                    1                   1048576  thrpt        6324.643          ops/ms
FileCacheBenchmark.remove                    8                     65536  thrpt       22161.031          ops/ms
FileCacheBenchmark.remove                    8                   1048576  thrpt       14826.586          ops/ms
FileCacheBenchmark.replace                   1                     65536  thrpt        2146.572          ops/ms
FileCacheBenchmark.replace                   1                   1048576  thrpt         947.612          ops/ms
FileCacheBenchmark.replace                   8                     65536  thrpt        4405.339          ops/ms
FileCacheBenchmark.replace                   8                   1048576  thrpt        2707.204          ops/ms

Check List

  • New functionality includes testing.
    • All tests pass
  • New functionality has been documented.
    • New functionality has javadoc added
  • Commits are signed per the DCO using --signoff
  • Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@github-actions
Copy link
Contributor

github-actions bot commented Mar 9, 2023

Gradle Check (Jenkins) Run Completed with:

  • RESULT: UNSTABLE ❕
  • TEST FAILURES:
      1 org.opensearch.search.SearchWeightedRoutingIT.testSearchAggregationWithNetworkDisruption_FailOpenEnabled

@codecov-commenter
Copy link

codecov-commenter commented Mar 9, 2023

Codecov Report

Merging #6610 (1d9aa41) into main (13d336c) will decrease coverage by 0.08%.
The diff coverage is 0.00%.

📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

@@             Coverage Diff              @@
##               main    #6610      +/-   ##
============================================
- Coverage     70.79%   70.72%   -0.08%     
+ Complexity    59121    59107      -14     
============================================
  Files          4804     4805       +1     
  Lines        283141   283173      +32     
  Branches      40815    40816       +1     
============================================
- Hits         200462   200268     -194     
- Misses        66229    66492     +263     
+ Partials      16450    16413      -37     
Impacted Files Coverage Δ
...ark/store/remote/filecache/FileCacheBenchmark.java 0.00% <0.00%> (ø)

... and 499 files with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.


@Benchmark
public void get(CacheParameters parameters) {
parameters.fileCache.get(randomKeyInCache(parameters));
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Curious if the benchmark should use blackhole or return the value, the call could be eliminated since the returned value is not used, thoughts?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, good call.

My guess is the call would not be eliminated because the JVM doesn't know if there are no side effects to invoking the get() method. I'm certainly no expert here but as a general practice I think the return values should be blackholed.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, that should make it:

    public Object get(CacheParameters parameters) {
        return parameters.fileCache.get(randomKeyInCache(parameters));
    }

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I went with:

@Benchmark
public void get(CacheParameters parameters, Blackhole blackhole) {
    blackhole.consume(parameters.fileCache.get(randomKeyInCache(parameters)));
}

Is that not the right way to use the blackhole here?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My apologies @andrross, probably the comment went into outdated code version, both approaches are correct, thank you.

The current implementation of the FileCache uses some hand-rolled data
structures that would be nice to replace with library implementations.
This benchmark will be useful to compare the performance of any future
replacements.

Sample results (AWS EC2 Ubuntu machine with 32 vCPUs and 64GB of RAM):

```
Benchmark                   (concurrencyLevel)  (maximumNumberOfEntries)   Mode  Cnt      Score   Error   Units
FileCacheBenchmark.get                       1                     65536  thrpt        2243.092          ops/ms
FileCacheBenchmark.get                       1                   1048576  thrpt         950.818          ops/ms
FileCacheBenchmark.get                       8                     65536  thrpt        5651.150          ops/ms
FileCacheBenchmark.get                       8                   1048576  thrpt        2831.012          ops/ms
FileCacheBenchmark.put                       1                     65536  thrpt        2206.027          ops/ms
FileCacheBenchmark.put                       1                   1048576  thrpt         921.248          ops/ms
FileCacheBenchmark.put                       8                     65536  thrpt        4421.122          ops/ms
FileCacheBenchmark.put                       8                   1048576  thrpt        2624.550          ops/ms
FileCacheBenchmark.remove                    1                     65536  thrpt       12387.999          ops/ms
FileCacheBenchmark.remove                    1                   1048576  thrpt        6324.643          ops/ms
FileCacheBenchmark.remove                    8                     65536  thrpt       22161.031          ops/ms
FileCacheBenchmark.remove                    8                   1048576  thrpt       14826.586          ops/ms
FileCacheBenchmark.replace                   1                     65536  thrpt        2146.572          ops/ms
FileCacheBenchmark.replace                   1                   1048576  thrpt         947.612          ops/ms
FileCacheBenchmark.replace                   8                     65536  thrpt        4405.339          ops/ms
FileCacheBenchmark.replace                   8                   1048576  thrpt        2707.204          ops/ms
```

Signed-off-by: Andrew Ross <andrross@amazon.com>
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

  • RESULT: UNSTABLE ❕
  • TEST FAILURES:
      1 org.opensearch.snapshots.DedicatedClusterSnapshotRestoreIT.testIndexDeletionDuringSnapshotCreationInQueue

@kotwanikunal
Copy link
Member

Whitesource fix: #6629

@kotwanikunal kotwanikunal merged commit 5eed61e into opensearch-project:main Mar 29, 2023
opensearch-trigger-bot bot pushed a commit that referenced this pull request Mar 29, 2023
The current implementation of the FileCache uses some hand-rolled data
structures that would be nice to replace with library implementations.
This benchmark will be useful to compare the performance of any future
replacements.

Sample results (AWS EC2 Ubuntu machine with 32 vCPUs and 64GB of RAM):

```
Benchmark                   (concurrencyLevel)  (maximumNumberOfEntries)   Mode  Cnt      Score   Error   Units
FileCacheBenchmark.get                       1                     65536  thrpt        2243.092          ops/ms
FileCacheBenchmark.get                       1                   1048576  thrpt         950.818          ops/ms
FileCacheBenchmark.get                       8                     65536  thrpt        5651.150          ops/ms
FileCacheBenchmark.get                       8                   1048576  thrpt        2831.012          ops/ms
FileCacheBenchmark.put                       1                     65536  thrpt        2206.027          ops/ms
FileCacheBenchmark.put                       1                   1048576  thrpt         921.248          ops/ms
FileCacheBenchmark.put                       8                     65536  thrpt        4421.122          ops/ms
FileCacheBenchmark.put                       8                   1048576  thrpt        2624.550          ops/ms
FileCacheBenchmark.remove                    1                     65536  thrpt       12387.999          ops/ms
FileCacheBenchmark.remove                    1                   1048576  thrpt        6324.643          ops/ms
FileCacheBenchmark.remove                    8                     65536  thrpt       22161.031          ops/ms
FileCacheBenchmark.remove                    8                   1048576  thrpt       14826.586          ops/ms
FileCacheBenchmark.replace                   1                     65536  thrpt        2146.572          ops/ms
FileCacheBenchmark.replace                   1                   1048576  thrpt         947.612          ops/ms
FileCacheBenchmark.replace                   8                     65536  thrpt        4405.339          ops/ms
FileCacheBenchmark.replace                   8                   1048576  thrpt        2707.204          ops/ms
```

Signed-off-by: Andrew Ross <andrross@amazon.com>
(cherry picked from commit 5eed61e)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
@andrross andrross deleted the cache-benchmark branch March 29, 2023 23:32
kotwanikunal pushed a commit that referenced this pull request Mar 30, 2023
The current implementation of the FileCache uses some hand-rolled data
structures that would be nice to replace with library implementations.
This benchmark will be useful to compare the performance of any future
replacements.

Sample results (AWS EC2 Ubuntu machine with 32 vCPUs and 64GB of RAM):

```
Benchmark                   (concurrencyLevel)  (maximumNumberOfEntries)   Mode  Cnt      Score   Error   Units
FileCacheBenchmark.get                       1                     65536  thrpt        2243.092          ops/ms
FileCacheBenchmark.get                       1                   1048576  thrpt         950.818          ops/ms
FileCacheBenchmark.get                       8                     65536  thrpt        5651.150          ops/ms
FileCacheBenchmark.get                       8                   1048576  thrpt        2831.012          ops/ms
FileCacheBenchmark.put                       1                     65536  thrpt        2206.027          ops/ms
FileCacheBenchmark.put                       1                   1048576  thrpt         921.248          ops/ms
FileCacheBenchmark.put                       8                     65536  thrpt        4421.122          ops/ms
FileCacheBenchmark.put                       8                   1048576  thrpt        2624.550          ops/ms
FileCacheBenchmark.remove                    1                     65536  thrpt       12387.999          ops/ms
FileCacheBenchmark.remove                    1                   1048576  thrpt        6324.643          ops/ms
FileCacheBenchmark.remove                    8                     65536  thrpt       22161.031          ops/ms
FileCacheBenchmark.remove                    8                   1048576  thrpt       14826.586          ops/ms
FileCacheBenchmark.replace                   1                     65536  thrpt        2146.572          ops/ms
FileCacheBenchmark.replace                   1                   1048576  thrpt         947.612          ops/ms
FileCacheBenchmark.replace                   8                     65536  thrpt        4405.339          ops/ms
FileCacheBenchmark.replace                   8                   1048576  thrpt        2707.204          ops/ms
```


(cherry picked from commit 5eed61e)

Signed-off-by: Andrew Ross <andrross@amazon.com>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
mitrofmep pushed a commit to mitrofmep/OpenSearch that referenced this pull request Apr 5, 2023
The current implementation of the FileCache uses some hand-rolled data
structures that would be nice to replace with library implementations.
This benchmark will be useful to compare the performance of any future
replacements.

Sample results (AWS EC2 Ubuntu machine with 32 vCPUs and 64GB of RAM):

```
Benchmark                   (concurrencyLevel)  (maximumNumberOfEntries)   Mode  Cnt      Score   Error   Units
FileCacheBenchmark.get                       1                     65536  thrpt        2243.092          ops/ms
FileCacheBenchmark.get                       1                   1048576  thrpt         950.818          ops/ms
FileCacheBenchmark.get                       8                     65536  thrpt        5651.150          ops/ms
FileCacheBenchmark.get                       8                   1048576  thrpt        2831.012          ops/ms
FileCacheBenchmark.put                       1                     65536  thrpt        2206.027          ops/ms
FileCacheBenchmark.put                       1                   1048576  thrpt         921.248          ops/ms
FileCacheBenchmark.put                       8                     65536  thrpt        4421.122          ops/ms
FileCacheBenchmark.put                       8                   1048576  thrpt        2624.550          ops/ms
FileCacheBenchmark.remove                    1                     65536  thrpt       12387.999          ops/ms
FileCacheBenchmark.remove                    1                   1048576  thrpt        6324.643          ops/ms
FileCacheBenchmark.remove                    8                     65536  thrpt       22161.031          ops/ms
FileCacheBenchmark.remove                    8                   1048576  thrpt       14826.586          ops/ms
FileCacheBenchmark.replace                   1                     65536  thrpt        2146.572          ops/ms
FileCacheBenchmark.replace                   1                   1048576  thrpt         947.612          ops/ms
FileCacheBenchmark.replace                   8                     65536  thrpt        4405.339          ops/ms
FileCacheBenchmark.replace                   8                   1048576  thrpt        2707.204          ops/ms
```

Signed-off-by: Andrew Ross <andrross@amazon.com>
Signed-off-by: Valentin Mitrofanov <mitrofmep@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport 2.x Backport to 2.x branch skip-changelog
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants