-
Notifications
You must be signed in to change notification settings - Fork 388
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(storage): make benchmark more robust on stalls #7113
feat(storage): make benchmark more robust on stalls #7113
Conversation
Change the benchmark to use a single queue (LIFO) with the set of objects to download, and have all the threads service this queue. Previously we separated the objects in groups and assigned them to threads. The previous approach wastes threads if one (or a small fraction) of the downloads gets stalled. I think the new approach better models a well-designed application.
Google Cloud Build Logs
ℹ️ NOTE: Kokoro logs are linked from "Details" below. |
Codecov Report
@@ Coverage Diff @@
## main #7113 +/- ##
=======================================
Coverage 94.49% 94.49%
=======================================
Files 1309 1309
Lines 113025 113035 +10
=======================================
+ Hits 106801 106812 +11
+ Misses 6224 6223 -1
Continue to review full report at Codecov.
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reviewable status: 0 of 1 files reviewed, all discussions resolved (waiting on @coryan)
google/cloud/storage/benchmarks/aggregate_throughput_benchmark.cc, line 257 at r1 (raw file):
Quoted 4 lines of code…
std::transform(configs.begin(), configs.end(), tasks.begin(), [&task](TaskConfig const& c) { return std::async(std::launch::async, task, std::cref(c)); });
if you did something like this, you can get rid of thetask
lambda:
std::transform(configs.begin(), configs.end(), tasks.begin(),
[&iteration](TaskConfig const& c) {
return std::async(std::launch::async, &Iteration::DownloadTask, iteration, std::cref(c));
});
The aesthetic judgment is up to you. I don't have a preference.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reviewable status: 0 of 1 files reviewed, all discussions resolved (waiting on @coryan)
google/cloud/storage/benchmarks/aggregate_throughput_benchmark.cc, line 257 at r1 (raw file):
Previously, dbolduc (Darren Bolduc) wrote…
std::transform(configs.begin(), configs.end(), tasks.begin(), [&task](TaskConfig const& c) { return std::async(std::launch::async, task, std::cref(c)); });
if you did something like this, you can get rid of the
task
lambda:std::transform(configs.begin(), configs.end(), tasks.begin(), [&iteration](TaskConfig const& c) { return std::async(std::launch::async, &Iteration::DownloadTask, iteration, std::cref(c)); });
The aesthetic judgment is up to you. I don't have a preference.
I am going to leave as-is, and probably change my mind later
Change the benchmark to use a single queue (LIFO) with the set of
objects to download, and have all the threads service this queue.
Previously we separated the objects in groups and assigned them to
threads. The previous approach wastes threads if one (or a small
fraction) of the downloads gets stalled. I think the new approach better
models a well-designed application.
This change is