Provide faster feedback loop for assessing performance of sequencer #1314
Assignees
Labels
sequencer
pertaining to the astria-sequencer crate
Comments
This was referenced Aug 1, 2024
github-merge-queue bot
pushed a commit
that referenced
this issue
Aug 21, 2024
## Summary
Addition of a new benchmark target to assess the performance of the
prepare_proposal method in sequencer's `App`.
## Background
Previous perf work has indicated this is a bottleneck. However, making
that determination was done via spamoor which is slightly convoluted to
run. Before working to improve the performance, we want to have a faster
feedback loop on the effects of updates, hence the need for a benchmark
which is easy to run and which isolates the slow function.
## Changes
- Added benchmark to `app` module. Currently this has only one case: a
mempool filled with transactions containing exclusively transfers. This
matches the shape of the data being sent when using spamoor.
- Added `benchmark` feature to enable sharing some of the existing test
utils.
## Testing
This is a new test.
Example of running `cargo bench --features=benchmark -qp
astria-sequencer app` on my Ryzen 7900X:
```
Timer precision: 10 ns
astria_sequencer fastest │ slowest │ median │ mean │ samples │ iters
╰─ app │ │ │ │ │
╰─ benchmarks │ │ │ │ │
╰─ execute_transactions_prepare_proposal 11.63 s │ 14.68 s │ 12.74 s │ 12.88 s │ 10 │ 10
```
Since rebasing after #1317 has merged, the same run shows (as expected)
a slowdown:
```
astria_sequencer fastest │ slowest │ median │ mean │ samples │ iters
╰─ app │ │ │ │ │
╰─ benchmarks │ │ │ │ │
╰─ execute_transactions_prepare_proposal 14.49 s │ 17 s │ 16.52 s │ 15.98 s │ 8 │ 8
```
## Related Issues
Closes #1314.
Fraser999
added a commit
to Fraser999/astria
that referenced
this issue
Aug 23, 2024
…org#1337) ## Summary Addition of a new benchmark target to assess the performance of the prepare_proposal method in sequencer's `App`. ## Background Previous perf work has indicated this is a bottleneck. However, making that determination was done via spamoor which is slightly convoluted to run. Before working to improve the performance, we want to have a faster feedback loop on the effects of updates, hence the need for a benchmark which is easy to run and which isolates the slow function. ## Changes - Added benchmark to `app` module. Currently this has only one case: a mempool filled with transactions containing exclusively transfers. This matches the shape of the data being sent when using spamoor. - Added `benchmark` feature to enable sharing some of the existing test utils. ## Testing This is a new test. Example of running `cargo bench --features=benchmark -qp astria-sequencer app` on my Ryzen 7900X: ``` Timer precision: 10 ns astria_sequencer fastest │ slowest │ median │ mean │ samples │ iters ╰─ app │ │ │ │ │ ╰─ benchmarks │ │ │ │ │ ╰─ execute_transactions_prepare_proposal 11.63 s │ 14.68 s │ 12.74 s │ 12.88 s │ 10 │ 10 ``` Since rebasing after astriaorg#1317 has merged, the same run shows (as expected) a slowdown: ``` astria_sequencer fastest │ slowest │ median │ mean │ samples │ iters ╰─ app │ │ │ │ │ ╰─ benchmarks │ │ │ │ │ ╰─ execute_transactions_prepare_proposal 14.49 s │ 17 s │ 16.52 s │ 15.98 s │ 8 │ 8 ``` ## Related Issues Closes astriaorg#1314.
Fraser999
added a commit
to Fraser999/astria
that referenced
this issue
Aug 25, 2024
…org#1337) ## Summary Addition of a new benchmark target to assess the performance of the prepare_proposal method in sequencer's `App`. ## Background Previous perf work has indicated this is a bottleneck. However, making that determination was done via spamoor which is slightly convoluted to run. Before working to improve the performance, we want to have a faster feedback loop on the effects of updates, hence the need for a benchmark which is easy to run and which isolates the slow function. ## Changes - Added benchmark to `app` module. Currently this has only one case: a mempool filled with transactions containing exclusively transfers. This matches the shape of the data being sent when using spamoor. - Added `benchmark` feature to enable sharing some of the existing test utils. ## Testing This is a new test. Example of running `cargo bench --features=benchmark -qp astria-sequencer app` on my Ryzen 7900X: ``` Timer precision: 10 ns astria_sequencer fastest │ slowest │ median │ mean │ samples │ iters ╰─ app │ │ │ │ │ ╰─ benchmarks │ │ │ │ │ ╰─ execute_transactions_prepare_proposal 11.63 s │ 14.68 s │ 12.74 s │ 12.88 s │ 10 │ 10 ``` Since rebasing after astriaorg#1317 has merged, the same run shows (as expected) a slowdown: ``` astria_sequencer fastest │ slowest │ median │ mean │ samples │ iters ╰─ app │ │ │ │ │ ╰─ benchmarks │ │ │ │ │ ╰─ execute_transactions_prepare_proposal 14.49 s │ 17 s │ 16.52 s │ 15.98 s │ 8 │ 8 ``` ## Related Issues Closes astriaorg#1314.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Investigation into sequencer slowdown under load is pointing towards block execution being a major contributor.
Currently this has been discovered via running spamoor, which requires running the composer and cometbft too. We would benefit from a more targeted solution, e.g. a benchmark suite for the
Sequencerstruct or even theApp.┆Issue Number: ENG-660
The text was updated successfully, but these errors were encountered: