Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

perf(levm): add more benchmarks to the suite #1803

Merged
merged 28 commits into from
Jan 29, 2025

Conversation

dsocolobsky
Copy link
Contributor

@dsocolobsky dsocolobsky commented Jan 24, 2025

Motivation

We need more complex contracts to benchmark our implementation against other VMs.

Description

  • Added 5 new benchmarks: FactorialRecursive, ManyHashes, ERC20Mint, ERC20Approval and ERC20Transfer.
  • Benchmarks contracts are now compiled from Solidity code in our codebase, instead of being hardcoded bytecode.
  • Simplified crates/vm/levm/Makefile since it had a lot of repetition and hardcoded values, mainly avoid copy-paste mistakes and make it easier to modify.
  • Benchmarks are now run from a single crates/vm/levm/bench/revm_comparison/src/benchmark.rs file with parameters, instead of two files per benchmark case.
  • In crates/vm/levm/bench/revm_comparison/lib.rs:run_with_levmwe now use VM::transact() instead of directly VM::execute(callframe). This might modify slightly our benchmarks because we now take into account prepare_execution, however this fixes the benchmarks not working with contracts that call themselves recursively (like FibonacciRecursive)
  • I lowered the repetitions for each benchmark from 100k to 10k, from what I've seen the results are the same and it runs much faster, we can change later if we need more precise measurements.

Further Steps

  • We can think of new benchmarks.
  • Compare if the results from LEVM and REVM are the same and fail if not.
  • Analyze the flamegraphs to see possible bottlenecks and add to Notion the findings.

Notes
Right now ci_bench_levm.yaml for the job benchmark-main is unchanged, because else it wouldn't pass the CI since the new benchmarks aren't in main. I think the best is to merge like this and then change the CI again in main in another small PR?

Closes #1623

Copy link

github-actions bot commented Jan 27, 2025

| File                                                                                  | Lines | Diff |
+---------------------------------------------------------------------------------------+-------+------+
| /home/runner/work/ethrex/ethrex/crates/vm/levm/bench/revm_comparison/src/benchmark.rs | 34    | +34  |
+---------------------------------------------------------------------------------------+-------+------+
| /home/runner/work/ethrex/ethrex/crates/vm/levm/bench/revm_comparison/src/compile.rs   | 63    | +63  |
+---------------------------------------------------------------------------------------+-------+------+
| /home/runner/work/ethrex/ethrex/crates/vm/levm/bench/revm_comparison/src/lib.rs       | 85    | +27  |
+---------------------------------------------------------------------------------------+-------+------+

Total lines added: +124
Total lines removed: 0
Total lines changed: 124

Copy link

github-actions bot commented Jan 27, 2025

Summary: 6475/6475 (100.00%)

Prague: 2373/2373 (100.00%)
Cancun: 3579/3579 (100.00%)
Shanghai: 221/221 (100.00%)
Paris: 62/62 (100.00%)
London: 39/39 (100.00%)
Berlin: 35/35 (100.00%)
Istanbul: 34/34 (100.00%)
Constantinople: 66/66 (100.00%)
Byzantium: 33/33 (100.00%)
Homestead: 17/17 (100.00%)
Frontier: 16/16 (100.00%)

@dsocolobsky dsocolobsky changed the title perf(levm): WIP Add more benchmarks to the suite perf(levm): add more benchmarks to the suite Jan 27, 2025
@dsocolobsky dsocolobsky force-pushed the levm/benchs/evm-bench-integration branch from 3a468fd to 86343bc Compare January 27, 2025 19:03
Copy link

github-actions bot commented Jan 27, 2025

Benchmark Results Comparison

PR Results

Benchmark Results: Factorial

Command Mean [ms] Min [ms] Max [ms] Relative
revm_Factorial 237.3 ± 2.2 234.2 241.4 1.00
levm_Factorial 841.0 ± 3.0 835.4 844.2 3.54 ± 0.04

Benchmark Results: Factorial - Recursive

Command Mean [s] Min [s] Max [s] Relative
revm_FactorialRecursive 1.452 ± 0.103 1.318 1.593 1.00
levm_FactorialRecursive 15.372 ± 0.126 15.256 15.523 10.58 ± 0.75

Benchmark Results: Fibonacci

Command Mean [ms] Min [ms] Max [ms] Relative
revm_Fibonacci 211.5 ± 1.3 209.0 213.1 1.00
levm_Fibonacci 837.1 ± 3.1 831.5 841.5 3.96 ± 0.03

Benchmark Results: ManyHashes

Command Mean [ms] Min [ms] Max [ms] Relative
revm_ManyHashes 8.7 ± 0.1 8.5 8.8 1.00
levm_ManyHashes 17.5 ± 0.2 17.3 17.8 2.02 ± 0.03

Benchmark Results: ERC20 - Transfer

Command Mean [ms] Min [ms] Max [ms] Relative
revm_ERC20Transfer 249.0 ± 2.5 246.9 255.7 1.00
levm_ERC20Transfer 3239.4 ± 173.1 3165.5 3728.1 13.01 ± 0.71

Benchmark Results: ERC20 - Mint

Command Mean [ms] Min [ms] Max [ms] Relative
revm_ERC20Mint 141.0 ± 1.2 140.0 144.2 1.00
levm_ERC20Mint 1676.4 ± 13.1 1660.5 1694.7 11.89 ± 0.14

Benchmark Results: ERC20 - Approval

Command Mean [s] Min [s] Max [s] Relative
revm_ERC20Approval 1.037 ± 0.006 1.026 1.046 1.00
levm_ERC20Approval 11.332 ± 0.095 11.188 11.432 10.93 ± 0.11

Main Results

Benchmark Results: Factorial

Command Mean [s] Min [s] Max [s] Relative
revm_factorial 7.373 ± 0.013 7.342 7.382 1.00
levm_factorial 26.197 ± 0.500 25.741 27.564 3.55 ± 0.07

Benchmark Results: Fibonacci

Command Mean [s] Min [s] Max [s] Relative
revm_fibonacci 8.212 ± 1.085 7.484 10.280 1.00
levm_fibonacci 24.232 ± 1.955 23.570 29.795 2.95 ± 0.46

@dsocolobsky dsocolobsky force-pushed the levm/benchs/evm-bench-integration branch 3 times, most recently from d58afee to c9c6fa7 Compare January 27, 2025 19:56
@dsocolobsky dsocolobsky force-pushed the levm/benchs/evm-bench-integration branch from c9c6fa7 to ad8d80b Compare January 27, 2025 20:10
@dsocolobsky dsocolobsky marked this pull request as ready for review January 27, 2025 20:23
@dsocolobsky dsocolobsky requested a review from a team as a code owner January 27, 2025 20:23
@lima-limon-inc lima-limon-inc added the levm Lambda EVM implementation label Jan 28, 2025
Copy link
Contributor

@fborello-lambda fborello-lambda left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Amazing.

The PR's note says:

I think the best is to merge like this and then change the CI again in main in another small PR?

Yes, i would do a small PR after this one is merged.

Now we run more preparation code for the VM; however without this change it wasn't executing correctly contracts where we call ourselves via STATICCALL or similar.
@ilitteri ilitteri added this pull request to the merge queue Jan 29, 2025
Merged via the queue into main with commit 6a29288 Jan 29, 2025
33 checks passed
@ilitteri ilitteri deleted the levm/benchs/evm-bench-integration branch January 29, 2025 15:33
github-merge-queue bot pushed a commit that referenced this pull request Jan 29, 2025
**Motivation**

Since the recently #1803 PR we now have new benchmarks. We need to
modify the CI to make the comparison.
I couldn't do that in the previous PR because since the new benchmarks
were not in main yet, the CI would always fail.

**Description**
I will fix this in the next benchmark PR so we don't have to create 2
PRs for each new benchmark.
<!-- A clear and concise general description of the changes this PR
introduces -->

<!-- Link to issues: Resolves #111, Resolves #222 -->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
levm Lambda EVM implementation
Projects
None yet
Development

Successfully merging this pull request may close these issues.

LEVM: Add more bechmarks to the suite
4 participants