Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add differential fuzzing #833

Merged
merged 4 commits into from
Dec 4, 2023
Merged

Add differential fuzzing #833

merged 4 commits into from
Dec 4, 2023

Conversation

Robbepop
Copy link
Member

@Robbepop Robbepop commented Dec 4, 2023

Closes #419.

@paritytech-cicd-pr
Copy link

BENCHMARKS

NATIVEWASMTIME
BENCHMARKMASTERPRDIFFMASTERPRDIFFWASMTIME OVERHEAD
execute/
br_table
1.48ms 1.48ms ⚪ 0.05% 1.30ms 1.30ms ⚪ 0.29% 🟢 -12%
execute/
call/host/1
47.41µs 47.25µs ⚪ -0.66% 63.73µs 68.94µs 🔴 8.30% 🟢 46%
execute/
call/rec
167.79µs 168.54µs ⚪ 1.27% 347.93µs 347.90µs ⚪ -0.05% 🔴 106%
execute/
count_until
6.67ms 6.64ms ⚪ -0.71% 7.51ms 7.51ms ⚪ -0.02% 🟢 13%
execute/
divrem
6.23ms 6.23ms ⚪ -0.05% 7.77ms 6.96ms 🟢 -10.51% 🟢 12%
execute/
factorial/iter
247.82µs 242.49µs 🟢 -2.12% 317.48µs 318.90µs ⚪ 0.38% 🟢 32%
execute/
factorial/rec
695.48µs 686.54µs 🟢 -1.33% 1.29ms 1.30ms ⚪ 0.80% 🟡 89%
execute/
fibonacci/iter
1.29ms 1.30ms ⚪ 0.60% 1.26ms 1.27ms ⚪ 0.72% 🟢 -2%
execute/
fibonacci/rec
6.22ms 6.20ms ⚪ -0.40% 13.29ms 13.36ms ⚪ 0.49% 🔴 116%
execute/
fibonacci/tail
1.39ms 1.51ms 🔴 8.83% 3.82ms 3.83ms ⚪ 0.31% 🔴 153%
execute/
fuse
7.34ms 7.33ms ⚪ 0.00% 11.42ms 11.41ms ⚪ 0.00% 🟡 56%
execute/
global/bump
1.32ms 1.33ms ⚪ 1.22% 1.54ms 1.54ms ⚪ -0.02% 🟢 16%
execute/
global/get_const
699.31µs 700.76µs ⚪ 0.24% 750.54µs 750.51µs ⚪ 0.00% 🟢 7%
execute/
is_even/rec
1.10ms 1.11ms ⚪ 0.66% 2.19ms 2.21ms ⚪ 1.08% 🟡 100%
execute/
memory/fill_bytes
1.08ms 1.08ms ⚪ 0.08% 1.32ms 1.34ms 🔴 1.33% 🟢 24%
execute/
memory/sum_bytes
1.09ms 1.09ms ⚪ 0.23% 1.25ms 1.26ms ⚪ 0.73% 🟢 15%
execute/
memory/vec_add
2.96ms 2.96ms ⚪ -0.05% 3.62ms 3.62ms ⚪ 0.02% 🟢 22%
execute/
recursive_scan
189.77µs 190.78µs ⚪ 0.07% 379.53µs 378.32µs ⚪ -0.13% 🟡 98%
execute/
recursive_trap
15.80µs 15.44µs 🟢 -2.28% 34.32µs 34.43µs ⚪ 0.45% 🔴 123%
execute/
regex_redux
590.97µs 601.79µs ⚪ 1.40% 1.09ms 1.10ms ⚪ 0.73% 🟡 82%
execute/
rev_complement
443.34µs 444.09µs ⚪ 0.08% 672.23µs 669.84µs ⚪ -0.53% 🟡 51%
execute/
tiny_keccak
355.39µs 350.24µs ⚪ -0.77% 381.44µs 380.90µs ⚪ -0.23% 🟢 9%
execute/
trunc_f2i
615.85µs 616.89µs ⚪ 0.13% 961.16µs 961.10µs ⚪ -0.03% 🟡 56%
instantiate/
wasm_kernel
54.75µs 56.81µs ⚪ 0.38% 56.71µs 56.64µs ⚪ -0.50% 🟢 -0%
overhead/
call/typed/0
1.23ms 1.22ms ⚪ -0.13% 776.93µs 774.72µs ⚪ -0.24% 🟢 -37%
overhead/
call/typed/16
1.61ms 1.60ms ⚪ -1.02% 1.84ms 1.83ms ⚪ -0.47% 🟢 15%
overhead/
call/untyped/0
1.62ms 1.61ms ⚪ 0.11% 1.22ms 1.21ms ⚪ -0.59% 🟢 -25%
overhead/
call/untyped/16
2.50ms 2.49ms ⚪ -0.41% 4.03ms 4.02ms ⚪ -0.16% 🟡 61%
translate/
bz2/checked/default
1.38ms 1.38ms ⚪ 0.16% 2.55ms 2.54ms ⚪ -0.71% 🟡 83%
translate/
bz2/checked/fuel
1.42ms 1.42ms ⚪ 0.10% 2.70ms 2.69ms ⚪ -0.03% 🟡 89%
translate/
bz2/unchecked/default
1.12ms 1.13ms ⚪ 0.50% 1.88ms 1.88ms ⚪ 0.49% 🟡 67%
translate/
bz2/unchecked/fuel
1.16ms 1.16ms ⚪ 0.24% 2.02ms 2.02ms ⚪ -0.04% 🟡 74%
translate/
erc1155/checked/default
281.86µs 283.00µs ⚪ 0.19% 488.70µs 488.03µs ⚪ 0.00% 🟡 72%
translate/
erc1155/checked/fuel
300.99µs 300.60µs ⚪ -0.16% 516.38µs 516.50µs ⚪ 0.03% 🟡 72%
translate/
erc1155/unchecked/default
233.91µs 232.10µs ⚪ -0.34% 372.66µs 375.38µs ⚪ 0.89% 🟡 62%
translate/
erc1155/unchecked/fuel
251.39µs 249.10µs ⚪ -0.87% 398.90µs 399.50µs ⚪ 0.21% 🟡 60%
translate/
erc20/checked/default
136.83µs 137.36µs ⚪ 0.34% 235.85µs 235.38µs ⚪ -0.22% 🟡 71%
translate/
erc20/checked/fuel
144.87µs 145.21µs ⚪ 0.27% 244.81µs 246.35µs ⚪ 0.55% 🟡 70%
translate/
erc20/unchecked/default
113.21µs 113.92µs ⚪ 0.69% 178.89µs 180.76µs ⚪ 1.26% 🟡 59%
translate/
erc20/unchecked/fuel
119.92µs 120.94µs ⚪ 1.00% 188.51µs 190.40µs ⚪ 0.66% 🟡 57%
translate/
erc721/checked/default
195.29µs 194.34µs ⚪ -0.50% 341.97µs 339.61µs ⚪ -0.74% 🟡 75%
translate/
erc721/checked/fuel
203.48µs 203.62µs ⚪ -0.20% 356.41µs 354.65µs ⚪ -0.41% 🟡 74%
translate/
erc721/unchecked/default
159.22µs 159.56µs ⚪ 0.36% 263.25µs 257.35µs 🟢 -2.16% 🟡 61%
translate/
erc721/unchecked/fuel
167.50µs 167.97µs ⚪ 0.20% 270.02µs 270.62µs ⚪ 0.00% 🟡 61%
translate/
pulldown_cmark/checked/default
3.76ms 3.83ms 🔴 1.84% 6.69ms 6.69ms ⚪ -0.02% 🟡 75%
translate/
pulldown_cmark/checked/fuel
3.87ms 3.92ms 🔴 1.48% 7.00ms 7.00ms ⚪ -0.10% 🟡 78%
translate/
pulldown_cmark/unchecked/default
3.14ms 3.19ms 🔴 1.59% 5.09ms 5.07ms ⚪ -0.30% 🟡 59%
translate/
pulldown_cmark/unchecked/fuel
3.21ms 3.26ms 🔴 1.49% 5.35ms 5.36ms ⚪ 0.44% 🟡 64%
translate/
spidermonkey/checked/default
0.00ns 0.00ns ⚪ 0.52% 0.00ns 0.00ns ⚪ -0.05% 🟢 0%
translate/
spidermonkey/checked/fuel
0.00ns 0.00ns ⚪ 0.90% 0.00ns 0.00ns ⚪ 0.11% 🟢 0%
translate/
spidermonkey/unchecked/default
72.08ms 72.52ms ⚪ 0.88% 0.00ns 0.00ns ⚪ 0.26% 🟢 -100%
translate/
spidermonkey/unchecked/fuel
0.00ns 0.00ns ⚪ 0.66% 0.00ns 0.00ns ⚪ 0.44% 🟢 0%
translate/
wasm_kernel/checked/default
5.05ms 5.04ms ⚪ -0.29% 8.95ms 8.96ms ⚪ 0.06% 🟡 78%
translate/
wasm_kernel/checked/fuel
5.18ms 5.17ms ⚪ -0.12% 9.34ms 9.36ms ⚪ 0.30% 🟡 81%
translate/
wasm_kernel/unchecked/default
4.11ms 4.12ms ⚪ 0.05% 6.72ms 6.78ms ⚪ 0.81% 🟡 65%
translate/
wasm_kernel/unchecked/fuel
4.24ms 4.24ms ⚪ -0.15% 7.08ms 7.13ms ⚪ 0.49% 🟡 68%

Link to pipeline

@codecov-commenter
Copy link

Codecov Report

Attention: 33 lines in your changes are missing coverage. Please review.

Comparison is base (41ca061) 80.93% compared to head (9709598) 80.94%.

Files Patch % Lines
fuzz/fuzz_targets/utils.rs 0.00% 33 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##           master     #833   +/-   ##
=======================================
  Coverage   80.93%   80.94%           
=======================================
  Files         257      257           
  Lines       22461    22461           
=======================================
+ Hits        18179    18180    +1     
+ Misses       4282     4281    -1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@Robbepop Robbepop merged commit 4a8725e into master Dec 4, 2023
20 checks passed
@Robbepop Robbepop deleted the rf-add-differential-fuzzing branch December 4, 2023 18:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Implement fuzz testing for Wasm execution
3 participants