Fuzz cairo compiled programs #1236

Juan-M-V · 2023-06-14T20:08:45Z

TITLE

Description

This PR adds a fuzzer that creates random cairo programs using different builtins, compile and run those programs.

Checklist

Linked to Github Issue
Unit tests added
Integration tests added.
This change requires new documentation.
- Documentation has been added/updated.
- CHANGELOG has been updated.

…airo-fuzzer

codecov · 2023-06-30T19:03:50Z

Codecov Report

Merging #1236 (8f75685) into main (b232d69) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1236   +/-   ##
=======================================
  Coverage   97.48%   97.48%           
=======================================
  Files          92       92           
  Lines       37718    37718           
=======================================
  Hits        36770    36770           
  Misses        948      948

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

github-actions · 2023-06-30T19:27:43Z

Benchmark Results for unmodified programs 🚀

Command	Mean [s]	Min [s]	Max [s]	Relative
`base big_factorial`	3.688 ± 0.010	3.674	3.701	1.00 ± 0.00
`head big_factorial`	3.683 ± 0.012	3.667	3.699	1.00

Command	Mean [s]	Min [s]	Max [s]	Relative
`base big_fibonacci`	3.183 ± 0.010	3.169	3.204	1.01 ± 0.01
`head big_fibonacci`	3.162 ± 0.015	3.143	3.193	1.00

Command	Mean [s]	Min [s]	Max [s]	Relative
`base blake2s_integration_benchmark`	9.053 ± 0.044	9.009	9.162	1.00
`head blake2s_integration_benchmark`	9.138 ± 0.114	9.047	9.434	1.01 ± 0.01

Command	Mean [s]	Min [s]	Max [s]	Relative
`base compare_arrays_200000`	3.224 ± 0.019	3.200	3.259	1.01 ± 0.01
`head compare_arrays_200000`	3.206 ± 0.024	3.182	3.264	1.00

Command	Mean [s]	Min [s]	Max [s]	Relative
`base dict_integration_benchmark`	2.205 ± 0.008	2.193	2.220	1.00 ± 0.00
`head dict_integration_benchmark`	2.196 ± 0.005	2.188	2.203	1.00

Command	Mean [s]	Min [s]	Max [s]	Relative
`base field_arithmetic_get_square_benchmark`	1.717 ± 0.015	1.700	1.745	1.00 ± 0.01
`head field_arithmetic_get_square_benchmark`	1.711 ± 0.009	1.703	1.727	1.00

Command	Mean [s]	Min [s]	Max [s]	Relative
`base integration_builtins`	9.246 ± 0.194	9.038	9.554	1.00 ± 0.02
`head integration_builtins`	9.240 ± 0.071	9.121	9.337	1.00

Command	Mean [s]	Min [s]	Max [s]	Relative
`base keccak_integration_benchmark`	9.440 ± 0.252	9.199	9.910	1.01 ± 0.05
`head keccak_integration_benchmark`	9.339 ± 0.344	9.138	10.302	1.00

Command	Mean [s]	Min [s]	Max [s]	Relative
`base linear_search`	3.295 ± 0.020	3.275	3.338	1.00 ± 0.01
`head linear_search`	3.292 ± 0.019	3.273	3.333	1.00

Command	Mean [s]	Min [s]	Max [s]	Relative
`base math_cmp_and_pow_integration_benchmark`	2.371 ± 0.005	2.361	2.377	1.00
`head math_cmp_and_pow_integration_benchmark`	2.390 ± 0.006	2.382	2.403	1.01 ± 0.00

Command	Mean [s]	Min [s]	Max [s]	Relative
`base math_integration_benchmark`	2.180 ± 0.010	2.166	2.196	1.00
`head math_integration_benchmark`	2.188 ± 0.004	2.182	2.193	1.00 ± 0.00

Command	Mean [s]	Min [s]	Max [s]	Relative
`base memory_integration_benchmark`	1.901 ± 0.008	1.892	1.917	1.00 ± 0.01
`head memory_integration_benchmark`	1.897 ± 0.006	1.889	1.909	1.00

Command	Mean [s]	Min [s]	Max [s]	Relative
`base operations_with_data_structures_benchmarks`	2.138 ± 0.009	2.127	2.152	1.00
`head operations_with_data_structures_benchmarks`	2.155 ± 0.005	2.147	2.163	1.01 ± 0.00

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
`base pedersen`	829.8 ± 2.3	827.2	833.7	1.00
`head pedersen`	831.9 ± 2.2	828.3	834.5	1.00 ± 0.00

Command	Mean [s]	Min [s]	Max [s]	Relative
`base poseidon_integration_benchmark`	1.519 ± 0.003	1.514	1.522	1.00 ± 0.00
`head poseidon_integration_benchmark`	1.516 ± 0.005	1.511	1.527	1.00

Command	Mean [s]	Min [s]	Max [s]	Relative
`base secp_integration_benchmark`	2.508 ± 0.008	2.500	2.529	1.00
`head secp_integration_benchmark`	2.518 ± 0.010	2.501	2.537	1.00 ± 0.01

Command	Mean [s]	Min [s]	Max [s]	Relative
`base set_integration_benchmark`	1.287 ± 0.037	1.270	1.391	1.00
`head set_integration_benchmark`	1.287 ± 0.026	1.272	1.357	1.00 ± 0.04

Command	Mean [s]	Min [s]	Max [s]	Relative
`base uint256_integration_benchmark`	6.123 ± 0.011	6.111	6.146	1.00
`head uint256_integration_benchmark`	6.133 ± 0.029	6.108	6.203	1.00 ± 0.01

github-actions · 2023-07-12T17:12:12Z

Benchmark Results for modified programs 🚀

Command	Mean [s]	Min [s]	Max [s]	Relative
`head blake2s_integration_benchmark`	9.150 ± 0.178	8.993	9.525	1.00

Command	Mean [s]	Min [s]	Max [s]	Relative
`head compare_arrays_200000`	3.206 ± 0.020	3.181	3.248	1.00

Command	Mean [s]	Min [s]	Max [s]	Relative
`head dict_integration_benchmark`	2.222 ± 0.013	2.203	2.244	1.00

Command	Mean [s]	Min [s]	Max [s]	Relative
`head field_arithmetic_get_square_benchmark`	1.705 ± 0.014	1.695	1.744	1.00

Command	Mean [s]	Min [s]	Max [s]	Relative
`head integration_builtins`	8.975 ± 0.009	8.963	8.994	1.00

Command	Mean [s]	Min [s]	Max [s]	Relative
`head keccak_integration_benchmark`	9.183 ± 0.146	9.053	9.471	1.00

Command	Mean [s]	Min [s]	Max [s]	Relative
`head linear_search`	3.233 ± 0.027	3.211	3.284	1.00

Command	Mean [s]	Min [s]	Max [s]	Relative
`head math_cmp_and_pow_integration_benchmark`	2.331 ± 0.014	2.314	2.354	1.00

Command	Mean [s]	Min [s]	Max [s]	Relative
`head math_integration_benchmark`	2.129 ± 0.016	2.108	2.152	1.00

Command	Mean [s]	Min [s]	Max [s]	Relative
`head memory_integration_benchmark`	1.868 ± 0.006	1.857	1.881	1.00

Command	Mean [s]	Min [s]	Max [s]	Relative
`head operations_with_data_structures_benchmarks`	2.112 ± 0.010	2.097	2.128	1.00

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
`head pedersen`	820.5 ± 1.4	818.7	822.5	1.00

Command	Mean [s]	Min [s]	Max [s]	Relative
`head poseidon_integration_benchmark`	1.483 ± 0.004	1.478	1.490	1.00

Command	Mean [s]	Min [s]	Max [s]	Relative
`head secp_integration_benchmark`	2.476 ± 0.010	2.458	2.489	1.00

Command	Mean [s]	Min [s]	Max [s]	Relative
`head set_integration_benchmark`	1.257 ± 0.004	1.251	1.267	1.00

Command	Mean [s]	Min [s]	Max [s]	Relative
`head uint256_integration_benchmark`	6.035 ± 0.015	6.020	6.070	1.00

fuzzer/src/cairo_compiled_programs_fuzzer.rs

pefontana · 2023-07-17T17:12:51Z

fuzzer/README.md

+## cairo_compiled_programs_fuzzer
+To run this fuzzer you need to be inside a py_env and be able to run cairo-compile command from the fuzzer folder beforehand.
+
+To run simply use `cargo +nightly fuzz run --fuzz-dir . cairo_compiled_programs_fuzzer`


I ran cargo +nightly install fuzz, the execute this command and got

[should i run ](error: no such command: fuzz)

I would guess it depends on cargo-fuzz and it needs to be installed first? In that case, we need to add it to the dependencies and document that need.

It seems to work (well, it's compiling right now at least) after running cargo install cargo-fuzz, FWIW.

Also, it would be desirable to document why exactly we need to use nightly here.

the command to install the tool to run this fuzzer is cargo +nightly install cargo-fuzz, i can add a command in the makefile to install the tool and other to run the fuzzer

added the documentation and the makefile commands!

Oppen

As @pefontana and @MegaRedHand pointed out:

There are a few fixes necessary for the docs
We need to use the Felt252 Arbitrary implementation to cover more ground, since many extreme cases will never be reached by u8 or u128.

daphneherlambda · 2023-07-17T18:59:55Z

There are a few fixes necessary for the docs
We need to use the Felt252 Arbitrary implementation to cover more ground, since many extreme cases will never be reached by u8 or u128.

working on it right now

MegaRedHand

Left some nits. Otherwise, LGTM

fuzzer/README.md

Juanma and others added 12 commits June 13, 2023 12:11

Create fuzzer

c3a30e6

Merge branch 'main' of https://github.com/lambdaclass/cairo-rs into c…

d85b25d

…airo-fuzzer

Remove cli

b04e55c

Update fuzzer

11eaf85

Change fuzzer name

dcd5e96

Move functions to utils; Create new fuzzer

406a53c

Remove file and update readme

9ddd363

Change filename

181c2b9

Fix filenames

06153e6

update fuzzer and merge

5bceae7

delete extra changes

351d6e3

delete extra changes

3f7cc27

daphneherlambda force-pushed the fuzz-cairo-compiled-programs branch from 0b48f0c to 3f7cc27 Compare July 12, 2023 17:07

Juanma and others added 3 commits July 12, 2023 14:25

Move to fuzzing dir

60fbefa

update and refactor cairo compiled programs fuzzer

6a4fb33

merge with main

9f0b7a8

daphneherlambda marked this pull request as ready for review July 14, 2023 19:29

daphneherlambda requested review from igaray, Oppen, fmoletta, entropidelic, juanbono and pefontana as code owners July 14, 2023 19:29

daphneherlambda added the coverage label Jul 14, 2023

dafifynn added 3 commits July 14, 2023 16:37

fmt

8c3d4de

add cairo_programs folder

ad0310e

fix warnings

3c9ff7f

MegaRedHand reviewed Jul 14, 2023

View reviewed changes

fuzzer/src/cairo_compiled_programs_fuzzer.rs Outdated Show resolved Hide resolved

fuzzer/src/cairo_compiled_programs_fuzzer.rs Outdated Show resolved Hide resolved

pefontana and others added 3 commits July 15, 2023 13:10

Merge branch 'main' into fuzz-cairo-compiled-programs

edd9274

Merge branch 'main' into fuzz-cairo-compiled-programs

82194e8

fix formatting

5e0a9fe

pefontana requested changes Jul 17, 2023

View reviewed changes

Oppen reviewed Jul 17, 2023

View reviewed changes

daphneherlambda added the WIP work in progress label Jul 17, 2023

dafifynn added 5 commits July 17, 2023 21:15

add felt arbitrary

f67aac7

add felt arbitrary

7a6b1ee

add info to readme

f9920f7

Merge branch 'main' into fuzz-cairo-compiled-programs

7da7d72

add fuzzer makefile command

7575a1b

daphneherlambda removed the WIP work in progress label Jul 18, 2023

MegaRedHand approved these changes Jul 18, 2023

View reviewed changes

fuzzer/README.md Outdated Show resolved Hide resolved

fuzzer/README.md Outdated Show resolved Hide resolved

fuzzer/README.md Outdated Show resolved Hide resolved

dafifynn added 2 commits July 18, 2023 14:38

Merge branch 'main' into fuzz-cairo-compiled-programs

affab55

fix corrections

d4de948

MegaRedHand approved these changes Jul 18, 2023

View reviewed changes

dafifynn and others added 3 commits July 19, 2023 12:36

Merge branch 'main' into fuzz-cairo-compiled-programs

0b0e142

Merge branch 'main' into fuzz-cairo-compiled-programs

d33f6ef

Merge branch 'main' into fuzz-cairo-compiled-programs

8f75685

pefontana approved these changes Jul 26, 2023

View reviewed changes

pefontana added this pull request to the merge queue Jul 26, 2023

Merged via the queue into main with commit c73b728 Jul 26, 2023

pefontana deleted the fuzz-cairo-compiled-programs branch July 26, 2023 15:52

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fuzz cairo compiled programs #1236

Fuzz cairo compiled programs #1236

Juan-M-V commented Jun 14, 2023 •

edited by daphneherlambda

Loading

codecov bot commented Jun 30, 2023 •

edited

Loading

github-actions bot commented Jun 30, 2023 •

edited

Loading

github-actions bot commented Jul 12, 2023 •

edited

Loading

pefontana Jul 17, 2023

Oppen Jul 17, 2023

Oppen Jul 17, 2023

Oppen Jul 17, 2023

daphneherlambda Jul 17, 2023

daphneherlambda Jul 18, 2023

Oppen left a comment

daphneherlambda commented Jul 17, 2023 •

edited

Loading

MegaRedHand left a comment

Fuzz cairo compiled programs #1236

Fuzz cairo compiled programs #1236

Conversation

Juan-M-V commented Jun 14, 2023 • edited by daphneherlambda Loading

TITLE

Description

Checklist

codecov bot commented Jun 30, 2023 • edited Loading

Codecov Report

github-actions bot commented Jun 30, 2023 • edited Loading

github-actions bot commented Jul 12, 2023 • edited Loading

pefontana Jul 17, 2023

Choose a reason for hiding this comment

Oppen Jul 17, 2023

Choose a reason for hiding this comment

Oppen Jul 17, 2023

Choose a reason for hiding this comment

Oppen Jul 17, 2023

Choose a reason for hiding this comment

daphneherlambda Jul 17, 2023

Choose a reason for hiding this comment

daphneherlambda Jul 18, 2023

Choose a reason for hiding this comment

Oppen left a comment

Choose a reason for hiding this comment

daphneherlambda commented Jul 17, 2023 • edited Loading

MegaRedHand left a comment

Choose a reason for hiding this comment

Juan-M-V commented Jun 14, 2023 •

edited by daphneherlambda

Loading

codecov bot commented Jun 30, 2023 •

edited

Loading

github-actions bot commented Jun 30, 2023 •

edited

Loading

github-actions bot commented Jul 12, 2023 •

edited

Loading

daphneherlambda commented Jul 17, 2023 •

edited

Loading