-
Notifications
You must be signed in to change notification settings - Fork 1.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: enable saving and loading of corpus for deterministic fuzzing #991
Comments
Can you expand a bit on what is meant by campaign and corpus for the uninitiated? |
A fuzz campaign is a term that basically means “execution of the fuzzer test suite with the configured params” Corpus is a fuzzing term which refers to the set of inputs used by the fuzzer. More specifically it often (but not always) refers to the set of interesting inputs, i.e. the set of inputs that maximizes coverage. I can see an argument the “save corpus” isn’t the right terminology given we don’t have code coverage, but I think it’s a good term here since it makes it clear what you’re saving. Pinging @transmissions11 for thoughts here too |
Appreciate the clarification 😄 |
As part of this, it would also be great if failed fuzz runs were always saved off as |
Is there a preferred format for the corpus json files? |
I don't think so. If proptest has a default format it uses to save/load the data that should be fine, but I'll let @gakonst confirm to make sure there's nothing I'm missing |
Hi hi, I did some preliminary digging into
There is a bit of chatter about #1359 but since this should ultimately manage/interact with the fuzzer rather than solidity-land they shouldn't overlap. Will take a whack at this soooooon |
Thanks for looking into this, very excited for this feature!
None that I'm aware of. I'm guessing we just never set it up so used Which brings up the question—where should the output files be saved? I think the best approach may be:
This seems like a nice option to expose in the config to allow deterministic fuzz runs by specifying a seed. We can probably just expose the How exactly would the seed be used by foundry? Specifically what I'm trying to get at is if the randomness is isolated per test, i.e. if I keep the seed constant, then add/remove tests, will each unchanged test still have the same inputs? Ideal answer is yes.
How does this work / what exactly would get cached here? Guessing it's the full set of inputs to each fuzz method, as well as pass/fail result for each input?
Depending on how the
|
Thank you for the additional clarification! Based on the above, my initial implementation should look something like... RNG Seeding
Failure Persistence
Corpus Saving & Replay
lmk if I'm missing anything or should take a different path, otherwise I'll start chipping away! |
I think it would be better to place the failures etc. in |
Overall this plan sounds great to me as well, no objections with defaulting to The "RNG Seeding", "Failure Persistence", and "Corpus Saving and Replay" sections seem isolated enough that it could be good to do them incrementally as smaller PRs to keep scope of each smaller (will leave it up to @Bind and @onbjerg to figure out what's best, just a thought).
One question is what do we need both of these for? I'm not sure exactly how the seed would be specified, but if it's something simple like an integer, then an alternate file option may be unnecessary? |
Good call, will do!
Fair point, I'll implement the |
@Bind curious if you had any follow up on the fuzz seed option? |
Sorry about the hold up! Getting back from some traveling tomorrow, will get RNG Seeding over the line in the next few days! |
Didn't want this to drag for too long! I opened a Draft PR with my implementation and some outstanding questions I have about the best path for testing this feature! |
* feat: basic rng seeding * chore: bump u32 to U256 * feat(config): add additional helper macro * feat: finish fuzz seed impl * bump ethers Co-authored-by: Matthias Seitz <matthias.seitz@outlook.de>
* feat: basic rng seeding * chore: bump u32 to U256 * feat(config): add additional helper macro * feat: finish fuzz seed impl * bump ethers Co-authored-by: Matthias Seitz <matthias.seitz@outlook.de>
Component
Forge
Describe the feature you would like
Add
--save-corpus
and--load-corpus <path>
options to allow replaying of an entire fuzz campaign. I think proptest has methods to facilitate this but need to double check. It would also be great if saving a corpus saves off all inputs used (as opposed to just rng seeds) so they can be inspected/analyzedAdditional context
No response
The text was updated successfully, but these errors were encountered: