Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

max_test_reject_rate: set a maximum test rejection rate per test function #4091

Open
0xPhaze opened this issue Jan 14, 2023 · 3 comments
Open
Labels
A-testing Area: testing C-forge Command: forge T-feature Type: feature

Comments

@0xPhaze
Copy link

0xPhaze commented Jan 14, 2023

Component

Forge

Describe the feature you would like

I never found using max_test_rejects and vm.assume helpful, because changing the number of fuzz runs to a different value requires re-calibrating max_test_rejects to a sensible value. What's more is that this value will likely depend on the amount of tests you have that are making use of vm.assume.

It would be useful to be able to set a maximum reject rate that is applied per test function. Say, if I want the rejection rate to be a maximum of 10%, then this should only apply to the rejection count of each function and not globally discard the whole fuzz campaign.

Perhaps adding this information as natspec could be useful (#4085).

Edit: I see the max test rejects actually apply per function already, still it would be great to specify this in a percentage.

@0xPhaze 0xPhaze added the T-feature Type: feature label Jan 14, 2023
@0xPhaze 0xPhaze changed the title max_test_reject_rate: max_test_reject_rate: set a maximum test rejection rate per test function Jan 14, 2023
@mds1
Copy link
Collaborator

mds1 commented Jan 16, 2023

Hmm, I'm hesitant about overcomplicating this flag—if you change the number of fuzz runs by changing your profile, you can also update the max number of rejects in that profile so the two are always in sync. So this doesn't seem to add too much value

@0xPhaze
Copy link
Author

0xPhaze commented Jan 17, 2023

Personally, I'd still find it helpful. Though, if no one else sees the need, then we can close this for now, as it's not high-priority.

@PaulRBerg
Copy link
Contributor

I am with @0xPhaze, I would also find this feature useful.

Related: #4085.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-testing Area: testing C-forge Command: forge T-feature Type: feature
Projects
Status: Todo
Development

No branches or pull requests

6 participants