You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We've seen a number of disruptive flakes being merged into CI lately. Flakes are most disruptive when they happen with a high probability. If we say that a flake probability of 5% is considered "disruptive" then we want to make sure that tests that will typically flake at a probability of 5% (with 512 fuzz runs) then we can reduce the number of disruptive flakes significantly by increasing the number of CI runs on merge ONLY when a new fuzz test is added.
Assuming that the probability of catching a flake is the same on every fuzz test run, some basic probability math shows that setting the fuzz runs to 512 * 90 = 46080 when a new fuzz test is added will catch an error with 99% probability.
Making this change will increase CI time only when adding new fuzz tests.
The text was updated successfully, but these errors were encountered:
We've seen a number of disruptive flakes being merged into CI lately. Flakes are most disruptive when they happen with a high probability. If we say that a flake probability of 5% is considered "disruptive" then we want to make sure that tests that will typically flake at a probability of 5% (with 512 fuzz runs) then we can reduce the number of disruptive flakes significantly by increasing the number of CI runs on merge ONLY when a new fuzz test is added.
Assuming that the probability of catching a flake is the same on every fuzz test run, some basic probability math shows that setting the fuzz runs to
512 * 90 = 46080
when a new fuzz test is added will catch an error with 99% probability.Making this change will increase CI time only when adding new fuzz tests.
The text was updated successfully, but these errors were encountered: