py-fuzzer: fix minimization logic when --only-new-bugs is passed
#17739
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This PR fixes an issue in the py-fuzzer script described in #17702 (comment). Namely, if run with
--only-new-bugs, it's not sufficient for the script to just check that the bug exists in the new version of Ruff/red-knot when the script is minimizing the code that triggers the bug. The script must also check that the minimized version of the code also continues to not cause a panic on the baseline executable if--only-new-bugsis passed. Otherwise, the minimization has gone "too far". It's no longer a new bug if it's been minimized to a snippet that causes a panic on both the test executable and the baseline executable.The first commit here just simplifies the way we run red-knot on the randomly generated code: the
--projectCLI argument is no longer necessary, and it's no longer necessary to write apyproject.tomlfile to the temporary directory. The second commit here is the one that meaningfully fixes the bug.Test Plan
I checked that the script continues to pass with mypy using
uv run --directory ./python/py-fuzzer --dev mypyfrom the root of my local Ruff clone.I also did the following to test this change:
I checked out 4a621c2 (the commit prior to 8a6787b)
I ran
cargo build -p red_knot --target-dir ./target/baselineto create a baseline executable.I checked out this branch again.
I ran the following command:
which produced the following output:
I then confirmed that the script had correctly minimized the snippet to something that caused a panic on this branch but had not caused a panic prior to 8a6787b, unlike the buggy behaviour that the script displays on
main.