Skip to content

Conversation

@AlexWaygood
Copy link
Member

Summary

This PR fixes an issue in the py-fuzzer script described in #17702 (comment). Namely, if run with --only-new-bugs, it's not sufficient for the script to just check that the bug exists in the new version of Ruff/red-knot when the script is minimizing the code that triggers the bug. The script must also check that the minimized version of the code also continues to not cause a panic on the baseline executable if --only-new-bugs is passed. Otherwise, the minimization has gone "too far". It's no longer a new bug if it's been minimized to a snippet that causes a panic on both the test executable and the baseline executable.

The first commit here just simplifies the way we run red-knot on the randomly generated code: the --project CLI argument is no longer necessary, and it's no longer necessary to write a pyproject.toml file to the temporary directory. The second commit here is the one that meaningfully fixes the bug.

Test Plan

I checked that the script continues to pass with mypy using uv run --directory ./python/py-fuzzer --dev mypy from the root of my local Ruff clone.

I also did the following to test this change:

  1. I checked out 4a621c2 (the commit prior to 8a6787b)

  2. I ran cargo build -p red_knot --target-dir ./target/baseline to create a baseline executable.

  3. I checked out this branch again.

  4. I ran the following command:

    uvx \                                           
    --python=3.13 \
    --force-reinstall \
    --from=./python/py-fuzzer \
    fuzz \
    --baseline-executable="./target/baseline/debug/red_knot" \
    --bin=red_knot \
    --only-new-bugs \
    50
    

    which produced the following output:

    Running `cargo build --release` since no test executable was specified...
    Sequentially running the fuzzer on 1 randomly generated source-code files...
    Ran fuzzer on seed 50                                         [1/1]
    The following code triggers a red-knot panic:
    
    class name_3[name_2: name_0]:
        pass
    name_3.name_4
    assert name_3[name_0]
    import name_0
    
    New bugs found in the following seeds:
    50
    

    I then confirmed that the script had correctly minimized the snippet to something that caused a panic on this branch but had not caused a panic prior to 8a6787b, unlike the buggy behaviour that the script displays on main.

@AlexWaygood AlexWaygood added bug Something isn't working testing Related to testing Ruff itself labels Apr 30, 2025
@AlexWaygood AlexWaygood requested review from carljm and sharkdp April 30, 2025 15:27
@github-actions
Copy link
Contributor

ruff-ecosystem results

Linter (stable)

✅ ecosystem check detected no linter changes.

Linter (preview)

✅ ecosystem check detected no linter changes.

Formatter (stable)

✅ ecosystem check detected no format changes.

Formatter (preview)

✅ ecosystem check detected no format changes.

Copy link
Contributor

@carljm carljm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

@AlexWaygood AlexWaygood merged commit f31b1c6 into main Apr 30, 2025
34 checks passed
@AlexWaygood AlexWaygood deleted the alex/fix-fuzzer branch April 30, 2025 17:48
dcreager added a commit that referenced this pull request May 1, 2025
* main:
  [red-knot] Preliminary `NamedTuple` support (#17738)
  [red-knot] Add tests for classes that have incompatible `__new__` and `__init__` methods (#17747)
  Update dependency vite to v6.2.7 (#17746)
  [red-knot] Update call binding to return all matching overloads (#17618)
  [`airflow`] apply Replacement::AutoImport to `AIR312` (#17570)
  [`ruff`] Add fix safety section (`RUF028`) (#17722)
  [syntax-errors] Detect single starred expression assignment `x = *y` (#17624)
  py-fuzzer: fix minimization logic when `--only-new-bugs` is passed (#17739)
  Fix example syntax for pydocstyle ignore_var_parameters option (#17740)
  [red-knot] Update salsa to prevent panic in custom panic-handler (#17742)
  [red-knot] Ban direct instantiation of generic protocols as well as non-generic ones (#17741)
  [red-knot] Lookup of `__new__` (#17733)
  [red-knot] Check decorator consistency on overloads (#17684)
  [`flake8-use-pathlib`] Avoid suggesting `Path.iterdir()` for `os.listdir` with file descriptor (`PTH208`) (#17715)
  [red-knot] Check overloads without an implementation (#17681)
  Expand Semantic Syntax Coverage (#17725)
  [red-knot] Check for invalid overload usages (#17609)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working testing Related to testing Ruff itself

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants