Add a script that tests formatter stability on repositories #5055
Conversation
|
Current dependencies on/for this PR: This comment was auto-generated by Graphite. |
PR Check ResultsEcosystem✅ ecosystem check detected no changes. BenchmarkLinuxWindows |
|
Potentially would also be interesting to integrate this with the fuzzer for generating that initial corpus. The corpus I used is from a somewhat outdated python3 dataset currently, so it might not be representative of all modern features. |
|
As someone who has little experience with fuzzer, what would it mean to use cpython for the initial corpus? |
## Summary This fixes a number of problems in the formatter that showed up with various files in the [cpython](https://github.com/python/cpython) repository. These problems surfaced as unstable formatting and invalid code. This is not the entirety of problems discovered through cpython, but a big enough chunk to separate it. Individual fixes are generally individual commits. They were discovered with #5055, which i update as i work through the output ## Test Plan I added regression tests with links to cpython for each entry, except for the two stubs that also got comment stubs since they'll be implemented properly later.
|
The corpus is the set of inputs on which the fuzzer tries to modify such that it triggers new code paths. If the fuzzer is provided with a corpus containing representative samples of inputs that can be handled by the program being tested, then it is more likely to mutate into an input that triggers unexpected behaviour. This is because the parser is less likely to outright reject the input and the fuzzer more likely to introduce new data to the input that triggers unexpected behaviour while processing the parsed input. Also, it means the fuzzer has less work to do to "discover" new functionality or features. |
|
Thanks for the explanation! In this case cpython plus our own test fixture should be a great start |
|
In that case, once this lands, I'll update the (re)init-fuzzer script to use cpython source as part of the corpus init. |
|
@konstin - Is this ready for review? Tag me when ready :) |
|
it is :) |
| match &args.command { | ||
| Command::GenerateAll(args) => generate_all::main(args)?, | ||
| Command::GenerateJSONSchema(args) => generate_json_schema::main(args)?, | ||
| match args.command { |
There was a problem hiding this comment.
i've changed this back and forth like three times 😅 i can also roll this back once again
konstin
force-pushed
the
formatter-stability-repo-check
branch
from
c4bbe44
to
243253c
Compare
Progress as of today, can check CPython and finds a bunch of errors (`cargo run --bin ruff_dev projects/cpython`). Needs proper integration, output status and a better PR description.
konstin
force-pushed
the
formatter-stability-repo-check
branch
from
243253c
to
a1f99e1
Compare
| /// Control the verbosity of the output | ||
| #[derive(Copy, Clone, PartialEq, Eq, clap::ValueEnum, Default)] | ||
| pub(crate) enum Format { | ||
| // Filenames only |
There was a problem hiding this comment.
These appear in the Clap output, right? Can we make them actual rustdocs?
Following #5055, add cpython as a member of the fuzzer corpus unconditionally.
Summary
We want to ensure that once formatted content stays the same when formatted again, which is known as formatter stability or formatter idempotency, and that the formatter prints syntactically valid code. As our test cases cover only a limited amount of code, this allows checking entire repositories.
This adds a new subcommand to
ruff_devwhich can be invoked ascargo run --bin ruff_dev -- check-formatter-stability <repo>. While initially only intended to check stability, it has also found cases where the formatter printed invalid syntax or panicked.Test Plan
Running this on cpython is already identifying bugs (#5089)