[SPARK-54632][PYTHON] Add the option to use ruff for lint#53378
[SPARK-54632][PYTHON] Add the option to use ruff for lint#53378gaogaotiantian wants to merge 2 commits intoapache:masterfrom
Conversation
|
Tried ruff and it is super quick! in this case shall we add ruff to requirements.txt and pin the version? |
|
Well for this PR I think I'd just focus on making it an option. Then I'll start a PR to include this in our CI and probably replace flake8 by default (leaving flake8 an option if users explicitly ask for it). In that PR we will add the requirements. |
allisonwang-db
left a comment
allisonwang-db
left a comment
There was a problem hiding this comment.
I like ruff linter. But should we pick one linter to use instead of adding it as an option?
|
We definitely should. Eventually we will use |
HyukjinKwon
left a comment
HyukjinKwon
left a comment
There was a problem hiding this comment.
LGTM given that we could replace flake8 and black together with a minimal change in the near future.
|
Merged to master. |
### What changes were proposed in this pull request? Add `--ruff` to `./dev/lint-python` help messages ### Why are the changes needed? Forgot to add it in #53378 ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? `./dev/lint-python -h` shows `ruff`. ### Was this patch authored or co-authored using generative AI tooling? No Closes #53685 from gaogaotiantian/add-ruff-to-cli. Authored-by: Tian Gao <gaogaotiantian@hotmail.com> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
4 participants
What changes were proposed in this pull request?
Add
ruffas an option to lint our code.Why are the changes needed?
Our pinned
flake8version is just too old - it can't even run on 3.12+. We can upgrade flake8 version but I think gradually switch toruffis a better options. The main reason isruffis much much faster thanflake8.ruffreturns the result almost immediately (ms-level) on whole spark repo - which means we can even hook it in the pre-commit in the future.It is surprisingly compatible with flake8 - there's almost no code change needed (with two extra ignored lint types which we can fix in the future). Everything it finds is a real issue instead of a different taste.
ruffcan also serve as a black-compatible formatter which means we can probably ditch bothflake8andblackin the future.For now we only enable this option - it's not hooked into any CI or full
./dev/lint-python. However, I think we should do that soon.Does this PR introduce any user-facing change?
No
How was this patch tested?
Local lint test passed.
Was this patch authored or co-authored using generative AI tooling?
No