Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

📋 Formatter black compatibility tracking issue #5828

Closed
konstin opened this issue Jul 17, 2023 · 11 comments
Closed

📋 Formatter black compatibility tracking issue #5828

konstin opened this issue Jul 17, 2023 · 11 comments
Labels
formatter Related to the formatter

Comments

@konstin
Copy link
Member

konstin commented Jul 17, 2023

We will continuously update this issue with compatibility measure between our formatter and black and the general status of the formatter.

Not Implemented

not_yet_implemented usages: 0
not_yet_implemented_custom_text usages: 0

Select Projects / Small ecosystem check

To condense the compatibility over an entire repository, we use the similarity index. It is defined as the percentage of lines that remains unchanged between black's formatting and our formatting. You could compute it as the number of neutral lines in a diff divided by the neutral plus the removed lines.

I chose 6 different projects to check for now, we may update this list in the future.
To reproduce these numbers or to check your own changes, you can run scripts/formatter_progress.sh:

django (web framework): 2752 files, similarity index 0.99805:

transformers (ML framework): 2527 files, similarity index 0.99620

twine (small util library): 32 files, similarity index 0.99876

typeshed (stub files): 3627 files, similarity index 0.99953

warehouse (pypi backend): 641 files, similarity index 0.99601

zulip (a django user): 1424 files, similarity index 0.99727

Large ecosystem check

There is a number of formatter bugs showing up in the ecosystem checks, a list from 2023-08-21 is in https://gist.github.com/konstin/c7183dda99e65c7a1acbd970d2f44b42. Implicit string concatenation and fluent style seem to be the two common causes. Gist created by:

target/release/ruff_dev format-dev --stability-check --error-file target/errors-$(date --iso-8601).txt --multi-project target/checkouts > target/stability-check-release-$(date --iso-8601).txt

and manually removing e.g. slow formatting.

@konstin konstin added the formatter Related to the formatter label Jul 17, 2023
@cnpryer
Copy link
Contributor

cnpryer commented Jul 17, 2023

I know it's comment placement, but is #5630 related to black compatibility?

@konstin
Copy link
Member Author

konstin commented Jul 18, 2023

I know it's comment placement, but is #5630 related to black compatibility?

i'd expect fixing this would be a minor improvement in the compatibility score, but it's unfortunately hard to tell beforehand

@konstin konstin changed the title Formatter black compatibility tracking issue 📋 Formatter black compatibility tracking issue Jul 19, 2023
@konstin konstin changed the title 📋 Formatter black compatibility tracking issue 📋 Formatter black compatibility tracking issue Jul 19, 2023
@konstin
Copy link
Member Author

konstin commented Jul 19, 2023

Old numbers:

build: similarity index 0.738
django: similarity index 0.978
transformers: similarity index 0.972
typeshed: similarity index 0.827
zulip: similarity index 0.970

New numbers:

build: 40 files, similarity index 0.740
django: 2752 files, similarity index 0.983
transformers: 2527 files, similarity index 0.976
typeshed: 3627 files, similarity index 0.766
zulip: 1424 files, similarity index 0.975

I'm unsure what happened with typeshed

@konstin
Copy link
Member Author

konstin commented Jul 19, 2023

#5882 fixes the transformers bugs, after that all test projects pass without errors (except for those files that already have syntax errors in the input)

@konstin
Copy link
Member Author

konstin commented Jul 19, 2023

Updated the gist to today (https://gist.github.com/konstin/edd2a914fdf4196020d9efa3f2c7e6ff) and filed issues for recurring themes.

@konstin
Copy link
Member Author

konstin commented Jul 20, 2023

I minimized all cases of unstable formatting from the ecosystem checks: https://gist.github.com/konstin/61adab19d5615160750171944653673d . This gives us a single file that checks for all known cases of unstable formatting

konstin added a commit that referenced this issue Jul 21, 2023
**Summary** Add script to shrink all formatter errors: This started as a
fun idea and turned out really useful: This script gives us a single
Python file with all formatter stability errors. I want to keep it
around to occasionally update #5828 so I added it to the git.

**Test Plan** None, this is a helper script
@konstin
Copy link
Member Author

konstin commented Jul 24, 2023

#6035 documents how to run the formatter progress tracking script yourself, and it's also part of CI now

@konstin
Copy link
Member Author

konstin commented Jul 25, 2023

Tracking issue specifically for django: #6069

@MichaReiser MichaReiser added this to the Formatter: Stable milestone Jul 31, 2023
@konstin
Copy link
Member Author

konstin commented Aug 21, 2023

Updated the original post

@cnpryer
Copy link
Contributor

cnpryer commented Aug 28, 2023

not_yet_implemented_custom_text usages: 8

Was browsing. Figured I could cc #6905 (comment). Looks to be no longer used :)

@konstin
Copy link
Member Author

konstin commented Sep 4, 2023

project similarity index total files changed files
django 0.99957 2760 67
transformers 0.99928 2587 456
twine 0.99982 33 1
typeshed 0.99978 3496 2173
warehouse 0.99818 648 24
zulip 0.99942 1437 32

We've reached >99.9% in 5 out of 6 projects! There's still some differences remaining, but all nodes are implemented and all the big recurring themes are fixed, so i'm closing this issue.

@konstin konstin closed this as completed Sep 4, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
formatter Related to the formatter
Projects
None yet
Development

No branches or pull requests

3 participants