📋 Formatter black compatibility tracking issue

[konstin]

We will continuously update this issue with compatibility measure between our formatter and black and the general status of the formatter.

Not Implemented

not_yet_implemented usages: 0
not_yet_implemented_custom_text usages: 0

Select Projects / Small ecosystem check

To condense the compatibility over an entire repository, we use the similarity index. It is defined as the percentage of lines that remains unchanged between black's formatting and our formatting. You could compute it as the number of neutral lines in a diff divided by the neutral plus the removed lines.

I chose 6 different projects to check for now, we may update this list in the future.
To reproduce these numbers or to check your own changes, you can run scripts/formatter_progress.sh:

django (web framework): 2752 files, similarity index 0.99805:

• Tracked in 📋 Black-compatible formatting of django #6069

transformers (ML framework): 2527 files, similarity index 0.99620

• Skip cookiecutter directories by default #5824 (The project doesn’t use black . but black examples tests src utils , same thing for ruff)

twine (small util library): 32 files, similarity index 0.99876

typeshed (stub files): 3627 files, similarity index 0.99953

• Formatter: stub files only have one empty lines between top level items #5821
• Formatter: Ellipses in stub files get formatted on a single line #5822
• We're missing black's exact empty line rules: Formatter: Empty lines in stub files #6723

warehouse (pypi backend): 641 files, similarity index 0.99601

zulip (a django user): 1424 files, similarity index 0.99727

Large ecosystem check

There is a number of formatter bugs showing up in the ecosystem checks, a list from 2023-08-21 is in https://gist.github.com/konstin/c7183dda99e65c7a1acbd970d2f44b42. Implicit string concatenation and fluent style seem to be the two common causes. Gist created by:

target/release/ruff_dev format-dev --stability-check --error-file target/errors-$(date --iso-8601).txt --multi-project target/checkouts > target/stability-check-release-$(date --iso-8601).txt

and manually removing e.g. slow formatting.

[cnpryer]

I know it's comment placement, but is #5630 related to black compatibility?

[konstin]

I know it's comment placement, but is #5630 related to black compatibility?

i'd expect fixing this would be a minor improvement in the compatibility score, but it's unfortunately hard to tell beforehand

[konstin]

Old numbers:

build: similarity index 0.738
django: similarity index 0.978
transformers: similarity index 0.972
typeshed: similarity index 0.827
zulip: similarity index 0.970

New numbers:

build: 40 files, similarity index 0.740
django: 2752 files, similarity index 0.983
transformers: 2527 files, similarity index 0.976
typeshed: 3627 files, similarity index 0.766
zulip: 1424 files, similarity index 0.975

I'm unsure what happened with typeshed

[konstin]

#5882 fixes the transformers bugs, after that all test projects pass without errors (except for those files that already have syntax errors in the input)

[konstin]

Updated the gist to today (https://gist.github.com/konstin/edd2a914fdf4196020d9efa3f2c7e6ff) and filed issues for recurring themes.

[konstin]

I minimized all cases of unstable formatting from the ecosystem checks: https://gist.github.com/konstin/61adab19d5615160750171944653673d . This gives us a single file that checks for all known cases of unstable formatting

[konstin]

#6035 documents how to run the formatter progress tracking script yourself, and it's also part of CI now

[konstin]

Tracking issue specifically for django: #6069

[konstin]

Updated the original post

[cnpryer]

not_yet_implemented_custom_text usages: 8

Was browsing. Figured I could cc #6905 (comment). Looks to be no longer used :)

[konstin]

project	similarity index	total files	changed files
django	0.99957	2760	67
transformers	0.99928	2587	456
twine	0.99982	33	1
typeshed	0.99978	3496	2173
warehouse	0.99818	648	24
zulip	0.99942	1437	32

We've reached >99.9% in 5 out of 6 projects! There's still some differences remaining, but all nodes are implemented and all the big recurring themes are fixed, so i'm closing this issue.