Automated formatting

[bouweandela]

Automated formatting of Python code is currently done using yapf. There were a questions about

1. should we format all our Python code using the latest version of yapf?
2. should we switch from yapf to black?

Regarding 1) The motivation for formatting all code with a single version of yapf is that it will lead to fewer changes in pull requests if you use yapf to format a file where you only changed a single line of code. An alternative would be to use yapf-diff, which only formats the lines of code that have changed and thus avoids the large amount of changes. A downside of this is that it may offer a slightly less consistent look of the code.

We had some discussion of this topic at the workshop, if you have an opinion, please share it here!

[schlunma]

Regarding 2:

I would prefer black over yapf. Some time ago (~1 year) I started using yapf, but I was quickly annoyed with it becuase (1) it sometimes conflicted with prospector (i.e. yapf made changes that prospector then critized) and (2) during a version change it conflicted with itself.

As far as I understand from the iris repository, black can also be used as a format checker, which would solve (1). (2) was maybe just bad timing, but I have the feeling that black is more stringent (but that's just my feeling!).

[zklaus]

Just thought I'd leave a pointer to a currently ongoing discussion on the numpy mailing list that somewhat mirrors our thoughts here:
https://mail.python.org/archives/list/numpy-discussion@python.org/thread/WZTS3XZKKM3MKRPZ5XD6SHSREYLTQ3DC/

[ehogan]

I would also prefer black over yapf :)

[schlunma]

Another tool we could consider is ruff, which is an extremely fast linter (I've been using it for quite some time now and it's awesome!).

It can already replace flake8 and isort (and others we are not using), and will probably also provide autoformatting at some point.

[lukruh]

I like using black. It's easy to use and probably makes formatting code a lower hurdle for new users.
Edit: I tryed ruff too and it provides the same functionality and formats like black but is waaaay faster.

[bouweandela]

@ESMValGroup/esmvaltool-developmentteam There seems to be a fairly strong preference for using black instead of yapf in the contributions to this discussion so far. Please chime in if you feel the same or would prefer to stick with yapf.

[rbeucher]

+1 for black

[schlunma]

To revive this discussion: I think it would be helpful to consistently format our code base with a formatter like black. Maybe it makes sense to only do this for ESMValCore though, to avoid a massive amount of merge conflicts with existing branches (I think ESMValCore does not suffer from that problem as badly). This would avoid PRs where most of the changes were introduced by a formatter (like ESMValGroup/ESMValCore#2490).

We could also do it in a step-by-step procedure just like iris did (they basically ignored most rules at the beginning and slowly enforced them in subsequent PRs). This would lead to smaller PRs which can be reviewed very easily.

[bouweandela]

Using the ruff black formatter would also be a good first step towards moving from prospector to ruff. We would also need to make sure that we pin the version in pre-commit to avoid the endless stream of formatting changes we have now, set up automatic updates of the pin, and possibly remove the separate installation as a development dependency to avoid having a duplicate pin.

[bouweandela]

I went ahead and implemented this for ESMValCore: ESMValGroup/ESMValCore#2524

[bouweandela]

@ESMValGroup/esmvaltool-developmentteam We are considering reformatting the Python code in the ESMValTool repository using the modern ruff format tool and setting up a check that everyone who contributes code also does this. This may introduce some merge conflicts for anyone who currently has open pull requests. Please comment or respond with a 👍 if you like this and with a 👎 if you expect this will cause problems for you.

[rswamina]

@bouweandela - quick check to se ask what it will replace among the tools listed here?

[bouweandela]

The formatting tools isort, yapf, and docformatter.

Replacing prospector (and flake8) with ruff is also an option, but that would be a next step.

[bouweandela]

A pull request that implements this is available now: #3893