Stable release schedule

[timothycrosley]

isort needs a more stable minor and patch release cadence.
Now that issues are able to be resolved quickly, it is tempting to push out fixes and new features as they happen.
However, a fully rolling release approach is not the best fit for a linter / formatter. Even with good test coverage and best intentions - mistakes do still happen.

See:

• CI: pin isort < 5.4.0 pandas-dev/pandas#35705
• CI / Checks (pull_request) on Github Actions failing with isort 5.3.2 -> 5.4.0 pandas-dev/pandas#35703

And the root cause of both:

• Isort 5.4.0 with combine-as drops comments on some imports #1381

isort 5 has already adopted a more mature release policy and included formatting target guarantees: https://timothycrosley.github.io/isort/docs/major_releases/release_policy/. I think it is time for the project to move in the same direction for minor and patch releases.

As a start, I'm proposing:

• Minor releases (releases that can include new features) no more than once a month.
• Patch releases aim for no more than twice a month. Though significant errors should always be fixed as quickly as they are discovered.

Community feedback on what cadence would make the most sense would be greatly appreciated!

~Timothy

[r-richmond]

My :2_cents: I'd keep the current release cadence as is; (releasing with every new feature / frequently)

Pros:

1. When things do break between versions you have a smaller list of changes since the last release; making it easier to diagnose
2. When things do break between versions less time will have passed between making the change and the issue being reported; making it easier to diagnose
3. Keeps the excitement alive by pushing new features
   • It may sound silly but keeping the project exciting helps keep the project alive and avoids disheartening issues like these where popular projects can go many months and even years between releases
   • Release new version onto PyPI psf/black#1583
     • This one makes me sad since black is such an awesome project
   • If this project is dead, just tell us pypa/pipenv#4058
     • This made me really sad until the maintainers came through and released a new version 6 months later
     • The original release issue for that one actually took 18 months ☠️ April (then May) 2020 Release Tracking Issue pypa/pipenv#3369 I very nearly switched to poetry
4. Doing the same thing more often makes you good at it and helps you figure out how to do it better or automate it (in this case releases / catching bugs)
5. Avoid additional project requirements
   • Inevitably with additional project requirements comes additional overhead
   • Effort spent on following an artificial release cadence is effort not spent on improving isort
6. Users who need more stable behavior can already get more stable behavior by pinning isort's version
   • As you said Isort is already following semantic versioning

Cons:

1. When things break you'll get questions about why you are releasing things so fast
2. You spend more time releasing new versions?
   • I'm not sure how painful that process is for you right now but maybe you consider all of these releases an additional project requirement?
3. Implicitly argued above that you'll release less bugs like Isort 5.4.0 with combine-as drops comments on some imports #1381
   • However, imho a slower release cadence by itself won't deliver on that
   • In my experience if all other things are equal having unreleased changes ready to deploy won't improve the likelihood of bugs being reported or found on those unreleased changes
   • If you'd like to prevent more issues like Isort 5.4.0 with combine-as drops comments on some imports #1381 having a larger test-suite would help catch and prevent more errors; and that is separate from a slower release cadence
   • I suppose you could offer a nightly build for users who want to be on the bleeding edge and rely on them to report bugs; however the volume of users doing that will likely not be enough to find and report edge-case errors like Isort 5.4.0 with combine-as drops comments on some imports #1381

TLDR: I'd suggest sticking with your current release cadence rather than limiting the project to an artificial release cadence & apologies for the wall of text

[timothycrosley]

Thanks @r-richmond,

This was the exact kind of well thought out and informed feedback I was looking for!

If you'd like to prevent more issues like #1381 having a larger test-suite would help catch and prevent more errors; and that is separate from a slower release cadence

Couldn't agree more with this, and the test suite will continue to grow!

The one pattern I have seen, is a pull request or issue comes in, I accept the PR or fix the issue and then rely on the test suite and then deploy it immediately (all within a few hours), and then someone notices an issue the test suite doesn't catch. I do always add a new test when this happens. However, void of a release schedule, I'm tempted to introduce some delay (even if it's just 24 hours) for non urgent PRs to allow some time for other contributors or users of the repository that choose to, to look at the change before doing a release?

[sztamas]

Hi @timothycrosley

I'm a very new contributor to isort, so please have in mind that I don't have a good historical perspective and therefore can't understand the full impact of the decision.

By default, I would prefer not slowing down the release cadence "artificially" as it doesn't deal with the root of the problem that is not good enough test coverage.

Some random thoughts:

• there are a lot of configuration options in isort. I know that you want to be kind to your users but with great flexibility comes a lot of code branches :-)
  • not sure if you have a process of deciding on how you "allow" in new configuration options and how it would impact existing options etc.. It is always easy to add new features, but removing them isn't and you will have to support more and more configuration options and their combinations
  • maybe have more test coverage around the profiles as I would think most people just pick some profile and go with that instead of custom configurations?
• have you considered using some (semi-)automated way of testing regressions? Like having a repository of python files or pulling them in from various sources and then just diff the isorting (with different profiles/options) of the same files between the versions and report if there are any differences. The initial repository could probably be built up by looking at popular python projects on github that have isort configuration and pull them in?

[timothycrosley]

Thanks for the feedback!

Closing this, instead focusing on the following tickets to tackle the underlying goal:

1. Goal Zero: Proprety based testing applied to code snippets #1393
2. Goal Zero: Extensive profile testing #1392
3. Goal Zero: Create automated test that includes full code base of largest OpenSource isort users #1391