Cats Effect 3 migration

[kubukoz]

Just making an issue to track the efforts to migrate to CE3.

Do you expect any blockers?

Ref.

• Release CE 3 as separate library typelevel/cats-effect#1745
• Ecosystem upgrade megathread typelevel/cats-effect#1330

[Odomontois]

We need to see the final decision on this issue, and if it agrees with @djspiewak's and @mpilquist's opinion, I don't feel we are going to follow the Typelevel's scheme on mass upgrade, as it seems unhealthy for current adopters.
Some adopters, that won't upgrade to CE3 soon, still wanna contribute and get updates.
So I suppose the only choice we have is alienating cats-effect from the core part and some tricks like this https://blog.7mind.io/no-more-orphans.html to keep seamless integration.
If that is done carefully ZIO users will receive seamless integration as well.
If everything fails, I guess we'll stick with cats-effect-interop and cats-effect-3-interop packages

[djspiewak]

Some adopters, that won't upgrade to CE3 soon, still wanna contribute and get updates.

I mean, this is basically why we're maintaining multiple series/ branches. The reality is that this is just like any upstream breaking change. The goal is to still provide security updates, bugfixes, and selective backports on the CE2-relevant branches until the ecosystem has moved forward to the point where this is no longer necessary. If you do the same on tofu, then you basically get the best of both worlds, and then at such a time as the main bulk of your users have upgraded to CE3, you are unsaddled by compatibility machinery that was only necessary in the short term.

The primary cost in the short-term is you can't have a single main branch, and you need to split off two series/ branches. This is good practice anyway though for any situation in which you would want to do a breaking version upgrade of your own.

Unfortunately, the ecosystem is moving forward in this fashion. Even though we're going to try to support the CE2 universe for a long period of time, a lot of libraries are going to focus more of their development effort on the latest and greatest simply because… it's the latest and greatest. So downstream users who choose to remain on CE2 for a longer period of time are doing the equivalent of staying on Scala 2.12 or 2.11 for a longer period of time: it's going to function, but they're going to get increasingly out of date. Tofu won't really do itself any favors by making it harder to consume the library within the ecosystem which will increasingly be the primary and eventually exclusive dependency context. In other words, the short-term pain of the upgrade is dwarfed by the permanent pain that would be caused by some sort of shimming solution attempting to maintain simultaneous and non-breaking compatibility with both.

We wouldn't have made these kinds of changes if we didn't think it was absolutely necessary. I know it's painful and I'm not trying to undersell that, and we honestly tried to reduce this pain to an absolute minimum. As an example, some things that we otherwise would have renamed or moved around are being retained in their original naming in Cats Effect 3, simply to reduce the number of annoyances that must be dealt with. The fact that the scalafix covers so many of the migration cases suggests that it's not going to be a painful thing for downstream users, and the benefits will be substantial.

As a note, I've tried the "no more orphans" trick and it's… difficult to get it right. The fact that it depends on classpath linearization and/or poking holes in your published artifacts is pretty dangerous. It's also worth noting that doing something like this would actually create a breaking change within Tofu itself, meaning that you would be taking the pain of a breaking version upgrade without any of the commensurate benefits that you would otherwise get. It honestly seems like the worst of both worlds to me.

At the end of the day, this is just a breaking version upgrade. No, it's not pleasant. No, it's not easy. But these things are necessary from time to time. The best thing we can do is always communicate loudly and clearly in advance that these sorts of changes are coming, which we've been doing for about 18 months. My recommendation would be to treat this as a breaking upstream upgrade on a core dependency and simply make a breaking change in Tofu. Similar to what you would do if and when Cats 3 is released.

[catostrophe]

@adamw had the same choice with sttp and he chose to change namespaces:
softwaremill/sttp@3f47b1c

Big thanks to him for having made it possible to update huge monolith apps from sttp.client to client3 smoothly.

[kubukoz]

@catostrophe thing is, sttp has a common model between versions 2 and 3, so interop between the two is easier, because you don't need to constantly convert between the two.

You could achieve the same result by shading cats-effect yourself with some tooling (I think sbt-shading is a thing), but the interop won't be as smooth. Eventually you'll have two IOApps or be forced to convert between two IOs that don't have the same sets of functionality.

Also, cats-effect has a different place in your applications than sttp, if you're using it in the whole app it's likely a dependency of (almost?) every module/file, whereas sttp only appears in the HTTP-specific parts, making it easier to hide from other modules (and thus, avoiding the need for direct interop).

Edit: Also worth reminding that although this is a major change, and everyone would ideally migrate their libraries within a couple months from now, cats-effect has been binary compatible for about 2.5 years.

[Z1kkurat]

As a result of the discussion with the maintainers, the following approach will be taken:

• a separate branch will be created
• all efforts towards migration will be performed in this separate branch which will be published as snapshots to try for early adopters
• the migration will be performed by duplicating a module's code to a new module with some suffix (say, core3) and adapting it for usage with CE3
• all PRs for migration of a particular module should be made to this separate branch
• once all modules are done, a separate PR to the main branch will be made, which will include removing current modules, renaming the new ones so they replace current ones, and bumping the version along with publishing
• all new functionality should be implemented for a stable CE2 version with (optional) forward-porting to the separate branch
• once a new major version is published, the previous CE2 branch remains in an active support with bugfixes and back-porting of a new functionality for some period of time

Example:

• a branch ce3-migration is created
• core module is copied to a new module core3, adapted to CE3 and merged to this branch
• optics module is copied to a new module optics3 with dependendcy on core3, adapted to CE3 and merged to this branch
• the same for all other modules
• when all is done, all modules are dropped, core3 is renamed to core, optics3 to optics, etc. ce3-migration is merged to master and is now the main development branch

Please feel free to correct/suggest ideas.

[Z1kkurat]

Migration branch is https://github.com/tofu-tf/tofu/tree/ce3-migration.

[Odomontois]

Well. Let's consider it done