FR: Support multiple scala language versions in one workspace

[virusdave]

Not a new desire or request, of course. See this prior issue as an example.

We'd really like to be able to build different targets with different versions of Scala to make language version upgrades sane & realistic in large codebases. Atomically changing everything at once gets really difficult as codebases scale, especially with large external dependencies like Spark.

[liucijus]

@virusdave do you have any ideas how should multiple version support look from a user side? I think having Scala targets that build for different versions is involving, but not hard to achieve (fix compiler bootstrapping, toolchains and parametrize rules), what is hard how do we configure and provide deps for different Scala versions. I see a few challenges related to deps:

• how to integrate with loaders like rules_jvm_external and others, which assume single Maven artifact for Scala deps
• not breaking tools and IDEs compatibility

[lukaszwawrzyk]

@liucijus We want to work on supporting multiple scala versions in one build.

Inspired by python

I was looking into rules_python implementation and I hope that this approach can be implemented into rules_scala.

It has some utility macros which allows to specify which versions you want to use, which version is the default and it generates a repository for each version. The rules are wrapped in transition that changes python version. User can import py_* rules from repository with a specific version.

Intitial approach

This approach would at least enable us to create separate, unrelated targets that build for different version of scala in one build. In this basic approach, I believe users could specify each library with required version in rules_jvm_external and just depend on it. For example cats-core_2.13 and cats-core_3 and explicitly depend on right version when specifying targets.

Assuming shared targets that can be built with multiple versions of scala, this could be archieved with macros. For example scala_cross_library(name = "a", deps = ["@maven//:org_typelevel_cats_core_{scala_version}"]). Similarly a macro could help with adding 3rd party deps for multiple scala versions.

As for IDE support, I think that during import we could force to only use targets with default scala version and it should be good enough to work with existing tooling. I think that having separate targets with different scala versions that are not crossbuilt could be supported in the IDE. Targets with shared sources could be messy but I think we could somehow handle it.

More complex approach

Maybe more involved idea would be to only do target transition on _binary and _test targets and _library targets would be transitioned through the deps field of _binary/_test targets. The library targets would use select to configure themselves based on version that they are used with. It could also be wrapped in a convenience macros.
It still should play well with the IDE if we do import using default scala version.

Backward compatibility

Also, the single scala version in the repo would look as usual, using rules without transitions, so the simple use case would not break.

I'd be glad to hear what do you think about these ideas and if it makes sense. If it is doable, could you provide some initial guidance on what needs to be fixed to enable scala rules to use transitions? We already know that the SCALA_VERSION dependent code has to be refactored to read this information from the toolchain, but for example I don't know what is the issue with compiler bootstrapping.

[liucijus]

@lukaszwawrzyk thanks for looking into this!
Regrading backwards compatibility: I'm fine with making a breaking change, if don't lose important functionality and it's trivial to migrate old code.
Regarding approaches, I like the simplicity of macros approach though it puts some burden on users to make sure their code depends on the right version target. Regarding transitions, it would be very interesting to see what could be achieved with them.

[mateuszkuta256]

I prepared a poc that demonstrates a possible way to register multiple scala toolchains
First, I define multiple versions of "toolchain deps" by adding a "scala_version" suffix. For example, the compiler would look like this: io_bazel_rules_scala_scala_compiler_3_3_0. With this setup, it is possible to define a separate toolchain for each version:

setup_scala_toolchain(
    name = "3_3_0_toolchain",
    parser_combinators_deps = [
        "@io_bazel_rules_scala_scala_parser_combinators_3_3_0",
    ],
    ...
)

The toolchains are then restricted to a specific scala using: target_settings = ["//scala/versions:2_13_12"]
For a given rule, such as scala_library, we can add a transition responsible for routing to the appropriate toolchain
in my poc you can build both 2.13.12 and 3.3.0 versions:

scala_library(
    name = "test",
    scala_version = "2_13_12",
    srcs = ["Test.scala"],
    visibility = ["//visibility:public"],
)

Please let me know what you think about this. If this approach is approved, in the next step, I can modify scala_config() to automatically consume a list of supported scala versions and register the corresponding toolchains automatically.

[johnynek]

This exciting.

A couple of comments:

1. It feels like the scala_version setting should be a label, not a string that we match on internally.

2. if you chance the scala 3 version a target using scala 2 shouldn't be invalidated in the cache. I don't know if that's the case with the design now just something ideally we could test.

3. We should show an example of a target that has two different sources depending on scala version. That's a related problem where you want to have the same API but need to use different syntax in the different versions.

[mateuszkuta256]

thank you @johnynek, I will address these comments in a regular Pull request. Regarding 2), I am sure in such a case caches get invalidated now
and does the idea seem promising to you, @liucijus? if so, I will continue working on a legitimate implementation

[liucijus]

@mateuszkuta256 looks good and definitely worth trying out. @simuons what do you think?

[johnynek]

@lukaszwawrzyk thanks for looking into this! Regrading backwards compatibility: I'm fine with making a breaking change, if don't lose important functionality and it's trivial to migrate old code.

I really hope we can do this without breaking anyone. Migrating monorepos isn't a fun task. Putting that tax on everyone to help what I believe to be a rare case isn't great. Anything that requires a change to existing rules (which is to say O(N) changes in your repo) I hope we can rule out. Making some O(1) change to the workspace setup I think isn't too burdensome but if there is a speed bump to upgrade we are encouraging users to fork or stay on old versions.