RFC Better transitions

[laurynaslubys]

Description

Containers in essence only make sense for a very specific os/arch combination.
Transitions, as they were introduced in #1963 were a good starting step, however there are still quite a few issues and improvements suggested (#2055, #2052, #2037, #2063) . There are also some suggestions in the original pull request.

Describe the solution you'd like

Just to get the ball rolling, here's what I had in mind:

We could have a new attribute, let's say platform on *_image rules. This would transition the container dependencies to the specified platform. If no platform is specified, no transition would take place. This should fix the issues with golang and cc toolchain resolution and allow users to control the platform directly.

We should also consider enabling usage where a single target can build containers for multiple platforms, something like:

container_image(
   name = "image"
   architecture = select({
       "@platforms//cpu:x86_64": "amd64",
       "@platforms//cpu:arm64": "arm64"
    }),
   base = select({
       "@platforms//cpu:x86_64": "@some_base_x86_64",
       "@platforms//cpu:arm64": "@some_base_arm64"
   }),
   platform = select({
       "@platforms//cpu:x86_64": "//:my_x86_64_platform",
       "@platforms//cpu:arm64": "//:my_amd64_platform",
   })
)

bazel build --platforms="@io_bazel_rules_docker//platforms:x86_64" :image # builds a valid x86_64 image
bazel build --platforms="@io_bazel_rules_docker//platforms:arm64" :image  # builds a valid arm64 image

where //:my_x86_64_platform is a user defined or lang specific toolchain (e.g. @io_bazel_rules_go//go/toolchain:linux_amd64) and @io_bazel_rules_docker//platforms:x86_64 is platform that only has a single constraint -- @platforms//cpu:x86_64.

I believe the following changes need to made:

• make the transitions outgoing.
• add a platform attribute to all *_image rules. Make the transition only override the platform if it is the platform attr was set. Remove the synthetic platforms we have.
• add base images for architectures other than amd64 and select them based on the architecture.

Describe alternatives you've considered

Inheriting from base container

In an absolutely perfect world, the base image would probably determine what the platform of the final container is. Then a single select would be enough for a multi-platform image. However that is not possible because you cannot access targets in transitions.

Automatically cross compiling

This is what was introduced in #1963. Unfortunately, it does not work in reality for several reasons:

1. lang_images depend on rules from other bazel packages. Cross compiling them in the best case introduces a dependency on a cross compiling toolchain, which are not wide spread. In the worst case it requires dealing with a bunch of other things:
   • rules_nodejs needs a different node repository has to be specified and npm does not build dependencies hermetically, which may result in images that differ when built on a different os/arch.
   • rules_go essentially has two modes: pure and cgo. Pure binaries can be safely cross-compiled, cgo requires a cc toolchain to be present for the target platform. Additionally there's goos and goarch which somewhat clash with our architecture and operating_system.
   • It seems that getting it to work in rules_py also has some rough edges.
2. The transition to a synthetic platform breaks cases where the platform has constraints other than os and cpu. This can be somewhat worked around by making the transition on platform optional Make transitions optional #2068.

[laurynaslubys]

After hacking at this for a couple of days, I'm not sure if this can be added with a reasonable time investment. I'll take another stab next weekend probably. However, given the caveats for some lang_images described above, I'm not sure if this is something that rules_docker should support natively. In that case the best plan would probably be to revert #1963

[laurynaslubys]

Following on our discussion, we've desided to make transition optional by landing #2068, and revisiting proper multiarch support at some unspecified time in the future.