Let targets set what platforms they build for

[gregestren]

Tracking issue on Bazel Configurability Roadmap

Goal:

• Something like this works:

$ cat a/BUILD
cc_binary(name = "app_for_mac", platforms = ["//platforms:mac"])

$ bazel build //a:app_for_mac # Automatically builds for a Mac, no command line flags!

This syntax is theoretical; it may not look exactly like this. The point is rules can embed which platforms they target and all the correct toolchains get linked in without having to specify anything at the command line.

Furthermore, different rules needing different toolchains (e.g. $ bazel build //a:all) will all independently do the right thing.

This is v0 because it may have performance consequences. This also doesn't cover non-platform settings (like app version, enable/disable feature X), which will be covered elsewhere in the roadmap.

[gregestren]

Prerequisite work is still happening, so marking as a P3 now. This means we're not actively working on this yet. Will upgrade priority once the prereqs are out of the way.

[gregestren]

April '19 update:

Still blocked on prerequisites of basic platform & toolchain support. I'd still estimate a "second half of the year" time frame for starting to experiment with this, unless there's enough enthusiastic interest to jump on board even without concern about the scaling & caching concerns.

Technically I think really interested users will be able to explore this in the next month or two by manually combining together platform/toolchain support and Starlark transitions.

[wpieterse2825]

@gregestren Hi Greg, can you elaborate on how we can achieve multi-platform builds via transitions?

I'm building a small kernel using Bazel as the build engine. The boot-loader is being written targeting UEFI, which uses a PE COFF toolchain (basically targeting Windows via CLang/LLVM), and I want the kernel executables itself to be in ELF format (x86_64-none-elf).

At the moment, I have no clear path to go about it, was thinking more in the lines of having two separate Bazel build steps, and then combining the data into a boot disk, but if I could do everything inside Bazel it will be so much easier.

[gregestren]

Hey @wpieterse2825,

Building in separate steps is indeed an option if it comes to that. A major inspiration of these new APIs is to not have to resort to that, so we're on the same page there.

What BUILD rules are you using to represent the boot-loader and kernel executables and what flags are you using to switch the toolchain?

I suspect we can get a clear path for your use case.

[wpieterse2825]

@gregestren Thanks for the reply Greg.

At the moment they are just plain cc_binary and cc_library rules. You can have a look at the repository.

I'm not doing anything fancy at the moment, just settings the --crosstool-top switch to point to my custom toolchain that can build UEFI applications. I still need to separate out the various pieces, I'm just not sure what is the correct direction forward. What would be you're opinion on how I should handle this? I saw the configuration and build road-map on the Bazel website, and I'm quite excited for the option to use platforms inside the cc_binary and cc_library rules.

Also, this is just a pet project to build a small "DOS" like operating system, nothing serious.

[wpieterse2825]

Also, to add to the above, the uefi part needs to be built with the PE COFF toolchain, while the kernel and architecture parts I need in ELF format.

Thinking of it, I’m using LLVM and LLD, maybe I should try playing with creating a macro wrapping the cc rules for the UEFI and Kernel parts, where the macro itselfs sets copts to specify what linker to use.

[katre]

@wpieterse2825: Have you taken a look at #7260? Once you write toolchain() targets for your existing cc_toolchain() declarations, you can use the flag to enable platform-based cc toolchain resolution now, which might be simpler than using copts and crosstool_top. All of the missing features of full multiplatform builds are still present, of course, but it moves you closer to that world.

[wpieterse2825]

Hi @katre, thanks for mentioning it.

I did try and do that, defining the various architectures and firmware targets. You can have a look here. But when I try and build only with platforms, without specifying the --cpu parameter, it always tries to build with the MSVC toolchain, which obviously causes problems when linking the binary. I could not figure it out at the time, will give it a try again and let you know my results.

[wpieterse2825]

@katre I did what you suggested and it worked. Could finally remove the --cpu flag from my .bazelrc file as everything is now compiled with platforms. Thanks.

[katre]

Excellent, I'm glad it's working, we're trying hard to get this ready enough to enable by default.

[gregestren]

As for the multiplatform part, @juliexxia is working on much more extensive documentation. But in the meantime check out https://github.com/gregestren/snippets/wiki/Configuring-a-cc_binary-differently-than-a-cc_test for an example of configuring cc_ rules differently in the same build.

I believe you can apply this to --platforms today.

[wpieterse2825]

Alright, thanks @gregestren will try that thanks for all the help

[konkers]

Would this allow one target to depend on the output of another target with a different platform?

An example of this could be developing a piece of hardware with its own firmware and a companion app. The firmware and app target different platforms (say an ARM Cortex M0 microcontroller and windows x86_64) If the app wants to be able to bundle the firmware update, it needs to depend on the output of its compilation.

[gregestren]

That should automatically fall out of this, yes.

[konkers]

That's super exciting. Thanks!

[prestonvanloon]

Any chance this can bump to P2? Will it make it into bazel v1?

[gregestren]

Hi @prestonvanloon - this is sort of a meta-issue. Various other actively prioritized issues together will essentially fulfill this one.

In my mind prioritizing this specific bug would involve prioritizing the API as suggested in the original description. But depending on your exact needs it should be possible to get a lot of this work already. So I think it's reasonable to expect this for Bazel 1.0 but the details depend on which languages you want support for and your exact API needs.

All that said, what specifically do you want to do?

[prestonvanloon]

My mission is to have a target that can generate a tar / zip archive which binaries for multiple platforms. Specifically, with golang with rules_go.

Currently, we can manually specify each platform tool chain and build the target, but I would like to have some single target that can build the target for all four platforms.

# //:main is a go_binary 

# What we have today
bazel build --platforms=@io_bazel_rules_go//go/toolchain:linux_amd64_cgo //:main
bazel build --platforms=@io_bazel_rules_go//go/toolchain:linux_arm64_cgo //:main
bazel build --platforms=@io_bazel_rules_go//go/toolchain:darwin_amd64_cgo //:main
bazel build --platforms=@io_bazel_rules_go//go/toolchain:windows_amd64_cgo //:main

# What we want for bazel v1
bazel build //:distro_zip # genrule tar/zip with the outputs from the target/platforms above.

[gregestren]

I think this should be straightforward if you write a rule zipper using Starlark transitions on the --platforms flag.

Also see https://github.com/gregestren/snippets/wiki/Configuring-a-cc_binary-differently-than-a-cc_test for similar example code (not quite the same use case but covers the same principle of making a rule build its deps differently)

Starlark transitions even has support for split deps, which could offer more concise syntax (you're building the same dep each time). I'm not sure if that's 100% ready but it might be and I encourage exploration. :)

If any of this sounds promising, I suggest a separate issue or bazel-discuss@ thread to back-and-forth on the details (I'm happy to continue offering guidance). If this all looks obviously straightforward, please share your results! It'd be awesome to know if this stuff is helping solve real meaningful use cases.

[gregestren]

This bug has become too broad, and isn't actionably relevant. Let's follow https://bazel.build/roadmaps/configuration.html for a finer-grained breakdown of priorities, and add more specialized bazel-discuss questions or issues as needed.

[rajivshah3]

Does that mean #10061 should be reopened then? It was closed because it would be tracked by this issue (#10061 (comment))

[gregestren]

I see your point. Maybe we should keep this open, then.

My concern is to keep it well-scoped, since "flagless" implies "no flags" and there are many kinds of flags in the most generic sense. We're really only focusing on --cpu and --platforms.

I'll re-open and adjust the title some.

[uri-canva]

Is this now almost possible because of the work done to support transitions, or is that a completely separate thing?

[katre]

Yes, this is close to being doable. I'm working on some high-level proposals now (since my last attempt wasn't workable for a lot of users).

[sluongng]

Hey folks, any update on support for multiple target platforms?
Today we are seeing this warning

WARNING: --platforms only supports a single target platform: using the first option //platforms:linux_x86_64

and the code for this warning points back to this issue.

[katre]

Trying to get Bazel to directly support multiple values for --platforms is semantically very confusing, so we've made the decision to leave the specific semantics up to rule authors, who know their users' needs and ecosystems better.

To that end, my concrete recommendations are:

1. Rule authors should add platform and/or platforms attributes to appropriate rules, using the features from https://github.com/bazelbuild/proposals/blob/main/designs/2023-06-08-standard-platform-transitions.md, once we get those released (we're setting up the release process for that now, please imagine the sound of bureaucracy happening in the background).
2. Rule authors can also add custom flags for multi-platform support, like --android_platforms, where that makes sense (for example, Android uses this to create a fat APK from a single android_binary target).
3. Bazel should fix the --platforms flag to be less confusing (The platforms flag should be single-valued #19807):
   1. Make the existing multi-valued flag single-valued, but leave the name.
   2. Actually rename the flag to --platform (flag migrations are hard, sorry), leaving --platforms as a deprecated legacy name.
   3. Actually remove --platforms.

[Silic0nS0ldier]

I think I spied a regression in the --platforms invalid options handling due to 81acd2a#diff-a437b5091032394bcb6af037a2c2ffc063f330a9092144f498edb2feda14d782R252.

The check is performed in PlatformConfiguration.reportInvalidOptions which uses PlatformOptions, however the instance it receives has .platforms already normalised so extraneous platforms are silently ignored (previously a warning was emitted).

[katre]

Yes, there's a lot of confusion about the right way to handle this.