Refactor Incompatible Target Skipping #14096

philsc · 2021-10-12T01:00:34Z

This patch reworks how Incompatible Target Skipping is implemented.

The biggest change is that incompatibility is now checked earlier. This
allows us to avoid evaluating dependencies (such as toolchains) in
situations where a target is "directly" incompatible. "Direct incompatibility"
refers to incompatibility due to a target's target_compatible_with value.

Moving the incompatibility checking earlier had the undesired effect of
visibility restrictions being ignored on incompatible targets. This is
tracked in #16044. As per
#14096 (comment),
we will fix it in a separate patch.

Fixes #12897.

src/main/java/com/google/devtools/build/lib/skyframe/ConfiguredTargetFunction.java

gregestren

Left a whole bunch of comments! Thanks for all this effort. Looking forward to working through this with you.

src/main/java/com/google/devtools/build/lib/skyframe/ConfiguredTargetFunction.java

.../java/com/google/devtools/build/lib/analysis/constraints/RuleContextConstraintSemantics.java

src/main/java/com/google/devtools/build/lib/skyframe/ConfiguredTargetFunction.java

philsc · 2022-01-20T00:36:29Z

Thanks for the feedback @gregestren , I really appreciate it!
I'll try to get back to incorporating it next week.

gregestren · 2022-01-20T00:49:43Z

No worries. Thanks for the update!

Just FYI there's been some unrelated commits on ConfiguredTargetFunction to make it more efficient recently. If you run into any issues merging with head please let me know.

keith · 2022-04-25T19:08:32Z

I'd love to see this moving again! We hit this in the swift rules today

gregestren · 2022-04-25T19:35:49Z

FYI @keith : #15271 (comment)

philsc · 2022-04-28T23:18:17Z

@keith , yeah. Sorry about the delay. Now that robotics season is over, I'll get back to it.

philsc · 2022-07-20T05:46:46Z

Fair enough. I spent a couple of hours trying a couple of things, but didn't really get anywhere. The visibility code is more complicated than I expected. I would like to spend another couple hours on it to see if I can figure it out. But otherwise I am okay with creating a ticket for fixing the visibility/incompatibility problem and merging this as-is.

gregestren · 2022-07-20T14:53:56Z

Fair enough. I spent a couple of hours trying a couple of things, but didn't really get anywhere. The visibility code is more complicated than I expected. I would like to spend another couple hours on it to see if I can figure it out. But otherwise I am okay with creating a ticket for fixing the visibility/incompatibility problem and merging this as-is.

Sounds good to me.

philsc · 2022-07-27T21:14:05Z

I won't have time this week to experiment further with moving visibility checking. So I'll do that next week.

That being said, there is a question you could answer for me in case I still don't get anywhere.
I will file a Github issue to keep track of the visibility/incompatibility strangeness. Where should I document that behaviour? Is https://docs.bazel.build/versions/main/platforms.html#skipping-incompatible-targets a good place for this?

gregestren · 2022-07-27T21:21:27Z

I think that's good for a start. We could always expend as needed, but no need to overdo documentation at first.

philsc · 2022-08-04T07:03:47Z

I will keep looking into fixing the visibility issue, but as a separate, future effort. I filed #16044 and referenced it in the code and in the docs. Let me know if that looks reasonable.

katre

This looks good to me, I'll wait for Greg's review as well.

gregestren

LGTM to meet too.

The latest refactor unintentionally made it so you can list the same constraint multiple times in the `target_compatible_with` attribute. It was unintentional, but actually greatly simplifies a discussion point on bazelbuild/bazel-skylib#381. That skylib patch aims to make it easier to write non-trivial `target_compatible_with` expressions. For example, to express that something is compatible with everything except for Windows, one can use the following: foo_binary( name = "bin", target_compatible_with = select({ "@platforms//os:windows": ["@platforms//:incomaptible"], "//conditions:default: [], }), ) The skylib patch aims to reduce that to: foo_binary( name = "bin", target_compatible_with = compatibility.none_of("@platforms//os:windows"), ) This works fine on its own, but a problem arises when these expressions are composed. For example, a macro author might write the following: def foo_wrapped_binary(name, target_compatible_with = [], **kwargs): foo_binary( name = name, target_compatible_with = target_compatible_with + select({ "@platforms//os:none": ["@platforms//:incompatible"], "//conditions:default": [], }), ) A macro author should be able to express their own incompatibility while also honouring user-defined incompatibility. It turns out that my latest refactor (bazelbuild#14096) unintentionally made this work. This happened because we now check for incompatibility before we perform a lot of sanity checks. That's also one of the reasons that visibility restrictions are not honoured for incomaptible targets at the moment (bazelbuild#16044). Without the deduplicating behaviour, macro authors are stuck with either not allowing composition, or having to create unique incompatible constraints for every piece in a composed `target_compatible_with` expression. This patch adds a test to validate the deduplicating behaviour to cement it as a feature. Unfortunately, this means that `target_compatible_with` behaves differently from other label list attributes. For other label list attributes, bazel complains when labels are duplicated. For example, the following BUILD file: py_library( name = "lib", ) py_binary( name = "bin", srcs = ["bin.py"], deps = [ | ":lib", | ":lib", ], ) will result in the following error: $ bazel build :bin INFO: Invocation ID: 3d8345a4-2e76-493e-8343-c96a36185b52 ERROR: /home/jenkins/repos/test_repo/BUILD:41:10: Label '//:lib' is duplicated in the 'deps' attribute of rule 'bin' ERROR: error loading package '': Package '' contains errors INFO: Elapsed time: 2.514s INFO: 0 processes. FAILED: Build did NOT complete successfully (1 packages loaded)

The latest refactor unintentionally made it so you can list the same incompatible constraint multiple times in the `target_compatible_with` attribute. It was unintentional, but actually greatly simplifies a discussion point on bazelbuild/bazel-skylib#381. That skylib patch aims to make it easier to write non-trivial `target_compatible_with` expressions. For example, to express that something is compatible with everything except for Windows, one can use the following: foo_binary( name = "bin", target_compatible_with = select({ "@platforms//os:windows": ["@platforms//:incomaptible"], "//conditions:default: [], }), ) The skylib patch aims to reduce that to: foo_binary( name = "bin", target_compatible_with = compatibility.none_of("@platforms//os:windows"), ) This works fine on its own, but a problem arises when these expressions are composed. For example, a macro author might write the following: def foo_wrapped_binary(name, target_compatible_with = [], **kwargs): foo_binary( name = name, target_compatible_with = target_compatible_with + select({ "@platforms//os:none": ["@platforms//:incompatible"], "//conditions:default": [], }), ) A macro author should be able to express their own incompatibility while also honouring user-defined incompatibility. It turns out that my latest refactor (#14096) unintentionally made this work. This happened because we now check for incompatibility before we perform a lot of sanity checks. That's also one of the reasons that visibility restrictions are not honoured for incomaptible targets at the moment (#16044). Without the deduplicating behaviour, macro authors are stuck with either not allowing composition, or having to create unique incompatible constraints for every piece in a composed `target_compatible_with` expression. This patch adds a test to validate the deduplicating behaviour to cement it as a feature. Unfortunately, this means that `target_compatible_with` behaves differently from other label list attributes. For other label list attributes, bazel complains when labels are duplicated. For example, the following BUILD file: py_library( name = "lib", ) py_binary( name = "bin", srcs = ["bin.py"], deps = [ | ":lib", | ":lib", ], ) will result in the following error: $ bazel build :bin INFO: Invocation ID: 3d8345a4-2e76-493e-8343-c96a36185b52 ERROR: /home/jenkins/repos/test_repo/BUILD:41:10: Label '//:lib' is duplicated in the 'deps' attribute of rule 'bin' ERROR: error loading package '': Package '' contains errors INFO: Elapsed time: 2.514s INFO: 0 processes. FAILED: Build did NOT complete successfully (1 packages loaded) Closes #16274. PiperOrigin-RevId: 474747867 Change-Id: Iacc4de12ce90176d803e0bc9e695d4ec14603449

The latest refactor unintentionally made it so you can list the same incompatible constraint multiple times in the `target_compatible_with` attribute. It was unintentional, but actually greatly simplifies a discussion point on bazelbuild/bazel-skylib#381. That skylib patch aims to make it easier to write non-trivial `target_compatible_with` expressions. For example, to express that something is compatible with everything except for Windows, one can use the following: foo_binary( name = "bin", target_compatible_with = select({ "@platforms//os:windows": ["@platforms//:incomaptible"], "//conditions:default: [], }), ) The skylib patch aims to reduce that to: foo_binary( name = "bin", target_compatible_with = compatibility.none_of("@platforms//os:windows"), ) This works fine on its own, but a problem arises when these expressions are composed. For example, a macro author might write the following: def foo_wrapped_binary(name, target_compatible_with = [], **kwargs): foo_binary( name = name, target_compatible_with = target_compatible_with + select({ "@platforms//os:none": ["@platforms//:incompatible"], "//conditions:default": [], }), ) A macro author should be able to express their own incompatibility while also honouring user-defined incompatibility. It turns out that my latest refactor (bazelbuild#14096) unintentionally made this work. This happened because we now check for incompatibility before we perform a lot of sanity checks. That's also one of the reasons that visibility restrictions are not honoured for incomaptible targets at the moment (bazelbuild#16044). Without the deduplicating behaviour, macro authors are stuck with either not allowing composition, or having to create unique incompatible constraints for every piece in a composed `target_compatible_with` expression. This patch adds a test to validate the deduplicating behaviour to cement it as a feature. Unfortunately, this means that `target_compatible_with` behaves differently from other label list attributes. For other label list attributes, bazel complains when labels are duplicated. For example, the following BUILD file: py_library( name = "lib", ) py_binary( name = "bin", srcs = ["bin.py"], deps = [ | ":lib", | ":lib", ], ) will result in the following error: $ bazel build :bin INFO: Invocation ID: 3d8345a4-2e76-493e-8343-c96a36185b52 ERROR: /home/jenkins/repos/test_repo/BUILD:41:10: Label '//:lib' is duplicated in the 'deps' attribute of rule 'bin' ERROR: error loading package '': Package '' contains errors INFO: Elapsed time: 2.514s INFO: 0 processes. FAILED: Build did NOT complete successfully (1 packages loaded) Closes bazelbuild#16274. PiperOrigin-RevId: 474747867 Change-Id: Iacc4de12ce90176d803e0bc9e695d4ec14603449

EXPERIMENT: Investigate objc_library and target_compatible_with

da36c56

google-cla bot added the cla: yes label Oct 12, 2021

philsc added 12 commits October 20, 2021 20:39

Add more tests

793f85d

Add another test

63f5339

Add some comments

f2b0191

Add another test

1361941

First attempt at cleanup

76a70ec

clean up some more

8cb3cd8

minor cleanup

de7bfb0

Clean up some more

00a482e

A little more cleanup

5984a1b

Merge remote-tracking branch 'upstream/master' into HEAD

51c924c

Fix merge conflicts

51e79d6

clean up TestParams

9601882

philsc commented Nov 23, 2021

View reviewed changes

src/main/java/com/google/devtools/build/lib/skyframe/ConfiguredTargetFunction.java Outdated Show resolved Hide resolved

gregestren reviewed Nov 23, 2021

View reviewed changes

gregestren mentioned this pull request Nov 29, 2021

Bazel wants full dependency graph for incompatible targets - why aren't they just skipped? #14132

Closed

philwo force-pushed the master branch from 26cb401 to 168b89b Compare December 2, 2021 18:07

gregestren mentioned this pull request Dec 3, 2021

Cannot use target_compatible_with to ignore MacOS only targets #12897

Closed

philsc mentioned this pull request Mar 8, 2022

Bazel does not skip toolchain resolution for incompatible targets (target_compatible_with with --platform) #14867

Closed

philsc mentioned this pull request Apr 16, 2022

Aspects get falsely evaluated against incompatible targets #15271

Closed

philsc added 4 commits June 11, 2022 18:58

Merge commit '28c09f29bc' into HEAD

ce147fc

Merge commit '3fec0e1870' into HEAD

58fd3d6

Merge commit '33b248366b' into HEAD

055cd7f

Merge commit '3badca3f7e' into HEAD

024835c

Incorporate more feedback

16f098b

philsc marked this pull request as ready for review July 20, 2022 05:47

sgowroji added team-Configurability platforms, toolchains, cquery, select(), config transitions awaiting-review PR is awaiting review from an assigned reviewer labels Jul 20, 2022

Merge remote-tracking branch 'upstream/master' into HEAD

21a5fd1

philsc mentioned this pull request Aug 4, 2022

Incompatible Target Skipping ignores visibility restrictions #16044

Closed

philsc added 3 commits August 3, 2022 23:47

Reference the bug report

d0974ca

format

607ab12

add comment

ab7bdae

philsc requested a review from fweikert as a code owner August 4, 2022 06:52

katre approved these changes Aug 4, 2022

View reviewed changes

gregestren approved these changes Aug 5, 2022

View reviewed changes

gregestren added awaiting-PR-merge PR has been approved by a reviewer and is ready to be merge internally and removed awaiting-review PR is awaiting review from an assigned reviewer labels Aug 5, 2022

copybara-service bot closed this in 72787a1 Aug 9, 2022

philsc deleted the unreviewed/objc_library-b branch August 9, 2022 22:20

sgowroji removed the awaiting-PR-merge PR has been approved by a reviewer and is ready to be merge internally label Aug 10, 2022

philsc mentioned this pull request Aug 14, 2022

Using --cpu=darwin_x86_64 in platform_mappings triggers unnecessary toolchain evaluation #16099

Open

philsc mentioned this pull request Sep 14, 2022

Add lib/compatibility.bzl bazelbuild/bazel-skylib#381

Open

philsc mentioned this pull request Sep 14, 2022

Deduplicate incompatible target_compatible_with labels #16274

Closed

gregestren mentioned this pull request Mar 20, 2023

Weird behavior with alias and target_compatible_with #17663

Closed

philsc mentioned this pull request Apr 10, 2023

Bazel crash when target_compatible_with doesn't have a default entry. #18021

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactor Incompatible Target Skipping #14096

Refactor Incompatible Target Skipping #14096

philsc commented Oct 12, 2021 •

edited

Loading

gregestren left a comment

philsc commented Jan 20, 2022

gregestren commented Jan 20, 2022

keith commented Apr 25, 2022

gregestren commented Apr 25, 2022

philsc commented Apr 28, 2022

philsc commented Jul 20, 2022

gregestren commented Jul 20, 2022

philsc commented Jul 27, 2022 •

edited

Loading

gregestren commented Jul 27, 2022

philsc commented Aug 4, 2022 •

edited

Loading

katre left a comment

gregestren left a comment

Refactor Incompatible Target Skipping #14096

Refactor Incompatible Target Skipping #14096

Conversation

philsc commented Oct 12, 2021 • edited Loading

gregestren left a comment

Choose a reason for hiding this comment

philsc commented Jan 20, 2022

gregestren commented Jan 20, 2022

keith commented Apr 25, 2022

gregestren commented Apr 25, 2022

philsc commented Apr 28, 2022

philsc commented Jul 20, 2022

gregestren commented Jul 20, 2022

philsc commented Jul 27, 2022 • edited Loading

gregestren commented Jul 27, 2022

philsc commented Aug 4, 2022 • edited Loading

katre left a comment

Choose a reason for hiding this comment

gregestren left a comment

Choose a reason for hiding this comment

philsc commented Oct 12, 2021 •

edited

Loading

philsc commented Jul 27, 2022 •

edited

Loading

philsc commented Aug 4, 2022 •

edited

Loading