Keep separate build parameters for host and target #7164
Conversation
`BuildParameters` to date explicitly mentioned both host and target triples, but assumed you can use the same toolchain for both, which is not true. Additionally, it would assume that all other build parameters can be shared between the host and target, like debug info format, build configuration, sanitizers etc. That's not true either, one would use CodeView for build tools on Windows when compiling targets with DWARF for Linux, or build macros in release mode for performance when cross-compiling code that uses debug configuration itself.
|
@swift-ci test |
|
@swift-ci test |
|
@swift-ci test |
|
@swift-ci test |
|
@swift-ci test |
|
@swift-ci test |
|
@swift-ci test |
IMO we're trading one confusion for another, the main question is which is more confusing, as we're mixing up triple targets and build system targets in the same context. For example, we have |
|
@swift-ci test |
|
I don't have a strong opinion, but looking at our CLI, we currently are using these terms here:
So at least "build tools" seems well established at this point. |
Right, and The main question is about "target" naming. |
|
IMO |
|
IDK, at a glance it doesn't seem obvious whether To make things worse, our build plan and package graph targets don't even correspond to actual llbuild targets either. 🥲 |
|
@swift-ci test windows |
|
After sleeping on it, I'm leaning towards |
|
@swift-ci test |
|
@swift-ci test windows |
|
@swift-ci test |
|
@swift-ci test windows |
…xd/split-build-parameters # Conflicts: # Sources/Build/BuildDescription/SwiftTargetBuildDescription.swift # Tests/BuildTests/BuildPlanTests.swift
|
@swift-ci test |
|
@swift-ci test windows |
|
regardless of naming, im super excited to see this land! |
|
Thanks! It's early to celebrate, this one's technically an NFC. We still need to sort out things in |
…the target Since swiftlang#7164, dependencies of command plugins are once again being built for the _target_ rather than the host. This causes problem when cross compiling because the host needs to be able to run the plugin dependencies, but finds target binaries instead. This problem was fixed before in swiftlang#6791 by forcing command plugin dependencies to be built for the host by overriding the default build parameters in swiftTool.createBuildSystem(). The same solution still works in this commit, but a better long-term option would be to rework BuildOperation.plan() to handle command plugin dependencies specially, as it already does for build plugin dependencies. At present, BuildOperation.plan calls graph.invokeBuildToolPlugins to process sources. invokeBuildToolPlugins finds all build tool dependecies and builds them separately, using a specially-created BuildOperation instance: https://github.com/apple/swift-package-manager/blob/34efc0bfe9d40d9a019644ac8fcd0b852c491dfe/Sources/SPMBuildCore/Plugins/PluginInvocation.swift#L409 There is no equivalent step for command plugin dependencies, so they are built for the host architecture. Ideally we should rework BuildOperation.plan to build command and build plugin dependencies in the same way.
…the target Since swiftlang#7164, dependencies of command plugins are once again being built for the _target_ rather than the host. This causes problem when cross compiling because the host needs to be able to run the plugin dependencies, but finds target binaries instead. This problem was fixed before in swiftlang#6791 by forcing command plugin dependencies to be built for the host by overriding the default build parameters in swiftTool.createBuildSystem(). The same solution still works in this commit, but a better long-term option would be to rework BuildOperation.plan() to handle command plugin dependencies specially, as it already does for build plugin dependencies. At present, BuildOperation.plan calls graph.invokeBuildToolPlugins to process sources. invokeBuildToolPlugins finds all build tool dependecies and builds them separately, using a specially-created BuildOperation instance: https://github.com/apple/swift-package-manager/blob/34efc0bfe9d40d9a019644ac8fcd0b852c491dfe/Sources/SPMBuildCore/Plugins/PluginInvocation.swift#L409 There is no equivalent step for command plugin dependencies, so they are built for the host architecture. Ideally we should rework BuildOperation.plan to build command and build plugin dependencies in the same way.
…the target Since swiftlang#7164, dependencies of command plugins are once again being built for the _target_ rather than the host. This causes problem when cross compiling because the host needs to be able to run the plugin dependencies, but finds target binaries instead. This problem was fixed before in swiftlang#6791 by forcing command plugin dependencies to be built for the host by overriding the default build parameters in swiftTool.createBuildSystem(). The same solution still works in this commit, but a better long-term option would be to rework BuildOperation.plan() to handle command plugin dependencies specially, as it already does for build plugin dependencies. At present, BuildOperation.plan calls graph.invokeBuildToolPlugins to process sources. invokeBuildToolPlugins finds all build tool dependecies and builds them separately, using a specially-created BuildOperation instance: https://github.com/apple/swift-package-manager/blob/34efc0bfe9d40d9a019644ac8fcd0b852c491dfe/Sources/SPMBuildCore/Plugins/PluginInvocation.swift#L409 There is no equivalent step for command plugin dependencies, so they are built for the host architecture. Ideally we should rework BuildOperation.plan to build command and build plugin dependencies in the same way.
…the target (#7280) Always build command line plugin dependencies for the host triple. ### Motivation: Since #7164, dependencies of command plugins are once again being built for the _target_ rather than the host. This causes problem when cross compiling because the host needs to be able to run the plugin dependencies, but finds target binaries instead. This problem was fixed before in #6791 by forcing command plugin dependencies to be built for the host by overriding the default build parameters in swiftTool.createBuildSystem(). The same solution still works in this commit, but a better long-term option would be to rework BuildOperation.plan() to handle command plugin dependencies specially, as it already does for build plugin dependencies. ### Modifications: At present, BuildOperation.plan calls graph.invokeBuildToolPlugins to process sources. invokeBuildToolPlugins finds all build tool dependecies and builds them separately, using a specially-created BuildOperation instance: https://github.com/apple/swift-package-manager/blob/34efc0bfe9d40d9a019644ac8fcd0b852c491dfe/Sources/SPMBuildCore/Plugins/PluginInvocation.swift#L409 There is no equivalent step for command plugin dependencies, so they are built for the host architecture. Ideally we should rework BuildOperation.plan to build command and build plugin dependencies in the same way. This commit forces all plugin dependencies to be built for the host - this is similar to what was done in #6791 and #7273. ### Testing: An integration test checks that any targets depended on by a command plugin are built for the host, not for the target. * A new CommandPluginTestStub plugin has a dependency on a target executable which will be built automatically when the plugin is run. The test checks that the dependency is built for the host architecture, no matter which target architecture is selected using '--triple'. * The plugin also asks SwiftPM to build the 'placeholder' main target. The test checks that the dependency is built for the target architecture. The test is restricted to macOS because we can be sure of having a viable cross-compilation environment (arm64 to x86_64 and vice versa). The standard Linux build environments can't cross compile to other architectures. ### Result: Command plugins can be used again when cross-compiling.
Unblocks support for macros in cross-compilation implemented in #7118.
Motivation:
So far we used a single
BuildParametersvalue for the whole build graph, which explicitly mentioned both host and target triples, but assumed you can use the same toolchain for both, which is not true. Additionally, it would assume that all other build parameters can be shared between the host and target, like debug info format, build configuration, sanitizers etc. That's not true either, one would use CodeView for build tools on Windows when cross-compiling products with DWARF for Linux, or build macros in release mode for performance while cross-compiling code that uses debug build configuration.Modifications:
Removed
hostTripleandtargetTriplefromBuildParameters, keeping only a singletripleproperty ofTripletype in it. InSwiftToolandBuildPlanwe'll use two separatedestinationBuildParametersandtoolsBuildParametersinstead. The latter will pick one of the twoBuildParametersvalues for specific target and product build descriptions.Result:
We can compile build tools for the host, while cross-compiling end products for the target, in the same build graph. This also removes one of big roadblocks in folding build tool and package plugins into the build graph, instead of building and invoking them separately outside of the build graph.