Implement system package mode #18778
Conversation
andrewrk
force-pushed
the
system-package-mode
branch
from
03934cb to
9aeaa1a
Compare
andrewrk
added
breaking
andrewrk
force-pushed
the
system-package-mode
branch
from
9aeaa1a to
826f5e3
Compare
andrewrk
force-pushed
the
system-package-mode
branch
from
a2347c7 to
bd07968
Compare
* New --help section * Add b.systemLibraryOption * Rework the build runner CLI logic a bit
These are advanced options that make it possible to simultaneously target the "host" computer, while overriding assumptions about the host computer such as what CPU features are available, or what OS version range to target. This is useful only for the case of integrating with system package manager builds which want to pretend the host is also the target.
This value is very likely incorrect. When glibc_version is provided but no explicit ABI is provided, use the string "gnu" instead.
This also makes a long-overdue change of extracting common state from Build into a shared Graph object. Getting the semantics right for these flags turned out to be quite tricky. In the end it works like this: * The override only happens when the target is fully native, with no additional query parameters, such as versions or CPU features added. * The override affects the resolved Target but leaves the original Query unmodified. * The "is native?" detection logic operates on the original, unmodified query. This makes it possible to provide invalid host target information, causing confusing errors to occur. Don't do that. There are some minor breaking changes to std.Build API such as the fact that `b.zig_exe` is now moved to `b.graph.zig_exe`, as well as a handful of other similar flags.
On second thought, let's keep a bunch of these flags how they already were. Partial revert of the previous commit.
Before it was named "library" inconsistently. Now the CLI args are -fsys=[name] and -fno-sys=[name] and it is a more general-purpose "system integration" which could be a library name or perhaps a project name such as "ffmpeg" or a binary such as "nasm".
This allows a `zig build` command to specify intention to create a release build, regardless of what per-project options exist. It also allows the command to specify a "preferred optimization mode", which is chosen if the project itself does not choose one (in other words, the project gets first choice). If neither the build command nor the project specify a preferred release mode, an error occurs.
This prevents package fetching and enables system_package_mode in the build system which flips the defaults for system integrations.
This is how assertions work in zig.
Build manifest files support `lazy: true` for dependency sections. This causes the auto-generated dependencies.zig to have 2 more possibilities: 1. It communicates whether a dependency is lazy or not. 2. The dependency might be acknowledged, but missing due to being lazy and not fetched. Lazy dependencies are not fetched by default, but if they are already fetched then they are provided to the build script. The build runner reports the set of missing lazy dependenices that are required to the parent process via stdout and indicates the situation with exit code 3. std.Build now has a `lazyDependency` function. I'll let the doc comments speak for themselves: When this function is called, it means that the current build does, in fact, require this dependency. If the dependency is already fetched, it proceeds in the same manner as `dependency`. However if the dependency was not fetched, then when the build script is finished running, the build will not proceed to the make phase. Instead, the parent process will additionally fetch all the lazy dependencies that were actually required by running the build script, rebuild the build script, and then run it again. In other words, if this function returns `null` it means that the only purpose of completing the configure phase is to find out all the other lazy dependencies that are also required. It is allowed to use this function for non-lazy dependencies, in which case it will never return `null`. This allows toggling laziness via build.zig.zon without changing build.zig logic. The CLI for `zig build` detects this situation, but the logic for then redoing the build process with these extra dependencies fetched is not yet implemented.
This makes `zig build` notice when lazy dependencies were missing, fetch them, and then rebuild the build runner and run it again.
After parsing diagnostics files from clang we don't have any more use for those tmp files. Delete them to reduce clutter and disk usage.
Pass the required lazy dependencies from the build runner to the parent process via a tmp file instead of stdout. I'll reproduce this comment to explain it: The `zig build` parent process needs a way to obtain results from the configuration phase of the child process. In the future, the make phase will be executed in a separate process than the configure phase, and we can then use stdout from the configuration phase for this purpose. However, currently, both phases are in the same process, and Run Step provides API for making the runned subprocesses inherit stdout and stderr which means these streams are not available for passing metadata back to the parent. Until make and configure phases are separated into different processes, the strategy is to choose a temporary file name ahead of time, and then read this file in the parent to obtain the results, in the case the child exits with code 3. This commit also extracts some common logic from the loop that rebuilds the build runner so that it does not run again when the build runner is rebuilt.
This makes it easier to debug and avoids a false positive compile error in the build script.
Prevents unnecessary depedency on networking when bootstrapping zig.
andrewrk
force-pushed
the
system-package-mode
branch
from
b324ff7 to
3dad731
Compare
Does this mean the default should be switched? Maybe call the other way "side effect dependency" or something? (is that why you would want a dependency you don't use?) |
|
Dependencies do not have side effects. The use case is conditional dependencies - the configuration values chosen decides the set of actual dependencies used. |
|
Can you comment on this: if a build.zig file imports a dependency via My assumption would be no, and that if you have a dependency which needs any additional build-time logic to be imported like that, it becomes a viral dependency (if you use it, your dependency also cannot be optional)? I'm not necessarily opposed to that, but want to understand the limitations here if any. The reason I am asking is because in Mach currently, we use this to abstract some of the platform-specific nuances of desktop/mobile/web, where the former has a others (@MasterQ32 ?) may have other reasons to need |
|
@slimsag This should be supportable if the That way it wouldn't even be an issue if the set of modules imported for build-time logic depended on |
|
@slimsag as it stands in this branch, direct Regardless of what changes here I would encourage you to explore module based abstractions instead of inversion of control flow based abstractions. Happy to work with you on any limitations you encounter. But I'll tinker a bit and see if we can have the ability to import lazy dependencies build logic. I don't want to do what @rohlem is suggesting though so it would be a breaking change either way. |
|
Alright, I spent some more time investigating these follow-up issues, but ultimately I decided to move forward with this branch as-is, and address them later. This is certainly a step forward. Let's try this out for a while and then here are some follow-up issues to tackle: deprecating dependency() and making lazyDependency() the preferred APII think this might be a good idea to do in the future, since using However as it stands, this branch is actually not very invasive; the breaking changes are minimal. I think it would be a good idea to solve the other problems mentioned below before inflicting unnecessary breaking changes on build.zig users. direct
|
Sorry I missed this question earlier. I have not added test coverage for this yet, but it is not viral. When a lazy dependency is fetched, its own non-lazy dependencies will all be fetched and therefore |
|
@andrewrk Regarding
Could you clarify what you mean by "having the dependency take over the build script logic for the parent package"? Because as both a library author and user, being able to expose helper functions that compose and orchestrate build steps seems like one of the main benefits of being able to For non-trivial packages I would even say |
I think the proliferation of zig modules in the wild will the blur this line to the point of not really mattering.
I don't agree with this line at all and to me what it's calling for is a perfect example of designing around a local maximum. |
andrewrk
added
the
release notes
|
Partially reverted in #19684. |
Synopsis
Makes the zig build system significantly more friendly to system package maintainers by introducing System Integration Options.
Closes #14281
Closes #14597
Demo
Using groovebasin as an example project:
New API for declaring optional system library integration
This diff plus some similar diffs in the project's dependency tree.
New help section
There is a new
--helpsection:Using the system integration options
The result is using baseline CPU due to the
--host-targetflag. Note that--host-targetis always a valid option even if the build script does not expose any standard target option.Now, re-run the command but removing
-fsys=z:new release option
Lazy Dependencies
This makes the dependency only get fetched if it is actually used. The build runner will be rebuilt if any missing lazy dependencies are encountered.
Error for using dependency() instead of lazyDependency()
It's allowed to do the reverse - lazyDependency() when the manifest file does not mark it as lazy.
It's probably best practice to always use lazyDependency() in build.zig.
avoid fetching in system mode
Full List of Breaking Changes
Predicted breakage due to these changes: little to none
b.zig_exe=>b.graph.zig_exeb.env_map=>b.graph.env_mapb.search_prefixes: it's now an ArrayListUnmanaged.addSearchPrefixis unchangedb.global_cache_root=>b.graph.global_cache_rootb.cache=>b.graph.cacheMerge Checklist
--release--system [dir]command