Post-build script execution

[vosen]

Currently, cargo execute scrips before the build starts with the build field. I propose renaming build to pre_build and adding post_build (which would run after every successful build). It's useful for general postprocessing: running executable packers, zipping files, copying stuff, logging, etc.

[alexcrichton]

This will likely happen as part of an implementation of cargo install, but cargo build is not meant to be a general purpose build system for these sorts of applications.

[DiamondLovesYou]

What about integration tests? For example, PNaCl modules can be run on the host machine, but they have to be translated into a NaCl beforehand and then run with sel_ldr (from the NaCl SDK).

[jan-hudec]

@DiamondLovesYou, would #1411 work for the tests?

[caitp]

A "post_build" script would be useful for things like creating a MacOS bundle, rather than having a separate script to do that work separately. "install" could do this too, but maybe not as useful for testing there?

[vvanders]

+1 For rust projects that linked dynamically from other languages being able to deploy(move) the .so after a successful build would be nice.

[tbelaire]

This would be useful for running

arm-none-eabi-objcopy -O binary ${TARGET}.elf ${TARGET}.gba
gbafix ${TARGET}.gba

To fix up the headers and such after building my project.

[Boscop]

+1 for post build scripts

[mkroman]

This will likely happen as part of an implementation of cargo install, but cargo build is not meant to be a general purpose build system for these sorts of applications.

It doesn't have to mean that cargo will be a general purpose build system.
An example where a post-build action could be used for cargo is triggering a build of a crate in a sub-directory once the main crate has been built.

Crate B depends on crate A, and is in a sub-directory of crate A, and it automatically gets built every time crate A has been built.

[Boscop]

But what if crate B also depends on sub crates C and D? Shouldn't the build of dependencies be caused by building the crate that depends on them in a "by need" manner?
I would say the post build command should not involve building any related crates or calling cargo (can be easily enforced), only for post processing of the lib/exe that was just built. To keep it compartmentalized / clean.

[JinShil]

This would be useful for embedded systems programming. I often run arm-none-eabi-size to verify my binary size with each build, and occasionally other utilities from binutils to verify symbol tables, etc...

[steveklabnik]

Don't forget that there's nothing that says you have to use a non-Rust solution for these kinds of things; any "cargo-foo" executable on your $PATH works as "cargo foo", and can be written in any language, including Rust. Your custom command could invoke "cargo build" as a step first and then do anything.

On Oct 29, 2016, 13:35 -0400, Mike notifications@github.com, wrote:

This would be useful for embedded systems programming. I often run arm-none-eabi-size to verify my binary size with each build, and occasionally other utilities from binutils to verify symbol tables, etc...

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub (#545 (comment)), or mute the thread (https://github.com/notifications/unsubscribe-auth/AABsimputy3ucYbBxCoVRqmOdAF8GQPnks5q44P0gaJpZM4CgUcP).

[JinShil]

Don't forget that there's nothing that says you have to use a non-Rust solution for these kinds of things; any "cargo-foo" executable on your $PATH works as "cargo foo", and can be written in any language, including Rust. Your custom command could invoke "cargo build" as a step first and then do anything.

Yes, that is what I, and probably others, are already doing. It would just be more convenient to integrate this into the already familiar cargo build.

[ssokolow]

Not to mention that, in my experience, if you try to enforce your will too strongly, you start to get behaviours similar to what has been standard in the home inkjet printer market for well over a decade now.

(Where the first thing you see when you open the box is a slip that warns you, in big letters, to never plug the printer in until after you've installed the software off their CD, because Microsoft's automated driver installation workflow doesn't allow them to install their fancy, tray-resident, bloated, talking print drivers and their official position is that, if Microsoft's installer has had a chance to run, your OS is now broken and must be repaired first.)

Better to have an official way to run a post-build script than to see everyone and their dog reinventing broken wheels... especially since, as a Stubborn Person™, I can easily foresee these horrors:

• Crates which bail out early and insist that, even if they're being used as a dependency, you must run their own special cargo mybuild wrapper instead of cargo build (possibly not as an inherent thing, but as a result of depending on something else by the same author which has that build requirement).
• Developers recognizing the ugliness of the former situation and producing a "better" solution by hacking (subtly buggy) post-builds onto cargo build using pre-build scripts which leave something running in the background to trigger the post-build using inotify or equivalent.

(Also, keep in mind that, as Rust sees more corporate uptake, we'll see more "I'm too busy to learn the 'proper' way, so I'll just hack something in" WTFs. That's just a basic rule of programming in general.)

[Kixunil]

In embedded development it's useful to run objcopy to produce hex file or something similar. Adding this to post-build script would be much cleaner than reinventing wheel with my own shell script.

[yasammez]

This would be useful for stuff like

SignTool sign /v /s PrivateCertStore /n MyTestCrt /t http://timestamp.verisign.com/scripts/timstamp.dll driver.sys

when developing Windows drivers in Testmode.

[fschutt]

This would be useful for packaging debian applications, especially when combined with configuration
Like a configuration deploy which builds the crate in release mode, then creates the folders needed for a package (control files, etc.) and runs dpkg on it.

[kornelski]

cargo install touches files outside the current directory, so it doesn't feel like a proper post-build step. If I want to strip the built binary, would I have to work on a binary already copied to /bin?

[Boscop]

Why not do this:
If there is a postbuild.rs file, it gets executed after build. (similar to the build.rs file before.)
It would make sense to execute the script also if the build failed. The build status can be passed to it in an env var.

[Kixunil]

@Boscop I like that idea but I worry that executing another binary every compilation might slow down iterative development. Maybe using postbuild-success.rs for only success, postbuild-failure.rs for only failure and postbuild.rs for both?

[Boscop]

I think there won't really be a slowdown compared to having different scripts for success/failure because if your postbuid.rs script checks the build status first and then returns if it's not success/failure, it won't really slow things down. The postbuild.rs script won't have to be recompiled every time, only when it changes. So calling it will just incur the cost of starting an executable that immediately returns.
Having one postbuild.rs script follows the KISS principle, most projects won't even have a postbuild.rs script, and for those that will have one, I think having one script fulfills their needs.
If they notice a slowdown, we can always introduce the option to replace the postbuild.rs script by the other two later. But let's first have just the postbuild.rs and see if that's enough.

[Kixunil]

That sounds reasonable.

[tanis2000]

@Boscop idea of the postbuild.rs script sounds like a very good way of managing post build actions. Even though the general opinion seems to be split between this and avoiding to turn cargo into a general purpose make system, I feel that this won't hurt.

[JoeyAcc]

Not that I'm advocating for it per se, but Cargo already is a general purpose build system due to the cargo plugins (e.g. cargo tree) being able to do whatever they want. That that is merely a user-friendly(ish) facade for "random turing-complete script" doesn't diminish that in any way.

So the real question becomes: How ergonomic do we want to make this general-purpose build system?

[gdennie]

cargo_make seems to implement a flexible build tool...

[Qix-]

It's not a standard thing with cargo and requires a third-party installation (via cargo install cargo-make or something). Yes those tools exist, but getting things to run as part of a dependency chain or something whereby standard Cargo processes are happening, cargo make doesn't help.

To put it another way, if cargo make was the standard way cargo worked with projects and would run them even in nested dependency trees, that'd be a different ballgame. I'm not advocating for that to be the case, though. But it would solve the issue directly.

[leryn1122]

I do think it's neccessary to provide an official hook as post-build scripts, rather then

• write a glue-lang/shell script with no platform portability
• third party implemented cargo-make / cargo-blablabla, it's not a standard process

Two relevant cases I got are,

• manually apple-codesign executables with entitlements after every building, it makes unit test quite exhausting
• package my app as a desired bundle or tarball for distro

Possible implemention could be:

• println!("cargo:post-build=/path/to/post-build.rs") in build.rs
• post-build.rs under project root, as equivalent as build.rs

[gdennie]

Perhaps support for the occurrence of scripts throughout the build process couldbe accommodated an enhanced with an in memory database or file system to facilitate communication or resource sharing between scripts...

pre_build
pre_compile
pre_link
post_build

I can't see where such a thing wouldn't be immediately valuable.

However, isn't there an existing need to sandbox build scripts somehow such as by restricting resource access through a monitoring agent.

[epage]

For myself, I feel like post-build scripts would be an incremental but incomplete improvement that is a dead-end. What we really need is a build orchestrator above cargo for solving these kind of use cases. For example, if you want a universal binary, you need a post-build script that combines the result of two targets.

I wrote more of my thoughts at https://epage.github.io/blog/2023/08/are-we-gui-build-yet/

[nvzqz]

@epage could you please give examples of prior art for such an orchestrator? Do you mean meta build systems like Nix/Bazel/Buck?

[dlight]

@epage an external build system can run a command after the whole build completed, but it can't run after the build of each crate, so it doesn't completely replace a Cargo integration for postbuild scripts. (one use case is to patch each object file after they are built, but before linking)

@nvzqz perhaps he is referring to something like just or cargo-make

[dlight]

I just found cargo-post which seems like a nice workaround.

However it always run the post_build.rs on every successful build (even if there is nothing to do), rather than running only when the output artifact changes.

[epage]

@epage an external build system can run a command after the whole build completed, but it can't run after the build of each crate, so it doesn't completely replace a Cargo integration for postbuild scripts. (one use case is to patch each object file after they are built, but before linking)

All the use cases I've seen are for final artifacts only (bins or SOs). what do you want to do to the rlibs?

[dlight]

@epage right now, indeed, I don't!

But here's an use case: what if I wanted to replace all instances of a certain function call in a crate for something else? (be it a call to another function, or even something more custom)

One idea to implement this is with a custom MIR -> MIR compiler pass, but I'm not sure this is even available and if it is, it's certainly nightly-only with not prospects for stabilization (I guess it would require stable MIR, at least)

But we could operate at a lower level. There are already lower level tools like wasm-snip that replace function bodies with no-ops, to save space if the function is only called by dead code that wasn't eliminated by the optimizer. But that operates on the whole artifact.

But if such a tool could operate on individual crates, it could allow for more fine grained manipulations, for other purposes. Something like: I want that all allocations done in this crate to be replaced by a panic, but I don't want to instrument the code or anything; I could just add a postbuild script to this crate to modify the rlib (or dylib or whatever).

[Kixunil]

After a long time I came to conclusion that make-like software may be really the best for this and not cargo however it needs information about which binaries are built from which sources. It already has an incomplete information in various *.d files. Specifically, the root .d file does not list the dependencies - only the source files from the project itself. I haven't figured out if this is fixable. Without this, incremental development is very messy.

[ssokolow]

After a long time I came to conclusion that make-like software may be really the best for this and not cargo

For the complex cases, perhaps, but it doesn't feel reasonable to require every Cargo integration (eg. VSCode plugins) to add support for inserting a project-specific wrapper around Cargo calls just so you can do stuff to the final artifact like non-automatic codesigning which may need to happen on some targets before something like cargo test will work or renaming libraries for a plugin/extension API that requires it. (That's still a problem, right? I didn't miss a fix for that?)

That feels like it's being disproportionately unfair to, for example, platforms which aren't self-hosting and require either emulation or integration with a remote device such as mobile and bare metal targets.

[gdennie]

Scripts invoked by hooks and provided pre-compiled libraries offering API into the build process can perform all manner of inspection and augmentation of the build throughout. In fact, too much perhaps; consequently, a sandbox, permission system, or a constrained API seems equally essential to manage the scope, at least for scripts of external dependencies. Ultimately, restricting scripts to a predetermined collection of domain specific libraries could very efficient, manage permissions of such scripts, and yet offer an extensive API into the build process.

[ssokolow]

Scripts invoked per their hook and provided domain specific precompiled libraries offer such thingsas an API intothe build process should be sufficient for all manner of mischief in any stage of the build. In fact, too much mischief; consequently, a sandbox or permission system seems equally essential to limit their scope, at least for scripts in dependencies. Ultimately, restricting scripts to a predetermined collection of domain specific libraries should reduce the required compilation effort, provide a means of managing their scope, and offer an extensive API into the build process.

A way to enable that was already proposed and accepted... it's just waiting for someone to implement it:

rust-lang/compiler-team#475

...though it'll probably be for proc macros first since they're much more likely to be pure.

See also #5720

[Kixunil]

@ssokolow oh, sure, I'm not against it. I just happen to have that more complex case and it'd be nice being able to get the correct list of dependencies. (Not a big deal since that is only needed for edge cases now.) Or better said, for my case post-build scripts (to generate man pages, completions...) are not very suitable while for generating binaries they are. So ideally we should have support for both.

You make good points about tests & stuff.

[Systemcluster]

A workaround for some use-cases is using a workspace and using the build.rs of the top-level crate to invoke cargo and build the inner crates.

[leryn1122]

So I argued on the officiality. It could be solved with third party cargo-post, cargo-make mentioned above, sometimes de-normalized in my point of view. An official hook would be the expected solution for complex intergration so far.

[timboudreau]

FWIW, as a workaround I have had some success (I'm building a MacOS xcframework which needs to be consumed by Swift and C code) with the following - this only is practical if your projects are set up as a Cargo workspace:

• Bunch of library projects that implement the functionality
• Static library project that depends on those that should be exposed via a C API which generates a binary to run cbindgen against itself (another thing that post-build scripts would be useful for)
• Another do-nothing project which contains a build.rs that runs cbindgen takes the .a file and the generated bindings and creates or updates the Apple xcframework the C and Swift code depend on

While it's not pretty, and it involves a lot of fishing around in build directories, it does work as a way to have code that can only run after a dependency is built.

I do not think this sort of thing is outside the scope of Cargo's purpose - it is a very common activity to need to post-process a binary or generate some additional artifacts that should only be generated at the conclusion of a successful build - foreign language bindings are the perfect example of that.

[Gronis]

• Another do-nothing project which contains a build.rs that runs cbindgen takes the .a file and the generated bindings and creates or updates the Apple xcframework the C and Swift code depend on

How did you setup the dependencies in this do-nothing-project so that the original project has finished compiling and produced a .a static library when build.rs of do-nothing-project starts executing? I had to do some weird buzy-wait and check modified timestamp so that I don't accidentally link an old version of the library.

I tried to use build-dependencies (so that it will be built before build.rs starts) but that attempts to link the original project to build.rs code which is not what I want.

[RealSuperRyn]

In an OS project I am doing, I need the compiled bootloader files to be put into position and then have a virtual machine run using those files. My current solution is a hacked together shell script that I annoyingly must run in place of cargo build or cargo run. Post-build scripts would be much appreciated, as it would make my workflow much more bearable.

[correabuscar]

In an OS project I am doing, I need the compiled bootloader files to be put into position and then have a virtual machine run using those files. My current solution is a hacked together shell script that I annoyingly must run in place of cargo build or cargo run. Post-build scripts would be much appreciated, as it would make my workflow much more bearable.

perhaps you can call the script via runner ie. via cargo run ?

[Scramjet911]

Those coming from other languages with different package managers might expect a postbuild option and be sad not to find it.

I hope something comes up, or atleast a standardized way to do it which doesn't involve experimenting with different packages for a basic use case