rfc: make cargo install extensible

[ashleygwilliams]

Rendered

The goal of this RFC is to introduce the ability for an end user to be able to extend the cargo install command arbitrarily to include instructions that should be executed occur after cargo install <project>.

---

This RFC was a collaborative effort between @joshtriplett @spacekookie and @ashleygwilliams.

---

A note about cargo as a distribution tool

We are in no way suggesting that cargo install should be the definitive or exclusive distribution mechanism, or that it should supplant other distribution mechanisms such as platform package management tools; we are only proposing to improve the experience for the class of applications that already depend on cargo as a distribution platform today, notably small CLI applications and development tools for the Rust ecosystem.

Additionally, this proposal will make cargo install more useful for the larger set of rust applications that leverage the larger, often preferred, platform-specific distribution workflows, which typically want to install into a temporary directory and package the result. This point is discussed further in the “Rationale and Alternatives” section.

Fundamentally, the aim of this RFC is aligned directly with Rust’s value of “productivity”. It aims to make both building and using small developer CLI tools simpler and easier which has a direct positive effect on the ability of all to improve their workflow.

[ashleygwilliams]

for context i would really like this (if considered to be viable) to be blocked on #2196 because i believe the pattern that this RFC introduces builds the affordances in a way that mitigates much of what i believe could be seen as a downside of this RFC

[est31]

Currently, cargo install tracks what files it installs, and supports cargo uninstall; this extension mechanism does not hook into that. Potentially, install.rs could emit a list of installed files and let cargo install install them, producing a log of those files for later uninstallation. However, we do not intend for cargo install to become a full-featured package management mechanism; rather, we expect cargo install to work analogously to make install. While the occasional package provides a make uninstall, few developers expect such a mechanism.

I guess I'm one of those who think that clean uninstallation is an important feature. I hope the statement that we are few is wrong.

I avoid make install like the plague and if I actually have to use it, if just invoking some binary in some build dir is not enough, I create a separate root fs and let it install inside there. This is a major hassle that I do just to have the simple pleasure of not getting my file system messed up with random stuff where I don't even know where it came from.

Currently, Cargo solves this in a beautiful way: it just puts a binary into a directory. It is simple, yes, not very extensible, yes. But it allows for clean and simple uninstallation. Updating is equivalent to "forcing" the installation.

With cargo install I actually started installing things into a per-user context because I trusted cargo: it just puts a binary into a directory. But if cargo install becomes extendible, it might put stuff anywhere in my home directory. And I'm not talking about malicious uses, but people who want to "do good" by instaling manpages or desktop shortcuts or file associations or similar.

If the RFC is merged in the current form, I'd probably stop using cargo install, and will be going back to patterns I've learned from C/C++: cargo build'ing and invoking binaries by their path inside the build directory.

Instead I think we should focus on artifact generation: allowing post-build scripts to create .msi files or .snap files or .deb files and allowing users to install them on their own, or maybe even on the user's behalf.

[liigo]

Maybe a more general way is post-build.rs, which can be used to invoke bindgen for example (not only for cargo install).

[joshtriplett]

@est31

Instead I think we should focus on artifact generation: allowing post-build scripts to create .msi files or .snap files or .deb files and allowing users to install them on their own, or maybe even on the user's behalf.

I'm all for supporting that as well. I'd like to make packaging easier. (That also includes policy-compliant packaging that could go into distributions, not just creating a .deb from thin air without a source package.)

What I'm looking for here is a way to let crates provide additional files, not just binaries. Generating manpages, generating completions, and putting them in the appropriate places.

We'd be open to talking about how exactly that should work. For instance, we certainly could have install.rs output a list of files to install and where they should go (including paths to files in the output directory that it just generated), and then cargo install could do the actual installation, and track the files for later uninstallation. That has downsides, in that it's a lot less flexible and puts more policy into Cargo, but it would support cargo uninstall.

Another possibility would be to just install those files into a temporary directory under the output directory, and cargo install would install everything under that temporary directory to the directory specified as its --root.

We'd like to do the simplest thing that could work, here.

[joshtriplett]

@liigo We talked about having a post-build as well, but that doesn't solve the problem we're trying to solve. The issue is installing these files, not just generating them.

[fbstj]

duplicate of previous point?

[ashleygwilliams]

whoops! excellent call. i'll fix that!

[est31]

@joshtriplett

For instance, we certainly could have install.rs output a list of files to install and where they should go (including paths to files in the output directory that it just generated), and then cargo install could do the actual installation, and track the files for later uninstallation.

That would work as well. in fact we could have post-build.rs generate those files. In the traditional make install set up, the installation step is only copying stuff to the appropriate places, while the previous steps did the remainder of the work.

Your proposal (calling it "microcargo") sounds fine @joshtriplett but I'd like to give an alternative "monolithic Cargo" proposal to just be able to compare the two:

You could have post-build.rs output whether installation of the program is supported and if yes, outputs a list of the files together with a semantic information what they actually are: executables, manpages, completion files, etc... all need to be put into different places and some would require an additional set up step.

Cargo would then store that list and cargo install would just use that list as a recipe for installation. It would then decide based on input parameters whether to install on a global, or on a per-user basis, and depending on that it could then also tell users to set up a ~/.bash_completion or decide which of the completion files to install based on the used shell. It would then track all the files it installed and support uninstallation. Of course this would require to put some logic into Cargo, but after all, Cargo is not a generic make replacement but a tool specifically built for the Rust ecosystem. It does many opinionated choices already.

So with the "microcargo" approach you have install.rs parse all the input parameters and environment info (per-user or global, name of the shell, distro directory layout, which files the user wants to be installed) and output a list of desired destinations. The advantage is flexibility. The disadvantage is that writing a install.rs becomes far more complicated... I've seen so many issues with make install. Often this is not the main attention of the developer and so often it sucks in one way or another. With the monolithic approach, you would have a central place where improvement can happen. However, this downside of the minicargo approach would be fixed by the metabuild RFC.

So summarizing, compared to the "shellout" proposal of the current RFC, I think "minicargo" and the monolithic approaches are both better, with me preffering the minicargo approach.

[Centril]

I'm generally in favor of a move in this direction, but I have some questions and concerns =)

[Centril]

"to be able" seems redundant here?
I propose the following wording:

This RFC gives end users the ability to extend the cargo install command arbitrarily, allowing them to include instructions executed after cargo install <project>.

[ashleygwilliams]

Sure thing. Happy to update.

[Centril]

What is the rationale for a metainstall function accepting no arguments, .. ?

This should be described precisely in the reference-level explanation. =)

[ashleygwilliams]

@Centril this is in #2196

[Centril]

Why is this a regular dependency and not a build or "meta-install" dependency?
Will the CLI app's code use cli-install as a library? It feels like there should be a separation...?

[ashleygwilliams]

This is probably best discussed on the metabuild repo where this syntax is being determined. #2196. Further discussing it here is not relevant, but discussion on that issue would be welcomed.

[Centril]

Oh I see; missed that =)

[Centril]

To @est31's point on uninstallation, should uninstall.rs be required if there is a install.rs?
Of course, a crate can cheat and have uninstall.rs which is not the inverse of install.rs, but it is better than nothing?

[ashleygwilliams]

Yeah we discussed this issue at length. We're not sure if requiring the uninstall.rs is better or worse as I would think, personally, that it gives the appearance of a guarantee that we cannot make, i.e. that it's not the inverse of install.rs by I'm open to being convinced otherwise.

[fbstj]

were any thoughts made for making/mandating that install.rs be reversible/isomorphic by design to allow the undoing automatically?

[est31]

I agree with @ashleygwilliams here: mandating (or even supporting) uninstall.rs is no guarantee for uninstall.rs actually uninstalling all of the artifacts. We should go either with the minicargo or monolithic approaches IMO. As for isomorphism @fbstj do you have any precedent of an isomorphic scripting language or similar that you can share? Also, we would have to make sure that all of the information that enters the program will stay the same, otherwise even if we have an isomorphic subset of Rust, we won't be able to guarantee equal outcome.

[Centril]

@fbstj Is that actually possible tho? Unless we have some snapshot of how things were before, install.rs being possibly non-deterministic should not allow you to find an inverse function / program. Of course you could require that the main of an install.rs file be a const fn which would get you determinism, but even then - we'd have to rule out many good programs to make reversibility efficient or decidable, no? And you'd need to extend the type system to rule out such good programs.

@ashleygwilliams Given that install.rs and indeed build.rs could do all sorts of things to your computer, including invoking UB, I think that guarantees were never something that we could make on this front, and that it's all inherently trust based. I think that if we clearly communicate that this is all best-effort, then it is better to require uninstall.rs scripts and build great helper crates around this.

@est31

As for isomorphism @fbstj do you have any precedent of an isomorphic scripting language or similar that you can share?

That would be quite the restrictive language, see: https://stackoverflow.com/questions/13404208/in-pure-functional-languages-is-there-an-algorithm-to-get-the-inverse-function

[est31]

guarantees were never something that we could make on this front, and that it's all inherently trust based.

Right now we still have the option to sandbox install.rs and friends to one directory where they can put stuff in, asking cargo to do the actual installation for them so that uninstallation is possible.

But even if we decide to not sandbox, this argument sounds too much like: "you need to already trust them so making it easier for them to not introduce bugs is not important".

[Centril]

This part seems left over from the RFC template?

[ashleygwilliams]

Yup. Can fix.

[Centril]

The use of the phrase "undefined behavior" should probably be reserved for when referring to UB as in your code not having any semantics. If that is what is intended here, then that is concerning.

[ashleygwilliams]

I don't think that's what is intended here, I think it was meant more as "potentially unexpected behavior"/"surprising behavior". I'm happy to update the language. Thanks for the catch!

[Centril]

Do we want to emulate make install tho? I've never found make to be a reliable program.
I also question the assertion on expectations. What is the evidence for it?

[ashleygwilliams]

I think this is fair. The relation to make was in a convo with @joshtriplett @alexcrichton and @withoutboats - I also share your concern a bit re this, but I found that the affordances this gives us for both cargo install distribution AS WELL as platform specific package managers outweighed it in my opinion. I'm not sure if there's any data/anecdata to be found but I would be happy to try to find some!

[aidanhs]

If we did want anecdata, I wonder if cross compilers and distro packagers would be a good source of information on make install behaviour as they're groups of people likely to spend a high percentage of their time compiling things.

My own experience/expectation from when I was doing a lot of cross-compilation with emscripten is that things supporting make install a) permit some way to set the installation 'prefix' (be it --prefix with autotools and autotools-mimicking things like the rust configure script or the cmake variable or something else) and nothing gets installed outside of that prefix (e.g. completions end up in $PREFIX/etc/bash_completion.d) and b) make uninstall is a bit flaky.

[Centril]

Small note: I don't think cross compiler and distro packagers are representative of the set "developers". ;) Even so, any and all anecdata would be nice to have even if unrepresentative.

[aidanhs]

Ah I take your point, perception is important. I was just thinking about the most efficient way to get feedback on large numbers of executed make install commands.

[joshtriplett]

Something that seems worth mentioning, explicitly: some responses have expressed the sentiment that the right answer is to work towards distribution packaging and similar, rather than making cargo install better. It seems worth pointing out, as someone who does care about distribution packaging as well, that making cargo install better will make it much easier to do distribution packaging. As mentioned in the RFC, we'd like to be able to cargo install to a directory and then package the result, and we'd like to make that include things like completions, manpages, and other associated files.

[ashleygwilliams]

So, as was mostly expected, the primary question brought up by the comments so far is the issue of uninstall. The current RFC presents the opinion that we should make no effort to offer an uninstall.rs, however it is an open question as to when an install.rs what we can do to mitigate the absence of an uninstall.rs. Several suggestions have been made, but I think that ensuring that cargo emits any files systems changes when an install.rs is run is a reasonable strategy to take.

Would that appease any of those who are concerned about uninstall? If not, let's try to aggregate other suggestions about mitigating the uninstall issue and discuss which if any would be reasonable to implement alongside this, with an eye that there is still the goal of not packing cargo with so many features that it begins to resemble a platform-specific package manager. (which i think we all agree is something we DON'T want to happen.)

[est31]

ensuring that cargo emits any files systems changes when an install.rs is run is a reasonable strategy to take.

I'm not sure I understand your proposal fully. Could you explain it? Suppose I did cargo install foo. With which command would I uninstall it? And how would it look implementation wise?

[Manishearth]

I think she's talking of a general class of solutions (correct me if I'm wrong), but one such proposal would be that install.rs emits cargo:installed-file:/path/to/manpage, and cargo keeps these strings around. On uninstall it removes these files and then removes the binary.

(This doesn't cover all cases of uninstall, but as the RFC mentions this isn't intended to be a generalized distribution solution)

[joshtriplett]

@Manishearth Right, the idea would be that install.rs either prints a list of all filesystem changes it wants to make but doesn't make them directly, or alternatively that install.rs only installs things to a subdirectory of the output directory and then cargo install installs those things. Either of those would allow cargo install to optionally remember what it installed for later uninstallation.

[ashleygwilliams]

@est31 yup, what both @Manishearth and @joshtriplett expansion of what i said is correct. i'm curious if that's enough to address the issue- and if not, i'm curious what strategies would best address the issue without growing cargo into a thing similar to a full fledged platform-specific package manager, which, i believe, we all agree that we don't want.

if we can agree on something i am very happy to update the RFC to include this strategy explicitly.

[Centril]

This strategy seems reasonable, and mostly assuages my concerns. There's of course the issue of regedit and such things on Windows where you are dealing with non-files. But I think that falls under not having a "full fledged platform-specific package manager"?

Personally I prefer the variant where cargo moves the actual files and the install.rs creates them and points to where they should be.

I think an argument in favor of this RFC is that we can always decide to permit an (optional) uninstall.rs file later on (if we wish / there's sufficient demand for it). And so we can start with a conservative initial feature.

[Screwtapello]

One of the problems that crops up with other languages' package managers (hi, Python!) is that the tool chooses one particular convention for where to install files and uses it on every platform in the name of consistency. Usually this convention is highly-POSIX-flavoured, making things weird and awkward to use on Windows.

I realise that ship has kind of already sailed for Cargo, since it installs binaries into $CARGO_HOME/bin, but is there a risk that adding more "install" support to Cargo will make it even more awkward on Windows, or might even make some tools unnecessarily Windows-incompatible if they try to install a file named "aux" or something?

[ashleygwilliams]

@Screwtapello so, i share this concern but i think this RFC alleviates it instead of exacerbating it. we specifically are picking to do a install.rs instead of a post-build type situation so that authors can detect platform and perform platform specific functions. there's no way to necessarily guarantee that authors will do it, but the motivation for this RFC, at least partially, is the fact that authors WANT a better cross-platform story.

[aidanhs]

I didn't see any mention of passing arguments to cargo install - is this a deliberate choice or just unspecified? If unspecified I'd like to propose allowing it (to avoid a world of ad-hoc environment variable usage).

[ashleygwilliams]

@aidanhs it was not opinionated! i think that would be fine, and likely uncontroversial? but i'm curious what others think (assuming no one objects i can specifically add that to the RFC)

[est31]

the idea would be that install.rs either prints a list of all filesystem changes it wants to make but doesn't make them directly, or alternatively that install.rs only installs things to a subdirectory of the output directory and then cargo install installs those things.

@ashleygwilliams

i'm curious if that's enough to address the issue

Yes, both variants as laid out by @joshtriplett would be enough (see quote above). I'd personally prefer the variant where install.rs is not doing the changes directly and would like install.rs to be sandboxed. It could have unrestricted read access but write access restricted to one directory inside the target/ hierarchy to ensure that install.rs is actually using cargo to install stuff.

Especially, it seems to me that the intention is not for install.rs to be as flexible as possible, as this would mean support for any kind of installation of applications (cargo replacing an OS package manager), but rather to allow the minimum of features for CLI applications to install their manpages and completion files.

[ehuss]

I realise that ship has kind of already sailed for Cargo, since it installs binaries into $CARGO_HOME/bin

@Screwtapello This may not be true in the future if something like rust-lang/cargo#5183 is merged.

[alexreg]

Installing to a staging area, and from there copying to either ~/.cargo or /usr/local would seem sensible (depending on whether the install is global or user-local). Then the uninstall actions can be easily generated just by walking the staging area directory tree. Possibly custom install/uninstall commands could also be added for things like registry edits on Windows. Alternatively, the staging area could be a permanent one, with symlinks into one of the two aforementioned dirs. this is how Mac’s Homebrew does it.

[casey]

I think that I'm against this RFC. Instead of allowing crate authors to put arbitrary steps inside of install.rs, I would prefer that we extend cargo and Cargo.toml to support additional crate artifacts types.

To make it concrete, I'll use man pages as an example. With this RFC, crate authors would write an install.rs script which would generate and move man pages into the correct location. I'll refer to this as the imperative approach.

An alternate approach would be to allow crate authors to put something like man-dir = "man" into their Cargo.toml, and let Cargo put any man pages it finds there in the appropriate place. I'll refer to this as the declarative approach.

I think that the declarative approach is superior for a number of reasons.

• Under the imperative approach, all crate authors must write their own install.rs scripts. Although they can certainly draw from each other's scripts, this is a great deal of duplicated effort. Additionally, installation is complicated by different distro and platform quirks and conventions, so many install.rs scripts will likely have bugs. With the declarative approach, this code lives in one place. This allows for less duplicated effort. And, when bugs are found, they can be fixed for everyone in one place.

• Different distros have different ideas about where things should go. For example, man pages will go somewhere entirely different on Ubuntu, homebrew, macports, nix, or windows. With the declarative approach, they can write a single adaptor script that parses Cargo.toml, which will work with all crates without further effort. With an imperative approach each distro will have to examine and adapt the install.rs script to identify man pages and put them into the correct location.

• With the declarative approach, we can add additional support for special handling of different kinds of crate artifacts. For example, we could support a command line flag to only install binaries and not man pages, or vice versa. It would be very hard to support this using the imperative approach, since all install.rs files would need to support this flag. Additionally, with the declarative approach, different crate artifact post-processing steps can be added as needed, for example converting to CRLF line endings on windows.

So, to sum up, I think we should extend cargo with the ability to understand and install crate artifacts on a type-by-type basis, so that they can be handled consistently and appropriately across platforms, and not left to individual crate authors.

[alexreg]

@casey I think that's fair for a large number of cases, but what about certain Rust projects like tectonic, for which this RFC would be hugely useful in its present form?

[casey]

@alexreg, what kinds of artifacts does tectonic need to install? I'm wondering if there might be a better way to support them.

[alexreg]

@alexreg, what kinds of artifacts does tectonic need to install? I'm wondering if there might be a better way to support them.

Probably best we check with the Tectonic people, like @pkgw, since I'm not 100% sure at this point.

[alexreg]

I suppose this hasn't been high on people's priority lists lately, given the lack of news. Any chance of reviving things?

Some questions I had:

• Is the plan now to make cargo closer to a lightweight package manager, per above discussions?
• Should we retain separate "repositories" of Cargo binaries & their support files that switch depending in the toolchain, in a pyenv-like manner?
• What's the interaction with Cargo components going to look like in the future? It seems like components could just be "official" versions of Cargo packages, if we extend the cargo install system to support a more general concept of "packages".

[richard-uk1]

It makes the emergence of conventions significantly more difficult as the option to reuse, share, or automate this task has significantly affordance than integrating it into the familiar cargo step.

I'm struggling to parse this. Can you tell me what you mean?

cargo install could optionally have a flag to disable the running of the install.rs file.

What's the use case for this?

[DrSensor]

One of my use cases of this feature is to enforce all of the collaborators to set up git-hooks scripts to enforce some policy like commit message and others. I usually use husky to enforce all collaborators to follow conventionalcommits

[alexreg]

Looks like this is being revived as part of rust-lang/crates-io-cargo-teams#14. @ashleygwilliams / anyone else, how are we going to proceed here?

[zslayton]

This RFC sounds excellent! In particular, this approach to offering cargo uninstall strikes me as very reasonable:

....Right, the idea would be that install.rs either prints a list of all filesystem changes it wants to make but doesn't make them directly, or alternatively that install.rs only installs things to a subdirectory of the output directory and then cargo install installs those things. Either of those would allow cargo install to optionally remember what it installed for later uninstallation.

I was surprised to see that there hasn't been any movement on this in a year and a half. Has it been superseded by more recent initiatives? If so, could someone provide a link?

[joshtriplett]

This came up in today's @rust-lang/cargo meeting. This hasn't seen activity in a long time, and as a result, we'd like to postpone it. This should not be taken as a lack of interest; we just don't want to leave this open and unresolved indefinitely. If someone would like to pick this up, they should feel free to do so, and post a new RFC.

In general, if someone is interested in reviving this, they should specifically write this in terms of being able to stage things in an output directory and letting Cargo install them, rather than installing things directly. We're generally not in favor of a completely general "install things directly onto the filesystem" script. (We're also uncertain about the amount of complexity this might create in Cargo, and we'd want to know this wouldn't impact long-term maintainability.)

Hopefully this guidance helps someone looking to work on this in the future.

@rfcbot postpone

[rfcbot]

Team member @joshtriplett has proposed to postpone this. The next step is review by the rest of the tagged team members:

• @Eh2406
• @alexcrichton
• @ehuss
• @joshtriplett

No concerns currently listed.

Once a majority of reviewers approve (and at most 2 approvals are outstanding), this will enter its final comment period. If you spot a major issue that hasn't been raised at any point in this process, please speak up!

See this document for info about what commands tagged team members can give me.

[rfcbot]

🔔 This is now entering its final comment period, as per the review above. 🔔

[rfcbot]

The final comment period, with a disposition to postpone, as per the review above, is now complete.

As the automated representative of the governance process, I would like to thank the author for their work and everyone else who contributed.

The RFC is now postponed.