Improve support for source based Linux distros

[cardoe]

Currently I am trying to support cargo based packages on two source based distros. I will caveat this issue saying that my approach might be wrong but its a use case that I need a solution for. I've run into two main issues:

• reproducible builds - cargo currently might use a newer dependency at a later point for a package
• limited networking - cargo reaches out to the internet when networking might not be available

For source based distros they separate out building of packages into different stages. Roughly they are separated into:

• fetch
• unpack
• build
• install

The unpack, build, and install phases are run without networking available. This means that the traditional cargo model of download a crate and then pull down the dependencies and build them does not work. The issue is that the source for the crate has not yet been unpacked before I need to fetch down the remaining dependencies.

To work around this I have implemented native fetchers for Yocto and Gentoo similar to how npm packages are handled in those distros. I considered npm to be analogous to cargo but for JavaScript in this case. The fetchers grab a package, its dependencies and a copy of the cargo registry index at a specific point in time and create a $CARGO_HOME that matches the layout that cargo expects. Unfortunately this on disk format changes from time to time which makes this method fragile.

In an effort to make packaging of these crates easier I've started https://github.com/cardoe/cargo-bitbake and https://github.com/cardoe/cargo-ebuild which aims to eventually produce recipes for Yocto and Gentoo respectively that do everything necessary to package a crate for those distros.

For an example of distro recipes see:

• https://github.com/cardoe/gentoo/blob/ca3e8fcfc273a9320ee3c6f8abec95d16ff4dadb/dev-util/rustfmt/rustfmt-0.5.0.ebuild
• https://github.com/starlab-io/meta-rust/blob/9bc90a04094a4b50b346b83acecb3df912cd9824/recipes-example/rustfmt/rustfmt_0.4.0.bb

[steveklabnik]

Have you seen #2857 ?

[alexcrichton]

Thanks for the report! This discussion here has definitely been a long and ongoing one for quite some time now, and there are a number of related threads especially on the internals forum about packaging which touch on this topic.

I think it's worth clarifying a few things, however, before diving in. First off, one of Cargo's main goals is indeed reproducible builds, that's the whole point of a lock file! Whether or not packagers end up using that is up to them, but Cargo is first and foremost built for reproducible builds, you'll just have to use the infrastructure. Second, Cargo does not reach out to the internet unless it actually needs to. If you run cargo build on a crate on a brand new system, the only way for that to succeed is to actually download dependencies. If you have, however, downloaded dependencies ahead of time with something like cargo fetch, then any future cargo build should recognize that lock file along with the crate cache and never touch the network. If the network is touched, that's a bug!

As @steveklabnik the support added in #2857 should help out packagers, especially when paired in tandem with #2811. This provides the ability to have a clear step (the "fetch" you mentioned above) where dependencies are downloaded and placed onto the filesystem. It could also be the case that distros even express dependencies on crates in their own mechanisms and pre-arrange for a directory of vendored sources to appear. Finally, #2811 allows you to assert to Cargo that the network should not be needed, and an informative error will be provided if access is needed.

In general though this bug isn't too actionable because there's not actually a concrete action to be taken. I'd recommend reviewing some of the threads on internals to see what the current state of affairs is, but this is definitely a work in progress! Cargo definitely wants to support this use case, and we're just making sure that it plays well with the rest of the workflows that Cargo has as well.