Skip to content

Support packaging workspaces

Jump to bottom
Tor Hovland edited this page May 10, 2024 · 14 revisions

cargo package --workspace and cargo publish --workspace won't do what you expect when run on a fresh workspace that hasn't been published before. For any inter-workspace dependencies, Cargo expects to be able to pull down the dependent crate from the registry, ignoring any path dependency. See https://github.com/rust-lang/cargo/issues/10948.

@jneem and @torhovland are trying to fix this issue. It is part of a bigger issue: https://github.com/rust-lang/cargo/issues/1169. If all goes well, we'll proceed with other tasks there.

Our work is in this branch: https://github.com/tweag/cargo/tree/package-workspace.

How to reproduce the issue

cargo test package_workspace

Solution strategy

Change the packaging procedure to something like this:

  1. Determine all intra-workspace path dependencies.
  2. Package each crate using the existing code (without trying to compile/build it, or otherwise resolve online dependencies).
  3. Unpack all packaged crates in a temporary directory, giving each one a random dir name.
  4. Go through all workspace dependencies from step 1 and fix them up in each crate. Can we do this in-memory?
  5. See if this builds.

More details

Determine all intra-workspace path dependencies

We need to allow for dependencies to other versions of the workspace crates. So we should only care about intra-workspace path dependencies where the dependency version matches the crate version. In other words, a dependency like this would be handled like an external dependency.

# crate a:
[package]
name = "a"
version = "0.2.0"

# crate b:
[dependencies]
a = { version = "0.1.0", path = "../a" }

Current challenges

Step 2 ends up needing to resolve online packages, even when we set verification to false:

   3: cargo::core::resolver::errors::activation_error
             at /Users/tor/projects/work/cargo/src/cargo/core/resolver/errors.rs:307:42
   4: cargo::core::resolver::activate_deps_loop::{{closure}}
             at /Users/tor/projects/work/cargo/src/cargo/core/resolver/mod.rs:349:29
   5: core::result::Result<T,E>::or_else
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/result.rs:1383:23
   6: cargo::core::resolver::activate_deps_loop
             at /Users/tor/projects/work/cargo/src/cargo/core/resolver/mod.rs:282:44
   7: cargo::core::resolver::resolve
             at /Users/tor/projects/work/cargo/src/cargo/core/resolver/mod.rs:143:13
   8: cargo::ops::resolve::resolve_with_previous
             at /Users/tor/projects/work/cargo/src/cargo/ops/resolve.rs:420:24
   9: cargo::ops::cargo_package::build_lock
             at /Users/tor/projects/work/cargo/src/cargo/ops/cargo_package.rs:512:27
  10: cargo::ops::cargo_package::tar
             at /Users/tor/projects/work/cargo/src/cargo/ops/cargo_package.rs:791:25
  11: cargo::ops::cargo_package::package_one
             at /Users/tor/projects/work/cargo/src/cargo/ops/cargo_package.rs:149:29
  12: cargo::ops::cargo_package::package
             at /Users/tor/projects/work/cargo/src/cargo/ops/cargo_package.rs:226:22

But why does this not successfully resolve the path dependency?

Because the path stripping happens quite early in the tar function (when calling prepare_for_publish). When build_lock gets called it has already happened.

We are OK with the path stripping, in order to build up a clean Cargo.toml file in the tar archive. We just don't want to try to resolve at this point.

But the lock file seems tricky. How are we going to handle that? When I build the workspace locally, I get a lock file like this:

[[package]]
name = "cargo-10948-ws-app"
version = "0.1.0"
dependencies = [
 "cargo-10948-ws-lib",
 "serde",
]

[[package]]
name = "cargo-10948-ws-lib"
version = "0.1.0"

[[package]]
name = "serde"
version = "1.0.201"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "780f1cebed1629e4753a1a38a3c72d30b97ec044f0aef68cb26650a3c5cf363c"

But when publishing, this lock file would need to include metadata for cargo-10948-ws-lib as if it was published, like for the serde dependency.

We can always set source ourselves easily enough (not really, only if we assume crates.io, which we really can't). Can we also calculate the necessary checksum? Yes, we can, it's a simple enough SHA hash of the archived file. But that means we would have to package each crate in the inverse order of dependencies.

Clone this wiki locally