Support different versions of the same dependency within a workspace

[alshdavid]

Problem

My Cargo workspace has multiple entry points that compile to executable/bin or dylib targets. Currently, when placed within a workspace, Cargo will attempt to combine their dependencies where they share the same "compatible" versions of a dependency.

The problem is that these project consume third party dependencies that have conflicting versions - either because library maintainers don't adhere to semver correctly or because library maintainers choose to specify exact versions of a dependency (e.g. =0.0.40).

My projects can be compiled on their own without issue however, if they share a workspace, cargo build will fail because Cargo tries/fails to resolve these conflicting dependencies.

Example Case

I have a workspace that has three packages

/crates-main
  /project-bin
    Cargo.toml
  /project-types
    Cargo.toml
/crates-plugins
  /plugin-dynamic-lib
    Cargo.toml
Cargo.toml

crates-main/project-bin
This is the main entrypoint for my application which compiles to an executable. This program is able to consume dynamic libraries using the libloading crate.

crates-main/project-types
This is a shared library compiled as a lib that has no external dependencies, only exporting types to be used by both the "plugins" and the main executable (a.k.a. the "contract").

crates-plugins/plugin-dynamic-lib
This is a library that is compiled to a dylib. It depends on project-typesfor the types required to initialize a plugin and will be consumed by the executable produced byproject-bin`

Note that this package could also be an executable, I am using a dylib in my example because that's my current use case

In summary we have 2 packages that compile to binaries (./project-bin and ./plugin-dynamic-lib.so) and one shared library that is statically linked within those two crates.

Problem

As an example, assume project-bin consumes log = "=0.4.20" and plugin-dynamic-lib consumes log = 0.4.21 (indirectly via a third party dependency external to the workspace).

In a combined workspace, cargo build will error saying that it cannot resolve a compatible version between the two specified.

In reality, these packages will compile to separate binaries so version conflicts of dependencies would not result in a material conflict at runtime.

Current Solution

To get around this today, I simply avoid using a Cargo workspace and compile the projects independently of each other from a build script where shared dependencies are referenced via my_pkg = { path = "../path/to/pkg" }.

The issue with this approach is that rust-analyzer is unable to provide suggestions for the packages when the top level folder is open in the editor - resulting in a less than ideal development experience

Proposed Solution

A few possible solutions to this:

Option 1

Allow for multiple incompatible dependencies to coexist within a workspace if their versions cannot be combined and their consumers are of crate-type bin or dylib

e.g. a package in crates.io with the latest version of 0.4.21

Consumer A

[bin]

[dependencies]
dependency = `^0.4.0`

Consumer B

[bin]

[dependencies]
dependency = `=0.4.20`

Consumer A would get 0.4.21
Consumer B would get 0.4.20

Option 2
Devise a way for rust-analyzer to work with multiple nested projects within a parent directory

Option 3
Perhaps some kind of support for workspaces, isolating dependencies between the workspaces

Notes

No response

[epage]

The problem is that these project consume third party dependencies that have conflicting versions - either because library maintainers don't adhere to semver correctly or because library maintainers choose to specify exact versions of a dependency (e.g. =0.0.40

One of the reasons Cargo is opinionated is to put pressure on libraries to "play nice" by

• Following Cargo community expectations around semver (which is different than a single person's interpretation)
• Discouraging the use of = operators (which we explicitly call out)

[epage]

As for the request, I feel like we should have an existing issue but my search skills are failing me.

• Specify in root that certain edges in the dependency graph should be "private" #7276 proposed the idea of private dependency graphs.
• In theory, RFC: Superseding public/private dependencies rfcs#3516 could help towards something like this.

I do wonder if we'll come to the point where dependency sub-graphs are treated as locked together, of the scale of gitoxide or bevy, making them act more like a cohesive single library. iiuc build-std has a specialized version of this.

[alshdavid]

put pressure on libraries to "play nice" by

I seriously wish this worked out in practice, it's been such a nightmare to work around. (you can skip the next part, I'm just ranting)

I'm writing a JavaScript bundler and want to embed Deno into it to function as a JavaScript runtime for dynamic plugins.

Deno depends on = versions of SWC because SWC doesn't play nice with semver.

I also depend on SWC directly however the conflict between Deno and my internal version of SWC prevents Cargo from resolving dependencies. Further, Deno has a fixed version of Tokio, log and other libraries which makes it difficult to embed.

To try to get around this, I created a new dylib crate in my workspace that bottles up my Deno integration and exposes behaviours via an FFI that I consume from my main application using libloading.

/deno_integration
/my_project

/target
  /debug
    my_project
    deno_integration.so

This is okay enough however my deno_integration dylib crate cannot be part of my workspace Cargo.toml because of those conflicting dependencies (even though it will compile to a discrete target).

So I add deno_integration to my workspace [workspace.exclude] and build it separately to the packages in my workspace.

cargo build # for the workspace
cd deno_integration && cargo build

If I have my workspace open in my editor, I get no rust-analyzer coverage on the deno_integration folder unless I open that folder separately in a new editor window.

It's such an annoying workflow - but it works and unblocks me for now.

[weihanglo]

I believe you've tried this. Just asking in case we forgot: Have you used the [patch] table to patch deno's Cargo.toml?

The other way can help upstream is helping them integrate cargo-semver-checks. The tool is quite handy and easy to use. Cargo the project has used it in every pull request

[epage]

My other question is if swc and deno are meant to be pulled in as libraries like this. These look to be fairly large, complex applications and sometimes the "libraries" for them are more meant for internal purposes. I know I maintain several bins whose lib portion is an implementation detail and I very intentionally do not offer semver compatibility guarantees for those APIs.

[alshdavid]

In the case of swc, it's definitely intended to be used as a library as that's its primary use case (it's an AST parser, etc) - and they have a reputation for poorly adhering to semver. I tried to politely bring it up with the maintainers but they were not receptive.

As for Deno, their crates and docs seem to indicate that this is a use case they support. I don't think this use case is a huge focus for them though as the dependency composition of their crates seem to indicate that.

There are other projects (like Supabase - an OSS "aws lambda"-like engine) who integrate Deno. Supabase's solution was to use some of the deno crates from crates.io and vendor the parts that conflicted

[jozanza]

fwiw, I have the same issue in my workspace, but it's with much smaller lib: zeroize.

This lib happens to be a dependency of a few dependencies across crates within my workspace. I never directly added it. However, I just added a new crate with a dependency on k256. This unfortunately introduced a version incompatibility. Previously, everything had worked well enough since cargo found the range of acceptable zeroize version to be >=1, <1.4. However k256 wants ^1.7.

In practice, since I don't have control of the zeroize dependency version directly in any workspace crates, I can only think of three less-than-ideal options to get around this while still keeping this new crate in the workspace:

1. Attempt to upgrade all of the deps of my workspace crates that depend on zeroize until 1.7 is the upper bounds.
2. Attempt to downgrade k256 until 1.4 is in the range it wants.
3. Fork zeroize on github and apply a patch section to Cargo.toml.

In this scenario, the packages are entirely separate and do not use each other as dependencies, so if zeroize happened to be a different versions in both packages, this wouldn't cause any problems for these packages. But cargo won't allow this for packages in the same workspace. I even attempted to remove the package from the workspace and import it by path, but that results in the same version conflict.

Overall, it's a pretty painful issue that seems increasingly likely to happen as a workspace grows and it's entirely dependency on how package authors specify their own version requirements, which you have no control over. I've run afoul of this issue numerous times over the past few months and it always a huge productivity loss resulting in many hours of twiddling with version numbers, opening issues and PRs in maintainer repos, and/or forking+patching. At this point, it's become the biggest issue I have maintaining a large codebase in rust.

[epage]

everything had worked well enough since cargo found the range of acceptable zeroize version to be >=1, <1.4

imo this is a bug and should be reported. Upper bounds should (almost) always be semver boundaries. Cargo's workflows are designed around that.

Overall, it's a pretty painful issue that seems increasingly likely to happen as a workspace grows and it's entirely dependency on how package authors specify their own version requirements,

Right now, there the design of Cargo puts pressure on people to be "conforming". If we loosen this, people will no longer be "conforming" and a lot of the uniqueness / value-proposition (of semver working) is lost. Cargo is intentionally opinionated. You mentioned running into it a lot. I rarely run into it. That doesn't invalidate your experience but to highlight that it isn't everywhere and that is the risk we introduce by loosening things up on this.

[jozanza]

Sure I take no issue with semver and putting pressure on authors to "conform" to it. Semver's cool and authors should adhere to it.

The issue here I'm raising (and what I assume is the topic at hand based on the issue title) is that there's no sane way to have two different versions of the same dependency in a workspace.

The semver version ranges are not problematic for these crates in isolation. However because cargo just wants them to use the same version in the workspace, it causes a lot of issues: poor dev experience, huge loss in productivity. It would be less bad if it was something that could be resolved unilaterally, but it often involves opening issues, forking, etc.

So in an ideal world, we could just opt out of unifying the dependency versions when needed. This is essentially what happens when we fork and patch, but its painstaking.

Regardless of how often developers run into this (we have very different experiences here), there should be a reasonable solution that doesn't require forking and opening github issues or upgrading and downgrading dependencies and compiling over and over to see what happens. The potential solutions right now all have heavy trade-offs and are quite time-consuming.

[epage]

Sure I take no issue with semver and putting pressure on authors to "conform" to it. Semver's cool and authors should adhere to it.

The issue here I'm raising (and what I assume is the topic at hand based on the issue title) is that there's no sane way to have two different versions of the same dependency in a workspace.

@jozanza the problem in your situation is that people are inherently not following semver because they are saying newer, compatible versions are not compatible. If they were, you wouldn't have a need to allow multiple versions.

[jozanza]

I agree that's definitely a problem here. I wish the library authors had done that. If they had, I would not have encountered this issue.

However, the other problem is what your options are as a developer when you do encounter this issue.

In this case, I "fixed" it by downgrading the k256 crate, which isn't great considering it's a cryptographic crate and I would like to have the latest version with the most recent bugfixes, etc. AFAICT, I have two other options:

• Upgrade dependencies, which could potentially take longer and require me to rewrite portions of my codebase.
• Getting a library author to publish a fix or fork + patch. These are by far the most time-consuming. There's no guarantee a maintainer will have time to implement or agree with your proposal. Forking is more certain, but it means maintainers miss the fix, and your fork misses out on the upstream (similar to the downgrading solution).

So I hope you can see what I'm trying to communicate here. I don't think you're wrong about semver. I agree. Library authors should conform. But I also think there's more that can be done to address this. A sane escape hatch for multiple versions of the same dependency within a workspace would be an absolute godsend for these scenarios.

[weihanglo]

Both of you hold valid points. As a downstream user, it's pretty reasonable we just want to fix our build and move on. As a Cargo maintainer whose decision might affect the community, we would like to encourage people to contribute back to upstream.

From the angle of implementation, accepting multiple SemVer-compatible version might complicate the already-complicated dependency/feature resolution as well as [patch]ing. @Eh2406 might have more insights on this.

Side node: For the specific zeroize it was pinned to certain version range because of MSRV: RustCrypto/utils#723, which can be mitigated with MSRV aware resolver. Hence I expect the situation of this kind of pinning will be less.

[jozanza]

Thanks @weihanglo. I totally agree with @epage's good/valid points.

Side node: For the specific zeroize it was pinned to certain version range because of MSRV: RustCrypto/utils#723, which can be mitigated with MSRV aware resolver. Hence I expect the situation of this kind of pinning will be less.

I hadn't even dug in that deeply into zeroize specifically. I had no idea about the MSRV resolver. So I guess there's a 4th solution that could work in some cases. Thanks for sharing.

As a downstream user, it's pretty reasonable we just want to fix our build and move on. As a Cargo maintainer whose decision might affect the community, we would like to encourage people to contribute back to upstream.

And I see where there's a sort of tension between what is ideal for downstream user and what we want to encourage from community. I have a sense that there is probably a way to eliminate that tension. Ideally, devs can fix builds quickly and contribute changes upstream just as quickly. I would love to learn more about the challenges of accepting multiple SemVer-compatible versions. I would be incredibly happy to help implement if it is something we would be open to.

[cgebe]

Right now, there the design of Cargo puts pressure on people to be "conforming". If we loosen this, people will no longer be "conforming" and a lot of the uniqueness / value-proposition (of semver working) is lost. Cargo is intentionally opinionated. You mentioned running into it a lot. I rarely run into it. That doesn't invalidate your experience but to highlight that it isn't everywhere and that is the risk we introduce by loosening things up on this.

Some people will not be "confirming", no matter what.

I encountered a similar issue with zeroize. One crate requires version ^1.5, while another needs >=1, <1.4. I need both of them in my project. As a downstream user, I just want a quick and clean resolution to this conflict. Unfortunately, Cargo doesn't provide a straightforward solution, so I'm forced to use separate workspaces as a workaround, which is far from ideal.

[epage]

@cgebe have you opened an issue about the non-semver upper found? What was the response?

[cgebe]

@epage You can follow it here: solana-labs/solana#26688 deep dependencies that are still not updated. My hands are restricted there, luckily there is a solution offered but still suboptimal.

[epage]

Following that link, it sounds like at least one source of those problems was people trying to use version reqs for MSRV and learned that they shouldn't and moved away from it, see rust-random/rand#1165 (comment)

I can't easily tell from that link what is still an issue and what is being done about it.

[blastrock]

This issue is still present. The solution of having all packages update their manifest to avoid over-restrictive is not viable. Some packages aren't updated anymore, and they depend on those mis-configured packages. In the case of solana, some packages are contracts that have been published on the blockchain. The contract is locked on the blockchain, so the corresponding crate should never be updated as it reflects the code that has been compiled to the contract.

As a non-expert in rust, I find the cargo errors very frustrating:

error: failed to select a version for `solana-program`.
    ... required by package `solana-zk-token-sdk v1.16.14`
    ... which satisfies dependency `solana-zk-token-sdk = "^1.16.14"` of package `client v0.1.0`
versions that meet the requirements `=1.16.14` are: 1.16.14

all possible versions conflict with previously selected packages.

  previously selected package `solana-program v1.17.13`
    ... which satisfies dependency `solana-program = "^1.16.14"` of package `raydium-ammv4 v0.1.0`
    ... which satisfies path dependency `raydium-ammv4` (locked to 0.1.0) of package `client v0.1.0`

failed to select a version for `solana-program` which could resolve this conflict

This error does not explain the issue at all (at least to me). It doesn't explain why using solana-program 1.16.14 is not a solution. I went down this rabbit hole thinking that my issue could be resolved by just overriding a dependency or two, but I have no idea what triggers the issue here.

[alshdavid]

Hi team, is it worth considering offering an env variable for the compiler to allow the use of conflicting versions, replacing the error with a warning?

env CARGO_ALLOW_VERSION_CONFLICTS=true cargo build

That way, projects that are stuck can be unblocked and the issues raised to the maintainers.

[ashi009]

That way, projects that are stuck can be unblocked and the issues raised to the maintainers.

Once unstuck, no one will ever care to fix the issue.

[mguentner]

I want to give a different perspective and a use-case that is prevented by this limitation:

Assume an application that gets released frequently. The legacy releases of the application share dependencies with the current release, however in different versions (older serde, serde_json etc.). Legacy versions are not maintained any longer, however migrations from that versions must be supported to the maintained, current version.
The migrations are needed as the application persists complex state to a json file and reads from it on startup. The state has a schema that relates to the version of the application, i.e. version 1.0 expects the state to be in version 1.0. As the application receives new features, the schema changes and hence some migration logic is needed, e.g. from 1.0. to 2.0. This logic should be encapsulated in a dedicated logic to keep this complexity out of the application itself - it only needs to know how to read the most recent state schema!

The following is not possible due to the limitations described in this ticket:

Cargo.toml of application in Version 2.0 has legacy releases defined as self-references to its own repo, all pointing to their respective tags.
These references make the type definitions of 1.0 and 1.1 available to code in 2.0 without duplicating / copying.

[dependencies.application_1_0]
git = "ssh://git@server:/this.repo.git"
tag = "1.0"

[dependencies.application_1_1]
git = "ssh://git@server:/this.repo.git"
tag = "1.1"

The migration (pseudo) code could then be written for a migration path from 1.0 -> 1.1 -> 2.0

let state_1_0 = application_1_0::state::read_from_file(file);
// ^ this is application_1_0::State
// fn migrate_1_0_to_1_1(state: application_1_0::State) -> application_1_1::State
let state_1_1 = migrate_1_0_to_1_1(state_1_0);
// ^ this is application_1_1::State
// fn migrate_1_0_to_2_0(state: application_1_1:State) -> application::State
let state_2_0 = migrate_1_1_to_2_0(state_1_1);
application::state::write_to_file(state_2_0);

Afterwards, in the application.rs (main binary)

application::state::read_from_file(file);

The solution presented here is really efficient and allows for a clean architecture as no code is copied or duplicated. It would also allow to use legacy types as they are defined within the application without the need to carefully extract them to a completely dependency-free (!) types crate, something which is often just not possible as foreign types are dependencies as well!
All workarounds take quite a lot of time in order to realize something similar as described above.