Possibility to set the package version dynamically

[vorner]

Hello

I'm building a rust application using internal teamcity CI job. I want the build number to be part of the version of the binary (so eg. ./app --version knows from which build it came).

The only way I found so far is to let the build first edit the Cargo.toml (sed -i -e 's/^version = .*/version = "%build.version%"/' Cargo.toml), which seems ugly and fragile.

It would be great if I could somehow override the version from Cargo.toml through the command line ‒ either as a parameter or an environment variable.

Would something like that make sense? Are there plans to support it?

[dwijnand]

I think this could be created as a cargo extension using https://github.com/ordian/toml_edit. You could use https://github.com/killercup/cargo-edit as inspiration.

[vorner]

Actually, I don't want to modify the Cargo.toml at all if possible, just use that different value of version during a single cargo build. So something like cargo build --config package.version="0.1.2".

But yes, you're right, there are certainly less fragile ways to edit Cargo.toml.

[dwijnand]

I think there's a ticket for being able to modify options (I guess I'm both Cargo.toml and .cargo/config) via a command, like git config, but I don't know if it extends to temporary overrides.

[mixalturek]

It's very common use case to specify the version from command line while releasing on a CI server that tracks and auto increments the build number. GNU/make, Maven and Gradle natively support it, manual editing of a build-spec file is not practical and error prone.

make target VERSION=%build.version%
mvn release:prepare -DreleaseVersion=%build.version%
./gradlew build -Pversion=%build.version%

[dwijnand]

mvn release:prepare is based on the maven release plugin. The equivalent is https://github.com/sunng87/cargo-release, and I don't know if it supports specifying the version on the command line.

[vorner]

I don't think it is equivalent for my use case. This is for releasing on crates.io and modifying Cargo.toml.

What I want is somewhat completely the opposite. I want to leave Cargo.toml as it is, not touch git (tags, push) at all ‒ that happens externally. I want to just build the binary, but the binary should be fed with a version different from the one in Cargo.toml, because for this use case, Cargo.toml is not the authoritative source.

[dwijnand]

What does

the binary should be fed with a version different from the one in Cargo.toml

mean? Is it just env!("CARGO_PKG_VERSION") (and variants, like CARGO_PKG_VERSION_MAJOR) usage?

[vorner]

I'm not 100% sure there are no other places the version is passed into the compilation, but I guess so. I haven't tried yet (I might), but I guess cargo will set the env var unconditionally and overwrite it even if I pass it from outside.

[dwijnand]

Maybe we could just change that: only set the environment var if unset.

[lukaslueg]

built may help your use-case to some extent.

[vorner]

Looking at built, it seems to be really nice and useful thing and I'll keep this in mind. But I don't think this will help me here ‒ this doesn't help me push the right version into eg. clap or structopt.

[dwijnand]

Can you not define the version in clap/structopt in terms of built_info::PKG_VERSION?

[vorner]

By default, they take the version from CARGO_PKG_VERSION. I could set that manually to built_info::PKG_VERSION but I don't see how that would help me, because built also takes the value from there.

[dwijnand]

You're right, sorry. I think the solution is to not override the environment variables, then.

[Kaiser1989]

Is there any progress? Also looking for the feature to externally manipulate the package version. (CI is setting the version, Cargo.toml should not be changed).

An Env-Variable would be nice:
CARGO_TOML_PACKAGE_VERSION

This could also be done for every other part of the Cargo.toml:

• CARGO_TOML_PACKAGE_NAME
• CARGO_TOML_PACKAGE_AUTHORS
• ...

The place within the code is here:
https://github.com/rust-lang/cargo/blob/master/src/cargo/util/toml/mod.rs#L68
There should be an env-override after parsing the toml and before checking for unused.

This should be analogous to https://doc.rust-lang.org/cargo/reference/config.html#environment-variables

I could add the implementation if we find a good name for env variables...

[mieubrisse]

Adding another +1 - we're setting the version upon release, and it'd be great to not need to edit Cargo.toml each time

[cgranade]

By way of another +1 and showing analogous functionality in other toolchains, the .NET SDK allows for overriding version numbers at the command line with commands like dotnet build /p:Version=1.0.1 (https://andrewlock.net/version-vs-versionsuffix-vs-packageversion-what-do-they-all-mean/). I've often found that overriding package versions that way can be useful in using CI pipelines to control how packages are versioned.

[Ri0n]

absolutely same for me. CI and stuff.
I was going to use "built"'s GIT_VERSION for that, but we build in docker containers not passing .git inside. So this info is lost.

[Humandoodlebug]

This seems to be working on nightly right now: #6699

e.g. cargo build -Z unstable-options --config 'package.version="1.5.2"'

[QAston]

@Humandoodlebug

cargo build -Z unstable-options --config 'package.version="1.5.2"'

Will not work because --config overrides the cargo configuration, not the build definition in Cargo.toml

[colemickens]

Editing the Cargo.toml isn't practical in some scenarios anyway - in Nix we don't really have conventional network access during build phases, and so it's not possible rewrite toml and regenerate lock files to workaround this.

I do see that there is a package macro that makes it easy to override the build version, falling back to the Cargo version, but it wouldn't be fun adding this to lots of Rust projects if Cargo could do it: https://docs.rs/git-version/latest/git_version/macro.git_version.html

[Snazzie]

Very surprising that this is still an open issue in feb 2022. 3 years have passed! probably lost/forgotten in the 1.2k issues.
Theres no good way to programmatically replace the version in cargo.toml.

Could we @ some of the active maintainers to get some visibility on this issue?

+1 to adding version override in CLI like dotnet.

[brandonros]

Is there a way to get cargo build to blend Cargo.toml version with [[bin]] name output? so that the cargo build --release makes a project-name-version.dll for example?

[burdiyan]

+1 here. I don't understand where the practice of statically specifying a version in a checked in manifest file comes from. I just want to release whatever is already checked in without having to commit again to change any manifests. It's very error prone unless you're using some fancy automated tools. It's so easy to overwrite a version that was already published previously if you release forgetting to change the manifest.

Being able to specify the version in the command line and completely omitting the version from the manifest would be my ideal workflow!

[epage]

I suspect this is a large enough semantic change that this would need a major change proposal or an RFC and need a person to champion the proposal and implementation. It is unlikely for someone else to pick up and do all of that leg work on behalf of someone interested in this work. One challenge though is cargo is currently on a soft feature freeze and the cargo team is wanting to avoid distractions, including mentoring / shepherding, to be able to handle basic maintenance, finish currently committed work ,and to reduce technical debt so we can increase our capacity.

Things off the top of my head that would need to be figured out

• Should the version field be made completely optional?
• What are cargo's semantics without a version field and when no version is specified?
• What all commands should accept a version?
• How do we pass in versions, especially when running against multiple packages?
• How do we help people when they miss specifying the version for one of the multiple packages?
• How do we help workspace users with path dependencies?
  • Currently, to publish a crate with a path dependency, it also needs to have a version requirement specified. That version requirement must match the crate its pointing to. Tools like cargo-release handle updating version requirements of dependents so they don't become stale
• Is there any prior art in other ecosystems that we can learn lessons from?

[burdiyan]

@epage This is a nice summary to figure out a considerate solution to this problem! Thanks for that!

It's a pity that it's unlikely that this effort would come from the core team, but it's understandable.

On the other hand, maybe an easier and good enough solution could be adding a flag (or env var) to pass the desired version to cargo so it could ignore the one specified in Cargo.toml? I'm absolutely unaware of cargo's internals, but if there's only one place where Cargo.toml is read to extract the version, then it seems like a pretty easy change to make.

In my (somewhat biased) opinion: there should be no static file defining the "current" version of a package at all. This should be determined at release time by looking up the VCS info or elsewhere.

[John15321]

@John15321 this issue is about setting the version without changing the source code. I do not believe there is yet an issue for cargo officially having a new subcommand for doing this and would recommend creating a separate issue if that is what you want.

There are also a couple of tools related to cargo bump mentioned in this thread

• cargo release, particularly cargo release version
• cargo-edit's cargo set-version

There is also cargo semver

Sorry I did not have that in mind. This issue popped up when I was searching for a solution to changing the version in the toml file using a cli tool, and simply given that I have found one I wanted to share as most likely other people would find this issue as well with the same problem at hand

[JosiahParry]

+1 here. It would be great to have a way to do this. It would be even better if it is not required to publish the crate to bump the crate version.

[dead10ck]

It would be even better if it is not required to publish the crate to bump the crate version.

It is not required to publish a crate to bump its version. All that is required to bump the version is editing the Cargo.toml

[kyle-rader-msft]

Hello, I've found myself here as someone doing a greenfield project at a large company where most of the current tools we build in CI use dotnet's ability to set the version baked into a binary at build time (shown somewhere far above) on the command line. While carefully controlling a semver in the Cargo.toml is ideal - when your team owns too many CLI tools, and needs to ship and consume them automatically, a generated date-based version number becomes very useful. I was quite surprised to find that cargo build --config package.name=1.2.3-1 and similar attempts, don't work for this. As I'm trying to sway folks into writing more tools in Rust, it's a hard up-hill battle at big companies and little things like this result in us having to put other checks in place, reminders to bump versions, and then parse out what it is in CI. In the meantime, I'm hoping the cargo set-version plugin will work, but now we have to install that in the CI pipeline too :(.

[juliusl]

I was curious about this problem so I prototyped how I originally was assuming this would work, which was setting the CARGO_PKG_VERSION_* variables before running cargo commands would override whatever cargo was about to set.

In my prototype I added this before where the CARGO_PKG_VERSION_* variables are set in compilation.rs,

https://github.com/juliusl/cargo/blob/0de91c89e6479016d0ed8719fdc2947044335b36/src/cargo/core/compiler/compilation.rs#L329

Here is what I wrote,

        // Determine if this compilation is for a dependency or for a workspace assembly
        let rpath = self.config.registry_base_path();
        let rpath = rpath.as_path_unlocked();

        // If the manifest path starts with the registry path, then it is a compilation for a dependency
        let version = if !pkg.manifest_path().starts_with(rpath) {
            // Handle pkg_version values
            let pkg_version = pkg.version();
            let major = if let Ok(major) = std::env::var("CARGO_PKG_VERSION_MAJOR") {
                major.parse().unwrap_or(pkg_version.major)
            } else {
                pkg_version.major
            };
            let minor = if let Ok(minor) = std::env::var("CARGO_PKG_VERSION_MINOR") {
                minor.parse().unwrap_or(pkg_version.minor)
            } else {
                pkg_version.minor
            };
            let patch = if let Ok(patch) = std::env::var("CARGO_PKG_VERSION_PATCH") {
                patch.parse().unwrap_or(pkg_version.patch)
            } else {
                pkg_version.patch
            };
            let pre = if let Ok(pre) = std::env::var("CARGO_PKG_VERSION_PRE") {
                pre
            } else {
                pkg_version.pre.to_string()
            };

            semver::Version {
                major,
                minor,
                patch,
                pre: pre.parse().unwrap_or(pkg_version.pre.clone()),
                build: pkg_version.build.clone(),
            }
        } else {
            pkg.version().clone()
        };

As a test I created a small app with this as the main.rs:

fn main() {
    println!("Hello, world!");

    println!("{}", env!("CARGO_PKG_VERSION"));
}

And I initialized with,

cargo init
cargo add tokio 

I wanted to add a dependency because I had a suspicion that part of the complexity is that if you handle this by setting an env variable it would apply to all dependencies being compiled. I confirmed that fact and so I handle this issue by checking the manifest path, to figure out if the manifest is from the current working directory or if it was from the cargo registry.

Finally, I tested the above code:

❯ CARGO_PKG_VERSION_PATCH=1004 ../cargo/target/debug/cargo build                                                                                                                                                                                                                                          testver -> master ?
   Compiling pin-project-lite v0.2.13
   Compiling tokio v1.32.0
   Compiling testver v0.1.0 (/home/juliusl/rs/testver)
    Finished dev [unoptimized + debuginfo] target(s) in 1.02s
❯ ./target/debug/testver                                                                                                                                                                                                                                                                                  testver -> master ?
Hello, world!
0.1.1004

I feel like this approach is straightforward enough and is intuitive/aligned with how the rest of cargo works, especially with usage in a CI pipeline.

So, I was wanted to post here and get some feedback and possibly start a PR? Maybe behind a --allow-version-overrides arg action? 👉👈

[faern]

Hello, I've found myself here as someone doing a greenfield project at a large company where most of the current tools we build in CI use dotnet's ability to set the version baked into a binary at build time (shown somewhere far above) on the command line. While carefully controlling a semver in the Cargo.toml is ideal - when your team owns too many CLI tools, and needs to ship and consume them automatically, a generated date-based version number becomes very useful. I was quite surprised to find that cargo build --config package.name=1.2.3-1 and similar attempts, don't work for this. As I'm trying to sway folks into writing more tools in Rust, it's a hard up-hill battle at big companies and little things like this result in us having to put other checks in place, reminders to bump versions, and then parse out what it is in CI. In the meantime, I'm hoping the cargo set-version plugin will work, but now we have to install that in the CI pipeline too :(.

Hi @kyle-rader-msft. We have solved a very similar problem in the following way:

• Setting all versions to 0.0.0 in all Cargo.toml files. This never changes. Cargo always treats all our crates as version 0.0.0
• Having a separate crate for version computation. This crate's build.rs computes the desired version number (can be based off of git commit hash, date or whatever) and exposes the version as a constant in its lib.rs (https://github.com/mullvad/mullvadvpn-app/tree/main/mullvad-version)
• All crates that needs the at-build-time computed version depends on the previous version-computation-crate and uses the constant:
  • For clap version: https://github.com/mullvad/mullvadvpn-app/blob/main/mullvad-cli/src/main.rs#L15
  • For Windows metadata: https://github.com/mullvad/mullvadvpn-app/blob/main/mullvad-cli/build.rs#L10

[kyle-rader-msft]

@faern tank you so much sharing that strategy! I need to delve into what's possible with build.rs :) But off the cuff, this sounds much better than my powershell and bash scripts to live update the Cargo.toml during CI... I'll try this out!

[epage]

FYI making the version field optional is effectively approved, see #12786. It comes with the limitation that you can't publish to a registry.

I think @Faer's solution is great and was what I was getting at with my earlier comment about separating the role of your product's version from Cargo.toml.

What id like to know if there are people who that solution doesn't work for. Most likely that would be people who have a reason for dynamically setting the version and publish to crates.io.

[WhyNotHugo]

Thanks for tackling this @epage

[epage]

#12786 is now merged.

Now that package.version is optional, I feel like this helps us better focus on that field being a tool for registry packages and that it might not be as relevant to products that are not being published to a registry.

In light of that, I propose to the cargo team that we close this issue and instead encourage people to look up the marketing version of their application as they best see fit (e.g. see #6583 (comment)), rather than trying to overload package.version as both the registry version and the marketing version (think Windows build numbers vs marketing version). If there is a reason the registry version needs to be dynamic, let us know!

[WhyNotHugo]

I need the package version to be dynamic (and was previously following #12144).

I use git for version control, so the exact version of my package is stored there, rather than in Cargo.toml. Duplicating the information into Cargo.toml is far from ideal:

• There's an overhead of updating Cargo.toml with the version defined in version control.
• Defining the same value in two places means that they can diverge, at which point it is unclear which one is correct.
• Anyone working on the codebase will permanently have a dirty checkout. They can try committing the new version into Cargo.toml, but it would be out-of-sync immediately after committing.

Ideally, if the current commit matches a git-tag, then that tag is the current version. If the current commit is 20 commit of the last tag,v0.1.0, then the version would be v0.1.0+20. This is essentially the same idea proposed by op in #12144.

As prior art on this topic, in Python-world I use setuptools-scm, which does exactly this: https://github.com/pypa/setuptools_scm

[epage]

@WhyNotHugo is your version for use in the package registry or is it a marketing version?

If its for marketing reasons, then my question would be "why does it need to be in Cargo.toml at all?".

If for the registry, it has the challenges of

• No way to specify path+registry dependencies without having to commit on version updates without something like Allow publishing with path dependencies without a version (RFC2906) #11133 which was originally deferred out of Tracking issue for Deduplicate Cargo workspace information RFC #8415 due to fundamental flaws that were found during implementation
  • We also have Workspace crates should have an implicit patch for crates in the workspace. #6126 which asks for us to adopt the opposite workflow and always specify the version
• Cargo having to decide on what default tag policies should be natively supported and providing a customization point for non-native tag formats
  • Being opinionated is easier when you are a brand new build tool being used for brand new projects
  • We are cautious in what policies we standardize due to our compatibility requirements
  • The precedence of setuptools_scm helps but the Python ecosystem doesn't have a native concept of a workspace which further complicates things
• Being incompatible with shallow clones when being used as a git dependency which we have active work towards supporting
  • Git dependencies are important for registry packages for being able to test out in-work changes in libraries
  • Shallow clone is important for performance, more so for when projects are always using them but having to conditionalize their support is less than ideal
• Requires more work on each cargo invocation
  • Some caching could help reduce it
  • This would only apply to non-registry dependencies as we'd have to put the version into the .crate file's Cargo.toml

Looking at the above, it feels like we can make something work but that it would have enough caveats and limitations that it wouldn't be up to the expected quality for a cargo feature.

That leaves having some kind of plugin system. At that point, I think of the cost (implementation + maintenance + user support) compared to the benefit. The benefits seems relatively small to me especially when there is the "workaround" of release management tools which we will now be pointing people to as of #12745.

[WhyNotHugo]

There is no "marketing version" or anything alike. Please see #12144 for more in-depth details; what I explained above is covered there in greater detail. I'm just talking about publishing regular libraries and applications written in Rust.

The precedence of setuptools_scm helps but the Python ecosystem doesn't have a native concept of a workspace which further complicates things

Python supports multiple libraries in a single git repository too. For overly complex scenarios, setuptools_scm allow configuring the exact git command used to extract the version. It is perfectly possible to filter tags. E.g.: a package in a multi-package repository might only respect tags prefixed with its name.

Being incompatible with shallow clones when being used as a git dependency which we have active work towards supporting

If somebody is explicitly avoiding version metadata when fetching code, then it is expected that version metadata will be absent. I'm not sure that I understand the issue here.

Looking at the above, it feels like we can make something work but that it would have enough caveats and limitations that it wouldn't be up to the expected quality for a cargo feature.

Let's not let perfect be the enemy of good. The current approach is terrible: someone needs to remember to copy-paste the version into Cargo.toml before tagging a new version. That someone also needs to remember to update the lockfile after updating the version and creating the tag (I've made several bug reports to projects that forget to do this step). Essentially, the version must be specified two and a half times (in concert) for each release.

And builds made from intermediate versions (e.g.: N commits after the last tag) just drag the version from the last tag, rather than having a version that identifies them as distinct.

That leaves having some kind of plugin system. At that point, I think of the cost (implementation + maintenance + user support) compared to the benefit. The benefits seems relatively small to me especially when there is the "workaround" of release management tools which we will now be pointing people to as of #12745.

I do agree that plugins might be an overkill here. Something simpler should definitely be the aim.

I looked briefly into the linked PR, but the suggested tools don't make sense to me. For example, cargo-release takes care of publishing and tagging a release. The usual approach for releasing a new version it to create a git-tag, then push that to CI and CI builds the package and pushes it to a registry or wherever. cargo-release seems to fit perfectly when git isn't you method of version control.

When I tag v1.2.0, I am establishing that the given commit is v1.2.0. The fact that cargo needs to read the value from its own source is an annoyance, because we suddenly need tooling to feed the version from the version control system into cargo. Of course, it's possible to put together a small set of shells scripts to patch this all over the place, but that sounds immensely brittle.

[epage]

There is no "marketing version" or anything alike. Please see #12144 for more in-depth details; what I explained above is covered there in greater detail. I'm just talking about publishing regular libraries and applications written in Rust.

I did look at that and the details I was asking for weren't there, hence why I asked.

The precedence of setuptools_scm helps but the Python ecosystem doesn't have a native concept of a workspace which further complicates things

Python supports multiple libraries in a single git repository too. For overly complex scenarios, setuptools_scm allow configuring the exact git command used to extract the version. It is perfectly possible to filter tags. E.g.: a package in a multi-package repository might only respect tags prefixed with its name.

There is a difference in expectations for a solutions to problems when contrasting python's "here are the tools, do it on your own" and cargo's "here is a first-class, native solution with a smooth workflow". We can learn from Python's experience but the Python experience is not optimized for the problems we'll have (it its janky).

Let's not let perfect be the enemy of good. The current approach is terrible: someone needs to remember to copy-paste the version into Cargo.toml before tagging a new version.

I agree with the sentiment generally but taking it too far you gloss over fundamental problems which I think is happening here.. Without solving path dependencies, the problem hasn't gone away unless you restrict this to just single packages. That is a pretty dramatic restriction. Besides likely leading to more frustration when a feature is accessible sometimes but not always, it makes it unclear if a solution that gets stabilized will scale to workspaces. We are then stuck with supporting that solution for ... forever, increasing the load for the cargo team.

. For example, cargo-release takes care of publishing and tagging a release. The usual approach for releasing a new version it to create a git-tag, then push that to CI and CI builds the package and pushes it to a registry or wherever. cargo-release seems to fit perfectly when git isn't you method of version control.

Let's not over-generalize. That is your workflow but I've not seen that generally applied in the Rust community. Instead, I've seen a lot of people fear release automation to the point that they don't even want to adopt a local-only tool that is dry-run by default and optionally logs everything it does to the screen.

[DavidAntliff]

@epage:

Now that package.version is optional

Maybe I'm missing something, but this, with cargo 1.78.0, doesn't seem to behave how I'd expect after reading this entire conversation:

1. Remove [package] version = "x.x.x" from Cargo.toml
2. cargo clean
3. Build with --config:

$ cargo build --config 'package.version="1.2.3"'
   Compiling my-program v0.0.0

I would expect the newly set version number to be applied as the build version.

I'm also using clap with the automatic -V/--version feature, and after building with this, it prints:

$ my-program --version
my-program 0.0.0

I would expect this to reflect the package.version value, as it 100% would if it was set in Cargo.toml instead.

This doesn't seem like "set the package version dynamically" is working? There's a good possibility I've messed something up though.

FWIW, my use case is that I need the version of my program, built in CI, to be the same as other programs built in the same pipeline (monorepo), so rather than modifying Cargo.toml with a sed script, I'm looking to pass the overall version to each program in the pipeline definition. This works fine for Python packages, but I'm having trouble with doing the same for Rust builds. I'm not pushing anything to crates.io BTW, it's all in-house.

Also, I should mention that the version number (used across the monorepo) comes from elsewhere in the pipeline, it's not determinable in the Rust project context - it must be injected.

If a sed script is current best practice, so be it.

EDIT: maybe cargo-edit is the recommended way to do this now?

[faern]

@DavidAntliff See this comment: #6583 (comment)

Now that package.version is optional, I feel like this helps us better focus on that field being a tool for registry packages and that it might not be as relevant to products that are not being published to a registry.

You are not really supposed to put a product version into the Cargo version field, according to this comment. It's versioning for crates.io style registries.

instead encourage people to look up the marketing version of their application as they best see fit (e.g. see #6583 (comment)), rather than trying to overload package.version as both the registry version and the marketing version

It links to my previous comment about how we have solved basically the same problem you have. You will find it is much easier if you stop trying to cram the version into Cargo's version field altogether :)

[bcyran]

That is your workflow but I've not seen that generally applied in the Rust community. Instead, I've seen a lot of people fear release automation to the point that they don't even want to adopt a local-only tool that is dry-run by default and optionally logs everything it does to the screen.

I feel like this is a chicken and egg problem. Of course Rust community doesn't apply that workflow because it's not supported by Cargo or any other tools. Of course they fear release automation if the only way to implement it is with brittle scripts.

I find it surprising that using a Version Control System as a single source of truth for managing versions of your software is a controversial topic.