-
Notifications
You must be signed in to change notification settings - Fork 2.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Turn the new lock file format on by default #7579
Conversation
r? @ehuss (rust_highfive has picked a reviewer for you, use r? to override) |
So I imagine this will be problematic for projects that have extra long backwards compatibility. Is it correct that this is a forced upgrade for everyone? What about having it so that it only applies to new projects for a while? That is, it only uses the new format if |
Yeah that's true that this would be an upgrade for everyone, but it notably only affects projects which span many toolchain versions and are also actively developed on all those versions. While that seems pretty rare to me, I think you've got a good idea to start out by just phasing in with new lock files, so I'll see if I can whip that up. |
This commit enables the support added in rust-lang#7070 by default. This means that gradually over time all `Cargo.lock` files will be migrated to the new format. Cargo shipped with Rust 1.38.0 supports this new lock file format, so any project using Rust 1.38.0 or above will converge quickly onto the new lock file format and continue working. The main benefit of the new format is to be more friendly to git merge conflicts. Information is deduplicated throughout the lock file to avoid verbose `depedencies` lists and the `checksum` data is all listed inline with `[[package]]`. This has been deployed with rust-lang/rust for some time now and it subjectively at least seems to have greatly reduced the amount of bouncing that happens for touching `Cargo.lock`.
This commit adds support to Cargo and refactors the lockfile versioning slightly. The goal here is that Cargo continually has two thresholds of lockfile formats to create: * One is used for new lock files at all times * The other is used to update old lock files The logic for these two thresholds is appropriately updated throughout Cargo, and then this commit also preserves the previous update where new lock files will get the new format, but old lockfiles will continue to retain the old format by default.
5789ec8
to
013c1af
Compare
Ok updated! |
@alexcrichton the macos test failure looks legitimate. I haven't been able to fully dig into it, but it looks like it is rebuilding when it shouldn't. I can maybe try to dig more tonight. |
@alexcrichton Here's a repro (any platform):
The issue is that the I'm a bit surprised this didn't trip up any other tests. |
Ok I've added a new test and pushed up what I believe should be a fix for this, but it's a pretty targeted fix and I don't have a ton of confidence in it. Next time we change formats I think we should do so via #7144 since that'll make this much easier to deal with. |
Seems good enough to me. @bors r+ |
📌 Commit c37a46c has been approved by |
Turn the new lock file format on by default This commit enables the support added in #7070 by default. This means that gradually over time all `Cargo.lock` files will be migrated to the new format. Cargo shipped with Rust 1.38.0 supports this new lock file format, so any project using Rust 1.38.0 or above will converge quickly onto the new lock file format and continue working. The main benefit of the new format is to be more friendly to git merge conflicts. Information is deduplicated throughout the lock file to avoid verbose `depedencies` lists and the `checksum` data is all listed inline with `[[package]]`. This has been deployed with rust-lang/rust for some time now and it subjectively at least seems to have greatly reduced the amount of bouncing that happens for touching `Cargo.lock`.
☀️ Test successful - checks-azure |
Update cargo, rls, books. ## nomicon 1 commits in 58e36e0e08dec5a379ac568827c058e25990d6cd..041c46e692a2592853aeca132c8dfe8eb5a79a9e 2019-10-30 08:14:24 -0500 to 2019-11-20 16:46:45 +0100 - Update unsafe-code-guidelines link (rust-lang/nomicon#175) ## cargo 15 commits in 8280633db680dec5bfe1de25156d1a1d53e6d190..750cb1482e4d0e74822cded7ab8b3c677ed8b041 2019-11-11 23:17:05 +0000 to 2019-11-23 23:06:36 +0000 - Some random comments and docstrings. (rust-lang/cargo#7625) - Add value OUT_DIR to build-script-executed JSON message (rust-lang/cargo#7622) - Update documentation for custom target dependencies. (rust-lang/cargo#7623) - Document private items for binary crates by default (rust-lang/cargo#7593) - Extend documentation on security concerns of crate names in a registry. (rust-lang/cargo#7616) - Stabilize install-upgrade. (rust-lang/cargo#7560) - Turn the new lock file format on by default (rust-lang/cargo#7579) - bump im-rc version (rust-lang/cargo#7609) - Ignore file lock errors if unsupported, on Windows (rust-lang/cargo#7602) - Add hack for fwdansi change. (rust-lang/cargo#7607) - Document Cargo's JSON output. (rust-lang/cargo#7595) - Remove "cargo login" from user input when asking for login token. (rust-lang/cargo#7588) - Fix all Clippy suggestions (but not add it to CI 🙃) (rust-lang/cargo#7574) - Add kind/platform info to `cargo metadata` (rust-lang/cargo#7132) - Update core-foundation requirement from 0.6.0 to 0.7.0 (rust-lang/cargo#7585) ## reference 2 commits in 45558c4..9e843ae 2019-11-08 14:47:35 +0100 to 2019-11-24 17:44:04 +0100 - Minor never type additions. (rust-lang/reference#723) - Update associated-items.md. "it"->is (rust-lang/reference#721) ## book 3 commits in e79dd62aa63396714278d484d91d48826737f47f..81ebaa2a3f88d4d106516c489682e64cacba4f60 2019-10-30 07:33:12 -0500 to 2019-11-15 08:30:04 -0800 - small fix ch04-03 & code block typo ch07-02 (rust-lang/book#2138) - Adapt content of Chapter 16.3 in order to be consistent with improved compiler message (rust-lang/book#1779) - [Rust 1.35] Remove FnBox and use builtin impl FnOnce for Box<FnOnce()> instead. (rust-lang/book#1906) ## rls 3 commits in 5db91c7b94ca81eead6b25bcf6196b869a44ece0..9ec2b8cb57c87517bcb506ac302eae339ffa2025 2019-10-30 16:04:39 +0100 to 2019-11-24 23:16:11 +0100 - Fix test for latest nightly. (rust-lang/rls#1595) - doc: contributing: Remove outdated LSP extension (rust-lang/rls#1594) - Update cargo. (rust-lang/rls#1591) ## rust-by-example 1 commits in dcee312c66267eb5a2f6f1561354003950e29105..4835e025826729827a94fdeb7cb85fed288d08bb 2019-10-31 11:26:53 -0300 to 2019-11-14 09:20:43 -0300 - crates: fix suggested value for --crate-type flag (rust-lang/rust-by-example#1292) ## edition-guide 1 commits in f553fb26c60c4623ea88a1cfe731eafe0643ce34..6601cab4666596494a569f94aa63b7b3230e9769 2019-10-30 08:27:42 -0500 to 2019-11-22 12:08:58 -0500 - Remove final nursery reference
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is there some way to continue using the old format? I have CI testing -Z minimal-versions
, but I test that configuration with my MSRV, which is older than 1.38, so some -Z
flag to get the older format would be appreciated.
Can migrate to an issue if that is better.
@@ -59,9 +59,12 @@ pub struct Resolve { | |||
/// | |||
/// It's theorized that we can add more here over time to track larger changes | |||
/// to the `Cargo.lock` format, but we've yet to see how that strategy pans out. | |||
#[derive(PartialEq, Clone, Debug)] | |||
#[derive(PartialEq, Eq, Clone, Copy, Debug, PartialOrd, Ord)] | |||
pub enum ResolveVersion { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The doccomment for this still says V1
is the default.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If you have the old format, it will continue to be used. If you deleted the lock file, you can use an older version of Cargo to recreate it. |
No, I can't, because I generate the file with nightly to get |
I guess to clarify, this is a library, so there's no |
And if a |
Unfortunately we are recommending that people discontinue using |
The new format [1,2] dropped the [metadata] table. As a consequence our cargo-crates.awk script no longer outputs CARGO_CRATES. We can get the crate list from the various [[package]] tables instead. This should work with the new as well as the old format. [1] rust-lang/cargo#7070 [2] rust-lang/cargo#7579 PR: 242416 Reported by: jbeich git-svn-id: svn+ssh://svn.freebsd.org/ports/head@519063 35697150-7ecd-e111-bb59-0022644237b5
The new format [1,2] dropped the [metadata] table. As a consequence our cargo-crates.awk script no longer outputs CARGO_CRATES. We can get the crate list from the various [[package]] tables instead. This should work with the new as well as the old format. [1] rust-lang/cargo#7070 [2] rust-lang/cargo#7579 PR: 242416 Reported by: jbeich
The new format [1,2] dropped the [metadata] table. As a consequence our cargo-crates.awk script no longer outputs CARGO_CRATES. We can get the crate list from the various [[package]] tables instead. This should work with the new as well as the old format. [1] rust-lang/cargo#7070 [2] rust-lang/cargo#7579 PR: 242416 Reported by: jbeich git-svn-id: svn+ssh://svn.freebsd.org/ports/head@519063 35697150-7ecd-e111-bb59-0022644237b5
Wasmtime no longer builds. This was debug to be an error parsing cargo.lock while building. According to rust-lang/cargo#7579 it seems rust prior to 1.38 is now obsolete w.r.t. to changes to the lock file format. Also updates the url for the repositories pointed to for wasmtime and lucet.
cargo as shipped in rustc 1.38.0 supports this format, while or current MSRV is 1.39.0. See: rust-lang/cargo#7579
Version 1.41.0 (2020-01-30) =========================== Language -------- - [You can now pass type parameters to foreign items when implementing traits.][65879] E.g. You can now write `impl<T> From<Foo> for Vec<T> {}`. - [You can now arbitrarily nest receiver types in the `self` position.][64325] E.g. you can now write `fn foo(self: Box<Box<Self>>) {}`. Previously only `Self`, `&Self`, `&mut Self`, `Arc<Self>`, `Rc<Self>`, and `Box<Self>` were allowed. - [You can now use any valid identifier in a `format_args` macro.][66847] Previously identifiers starting with an underscore were not allowed. - [Visibility modifiers (e.g. `pub`) are now syntactically allowed on trait items and enum variants.][66183] These are still rejected semantically, but can be seen and parsed by procedural macros and conditional compilation. Compiler -------- - [Rustc will now warn if you have unused loop `'label`s.][66325] - [Removed support for the `i686-unknown-dragonfly` target.][67255] - [Added tier 3 support\* for the `riscv64gc-unknown-linux-gnu` target.][66661] - [You can now pass an arguments file passing the `@path` syntax to rustc.][66172] Note that the format differs somewhat from what is found in other tooling; please see [the documentation][argfile-docs] for more information. - [You can now provide `--extern` flag without a path, indicating that it is available from the search path or specified with an `-L` flag.][64882] \* Refer to Rust's [platform support page][forge-platform-support] for more information on Rust's tiered platform support. [argfile-docs]: https://doc.rust-lang.org/nightly/rustc/command-line-arguments.html#path-load-command-line-flags-from-a-path Libraries --------- - [The `core::panic` module is now stable.][66771] It was already stable through `std`. - [`NonZero*` numerics now implement `From<NonZero*>` if it's a smaller integer width.][66277] E.g. `NonZeroU16` now implements `From<NonZeroU8>`. - [`MaybeUninit<T>` now implements `fmt::Debug`.][65013] Stabilized APIs --------------- - [`Result::map_or`] - [`Result::map_or_else`] - [`std::rc::Weak::weak_count`] - [`std::rc::Weak::strong_count`] - [`std::sync::Weak::weak_count`] - [`std::sync::Weak::strong_count`] Cargo ----- - [Cargo will now document all the private items for binary crates by default.][cargo/7593] - [`cargo-install` will now reinstall the package if it detects that it is out of date.][cargo/7560] - [Cargo.lock now uses a more git friendly format that should help to reduce merge conflicts.][cargo/7579] - [You can now override specific dependencies's build settings][cargo/7591] E.g. `[profile.dev.overrides.image] opt-level = 2` sets the `image` crate's optimisation level to `2` for debug builds. You can also use `[profile.<profile>.build_overrides]` to override build scripts and their dependencies. Misc ---- - [You can now specify `edition` in documentation code blocks to compile the block for that edition.][66238] E.g. `edition2018` tells rustdoc that the code sample should be compiled the 2018 edition of Rust. - [You can now provide custom themes to rustdoc with `--theme`, and check the current theme with `--check-theme`.][54733] - [You can use `#[cfg(doc)]` to compile an item when building documentation.][61351] Compatibility Notes ------------------- - [As previously announced 1.41.0 will be the last tier 1 release for 32-bit Apple targets.][apple-32bit-drop] This means that the source code is still available to build, but the targets are no longer being tested and release binaries for those platforms will no longer be distributed by the Rust project. Please refer to the linked blog post for more information. [54733]: rust-lang/rust#54733 [61351]: rust-lang/rust#61351 [67255]: rust-lang/rust#67255 [66661]: rust-lang/rust#66661 [66771]: rust-lang/rust#66771 [66847]: rust-lang/rust#66847 [66238]: rust-lang/rust#66238 [66277]: rust-lang/rust#66277 [66325]: rust-lang/rust#66325 [66172]: rust-lang/rust#66172 [66183]: rust-lang/rust#66183 [65879]: rust-lang/rust#65879 [65013]: rust-lang/rust#65013 [64882]: rust-lang/rust#64882 [64325]: rust-lang/rust#64325 [cargo/7560]: rust-lang/cargo#7560 [cargo/7579]: rust-lang/cargo#7579 [cargo/7591]: rust-lang/cargo#7591 [cargo/7593]: rust-lang/cargo#7593 [`Result::map_or_else`]: https://doc.rust-lang.org/std/result/enum.Result.html#method.map_or_else [`Result::map_or`]: https://doc.rust-lang.org/std/result/enum.Result.html#method.map_or [`std::rc::Weak::weak_count`]: https://doc.rust-lang.org/std/rc/struct.Weak.html#method.weak_count [`std::rc::Weak::strong_count`]: https://doc.rust-lang.org/std/rc/struct.Weak.html#method.strong_count [`std::sync::Weak::weak_count`]: https://doc.rust-lang.org/std/sync/struct.Weak.html#method.weak_count [`std::sync::Weak::strong_count`]: https://doc.rust-lang.org/std/sync/struct.Weak.html#method.strong_count [apple-32bit-drop]: https://blog.rust-lang.org/2020/01/03/reducing-support-for-32-bit-apple-targets.html
My group just hit this, and it seems like the error message that older cargo produces when reading a newer lockfile is particularly unhelpful -- I don't know that anything can be done about this now, but it doesn't seem great in terms of the debugging-time cost for people that encounter it. (Backwards-incompatible changes are easier to deal with if they at least produce obvious failures pointing to the need to upgrade.) I can't imagine this is really fixable at this point, since the behavior of old cargo can't be changed, and the format can't be adjusted without making a second breaking change, but is there some way this could be improved for the future? I see that newer cargo understands the file to be implicitly versioned, and determines this by its contents. But older cargo of course doesn't understand this and just crashes. For future updates, would an explicit version field help avoid this, so that older cargo can declare that it does not understand the newer format? (An approach I've seen before, which I think would work well here, would be a "minimum cargo version to parse this file" at the top of the file -- not the version that generated the file, but the first version that supports the format being used. Then older versions could not only error out helpfully, they could even point to what version is required to upgrade to.) |
@mathstuf I have duct taped together some utterly terrible code, that will mangle a new style Cargo.lock contents back into an old style lock file in "Cargo.lock". Its presented as-is in the hope its useful to somebody. It may be a bit messy to read given how I've had to implement parts of a statemachine and various parser grammar in-line, so ... good luck extending it. Regex ninja skills may be required, but all the regex is relatively simple, but in ways that mean anything weird I didn't see in my own lock files may not be dealt with at all. The logic is doable in rust, but I wanted to spend less than an hour making something work. Enjoy? Feel free to hate the perl. ( I had initially embarked on hand-transcribing a lockfile in a text editor before I decided it would be faster and more reliable to script it poorly ) #!perl
use strict;
use warnings;
# This is very crude.
#
# 1. Read ./Cargo.lock trying to extract crate name/version/checksum/source details from 'new' format
# 2. Read Cargo.lock a second time, writing it to Cargo.lock.new streamed, but
# a. strip checksum = from [[ package ]] entries on the way through
# b. rewrite dependency = [ sections using data collected on the first pass
# 3. Then, generate a [metadata] section from the data obtained in the first pass and emit to the new file.
#
# Its up to you to copy the new file back into place.
my $states = parse_meta("Cargo.lock");
rewrite_file("Cargo.lock","Cargo.lock.new", $states);
1;
sub fix_dep {
my ( $depstring, $states ) = @_;
if ( $depstring =~ /^([^ ]+)$/ ) {
# single map
my $package = $1;
if ( not exists $states->{$package} ) {
die "$package not found";
}
my ( $first, @garbage ) = values %{ $states->{$package} };
if (@garbage) {
die "Too many versions for $package";
}
return
$package . " "
. $first->{version} . " ("
. $first->{source} . ")";
}
if ( $depstring =~ /^([^ ]+)[ ]+([^ ]+)$/ ) {
my ( $package, $version ) = ( $1, $2 );
if ( not exists $states->{$package} ) {
die "$package not found";
}
if ( not exists $states->{$package}->{$version} ) {
die "$package v$version not found";
}
return
$package . " "
. $version . " ("
. $states->{$package}->{$version}->{source} . ")";
}
die "Unhandled depstring $depstring";
}
sub parse_meta {
my ($filename) = @_;
open my $fh, "<", $filename or die "Can't read $filename";
my $states = {};
lines: while ( my $line = <$fh> ) {
next unless $line =~ /^\[\[package\]\]/;
my $record;
fields: while ( my $line = <$fh> ) {
last fields if $line =~ /^\s*$/;
if ( $line =~ /^([^ ]+) = "([^"]+)"/ ) {
my ( $field, $value ) = ( $1, $2 );
$record->{$field} = $value;
}
}
if ( not exists $record->{name} ) {
die "Didn't parse record name";
}
if ( not exists $record->{version} ) {
die "Didn't parse record version for $record->{name}";
}
if ( not exists $record->{source} ) {
warn
"Didn't parse record source for $record->{name} $record->{version}";
}
if ( not exists $record->{checksum} ) {
warn
"Didn't parse record checksum for $record->{name} $record->{version}";
}
$states->{ $record->{name} }->{ $record->{version} } = $record;
}
return $states;
}
sub rewrite_file {
my ( $src, $dest, $states ) = @_;
open my $fh, "<", $src or die "Can't read $src";
open my $wfh, ">", $dest or die "Cant write $dest";
lines: while ( my $line = <$fh> ) {
if ( $line !~ /^\[\[package\]\]/ ) {
$wfh->print($line);
next lines;
}
$wfh->print($line);
fields: while ( my $line = <$fh> ) {
if ( $line =~ /^\s*$/ ) {
$wfh->print($line);
last fields;
}
if ( $line =~ /^([^ ]+) = "/ ) {
if ( "$1" ne "checksum" ) {
$wfh->print($line);
next fields;
}
}
if ( $line =~ /^dependencies = \[/ ) {
$wfh->print($line);
dep: while ( my $dep = <$fh> ) {
if ( $dep =~ /^\]/ ) {
$wfh->print($dep);
next fields;
}
if ( $dep =~ /^(\s+")([^"]+)(".+\z)/ms ) {
my ( $lpad, $content, $rpad ) = ( $1, $2, $3 );
$wfh->print(
$lpad . fix_dep( $content, $states ) . $rpad );
next dep;
}
die "Unhandled $dep";
}
}
}
}
# write a metadata section
$wfh->print("\n[metadata]\n");
for my $crate ( sort keys %{$states} ) {
for my $version ( sort keys %{$states->{$crate}} ) {
next unless exists $states->{$crate}->{$version}->{checksum};
$wfh->printf(qq["checksum %s %s (%s)" = "%s"\n],
$crate,
$version,
$states->{$crate}->{$version}->{source},
$states->{$crate}->{$version}->{checksum});
}
}
}
1; |
Btw, there is also the cargo-lock crate which can translate between formats: https://github.com/rustsec/cargo-lock |
This is mostly due to a backwards-incompatible change to the Cargo.lock format described here: rust-lang/cargo#7579
Between toml:0.5.6 and toml:0.5.7 a parse error message changed from config parsing failed: invalid number at line 2 column 15 to config parsing failed: invalid TOML value, did you mean to use a quoted string? at line 2 column 15 causing test_invalid_toml_from_string to fail on fresh builds. Update all dependencies and adjust this test accordingly. But because of a new lock file format in Rust 1.38 [1], "Cargo update" now produces a lock file that Cargo from Rust 1.36 can't read, causing build pinned-linux with MSRV 1.36 to fail. Fortunately cargo-lock [2] can forge a "v1" format from the new file. [1] rust-lang/cargo#7579 [2] https://github.com/rustsec/cargo-lock
As discussed in commit 1a18345 ("Update dependenies", 2020-12-17), Rust 1.38's Cargo produces a backwards incompatible lock file format [1], making "cargo update" effectively not work. cargo-lock [2] can change the format of a lock file but that really doesn't fit into normal operation. Bump MSRV to eliminate this hurdle; and move all the way to 1.40 to also eliminate the platform MSRV split. [1] rust-lang/cargo#7579 [2] https://github.com/rustsec/cargo-lock
This commit enables the support added in #7070 by default. This means
that gradually over time all
Cargo.lock
files will be migrated to thenew format. Cargo shipped with Rust 1.38.0 supports this new lock file
format, so any project using Rust 1.38.0 or above will converge quickly
onto the new lock file format and continue working.
The main benefit of the new format is to be more friendly to git merge
conflicts. Information is deduplicated throughout the lock file to avoid
verbose
depedencies
lists and thechecksum
data is all listed inlinewith
[[package]]
. This has been deployed with rust-lang/rust for sometime now and it subjectively at least seems to have greatly reduced the
amount of bouncing that happens for touching
Cargo.lock
.