Remove Freshness from DependencyQueue
#6832
Conversation
|
r? @nrc (rust_highfive has picked a reviewer for you, use r? to override) |
rust-highfive
added
the
S-waiting-on-review
|
r? @ehuss Note that this is still WIP a bit due to a failure in |
|
(This looks great, btw.) I think I see why Previously it was only keeping a With this, the I'm not sure what a good fix would be. Maybe Could also maybe disable the |
|
Ah right yeah that makes sense. So this is where we relied on dirty propagation through the dependency queue to correctly recompile libraries after the build script ran. With that removed we regained that feature by tracking hashes, but the hashes are too brittle here. Both before and after we correctly deduce that the hash of a build script run has changed but it doesn't need to be recompiled after TBH I found it very difficult to grapple with how the I think a better solution to this might be to completely remove mtime information from hashes. In theory all fingerprints should be 'this file relative to that file', and I wonder if we can do dirty propagation in the initial computation of the |
|
Oh I meant to say as well, IIRC when I first added all the JSON business if we did a deep serialization of fingerprints it actually resulted in gigabytes of json for servo because so many dependencies were repeated so many times! |
|
Alright well it's not a patch to fingerprints in Cargo unless it touches more than 1kloc! I believe all tests should be passing now. The general strategy I settled on was to basically ban This should neatly solve the docker case and did indeed naturally fix it quickly! It did result in some more invasive refactorings though which are all here now. We do have one final instance of So this should now be good to go! r? @ehuss |
Ever since the inception of Cargo and the advent of incremental compilation at the crate level via Cargo, Cargo has tracked whether it needs to recompile something at a unit level in its "dependency queue" which manages when items are ready for execution. Over time we've fixed lots and lots of bugs related to incremental compilation, and perhaps one of the most impactful realizations was that the model Cargo started with fundamentally doesn't handle interrupting Cargo halfway through and resuming the build later. The previous model relied upon implicitly propagating "dirtiness" based on whether the one of the dependencies of a build was rebuilt or not. This information is not available, however, if Cargo is interrupted and resumed (or performs a subset of steps and then later performs more). We've fixed this in a number of places historically but the purpose of this commit is to put a nail in this coffin once and for all. Implicit propagation of whether a unit is fresh or dirty is no longer present at all. Instead Cargo should always know, irrespective of it's in-memory state, whether a unit needs to be recompiled or not. This commit actually turns up a few bugs in the test suite, so later commits will be targeted at fixing this. Note that this required a good deal of work on the `fingerprint` module to fix some longstanding bugs (like rust-lang#6780) and some serious hoops had to be jumped through for others (like rust-lang#6779). While these were fallout from this change they weren't necessarily the primary motivation, but rather to help make `fingerprints` a bit more straightforward in what's an already confusing system! Closes rust-lang#6780
This has proven to be a very unreliable piece of information to hash, so let's not! Instead we track what files are supposed to be relative to, and we check both mtimes when necessary.
Removes the need for a wonky `with_local` method!
|
Should be updated! |
|
Nice! |
|
📌 Commit 22691b9 has been approved by |
bors
added
S-waiting-on-bors
Remove `Freshness` from `DependencyQueue` Ever since the inception of Cargo and the advent of incremental compilation at the crate level via Cargo, Cargo has tracked whether it needs to recompile something at a unit level in its "dependency queue" which manages when items are ready for execution. Over time we've fixed lots and lots of bugs related to incremental compilation, and perhaps one of the most impactful realizations was that the model Cargo started with fundamentally doesn't handle interrupting Cargo halfway through and resuming the build later. The previous model relied upon implicitly propagating "dirtiness" based on whether the one of the dependencies of a build was rebuilt or not. This information is not available, however, if Cargo is interrupted and resumed (or performs a subset of steps and then later performs more). We've fixed this in a number of places historically but the purpose of this commit is to put a nail in this coffin once and for all. Implicit propagation of whether a unit is fresh or dirty is no longer present at all. Instead Cargo should always know, irrespective of it's in-memory state, whether a unit needs to be recompiled or not. This commit actually turns up a few bugs in the test suite, so later commits will be targeted at fixing this. Note that this required a good deal of work on the `fingerprint` module to fix some longstanding bugs (like #6780) and some serious hoops had to be jumped through for others (like #6779). While these were fallout from this change they weren't necessarily the primary motivation, but rather to help make `fingerprints` a bit more straightforward in what's an already confusing system! Closes #6780
|
☀️ Test successful - checks-travis, status-appveyor |
Update cargo 12 commits in 6f3e9c367abb497c64f360c3839dab5e74928d5c..ef0223f12597b5e0d9d2feed1b92c41306b1fc05 2019-04-04 14:11:33 +0000 to 2019-04-15 14:36:55 +0000 - Fix test include_overrides_gitignore. (rust-lang/cargo#6850) - Clarify optional registry key behaviour (rust-lang/cargo#6851) - Ensure Summary::checksum works for registry crates (rust-lang/cargo#6842) - Better error if PathSource::walk can't access something. (rust-lang/cargo#6841) - Improve warning in `cargo new` with parse error. (rust-lang/cargo#6839) - Improve error message for `publish` key restriction. (rust-lang/cargo#6838) - Remove `Freshness` from `DependencyQueue` (rust-lang/cargo#6832) - testsuite: cleanup for `alternative-registries` (rust-lang/cargo#6837) - Improve error message to rerun a test in a workspace. (rust-lang/cargo#6824) - Fix mutable_borrow_reservation_conflict warning. (rust-lang/cargo#6829) - Add install-upgrade. (rust-lang/cargo#6798) - Clarify docs of install without <crate> (rust-lang/cargo#6823)
Update cargo 12 commits in 6f3e9c367abb497c64f360c3839dab5e74928d5c..ef0223f12597b5e0d9d2feed1b92c41306b1fc05 2019-04-04 14:11:33 +0000 to 2019-04-15 14:36:55 +0000 - Fix test include_overrides_gitignore. (rust-lang/cargo#6850) - Clarify optional registry key behaviour (rust-lang/cargo#6851) - Ensure Summary::checksum works for registry crates (rust-lang/cargo#6842) - Better error if PathSource::walk can't access something. (rust-lang/cargo#6841) - Improve warning in `cargo new` with parse error. (rust-lang/cargo#6839) - Improve error message for `publish` key restriction. (rust-lang/cargo#6838) - Remove `Freshness` from `DependencyQueue` (rust-lang/cargo#6832) - testsuite: cleanup for `alternative-registries` (rust-lang/cargo#6837) - Improve error message to rerun a test in a workspace. (rust-lang/cargo#6824) - Fix mutable_borrow_reservation_conflict warning. (rust-lang/cargo#6829) - Add install-upgrade. (rust-lang/cargo#6798) - Clarify docs of install without <crate> (rust-lang/cargo#6823)
Update cargo 16 commits in 6f3e9c367abb497c64f360c3839dab5e74928d5c..b6581d383ed596b133e330011658c6f83cf85c2f 2019-04-04 14:11:33 +0000 to 2019-04-16 16:02:11 +0000 - Fix new_warning_with_corrupt_ws missing "USER". (rust-lang/cargo#6857) - Ignore Clippy redundant_closure (rust-lang/cargo#6855) - Pass OsStr/OsString args through to the process spawned by cargo run. (rust-lang/cargo#6849) - Bump to 0.37.0 (rust-lang/cargo#6852) - Fix test include_overrides_gitignore. (rust-lang/cargo#6850) - Clarify optional registry key behaviour (rust-lang/cargo#6851) - Ensure Summary::checksum works for registry crates (rust-lang/cargo#6842) - Better error if PathSource::walk can't access something. (rust-lang/cargo#6841) - Improve warning in `cargo new` with parse error. (rust-lang/cargo#6839) - Improve error message for `publish` key restriction. (rust-lang/cargo#6838) - Remove `Freshness` from `DependencyQueue` (rust-lang/cargo#6832) - testsuite: cleanup for `alternative-registries` (rust-lang/cargo#6837) - Improve error message to rerun a test in a workspace. (rust-lang/cargo#6824) - Fix mutable_borrow_reservation_conflict warning. (rust-lang/cargo#6829) - Add install-upgrade. (rust-lang/cargo#6798) - Clarify docs of install without <crate> (rust-lang/cargo#6823)
Ever since the inception of Cargo and the advent of incremental
compilation at the crate level via Cargo, Cargo has tracked whether it
needs to recompile something at a unit level in its "dependency queue"
which manages when items are ready for execution. Over time we've fixed
lots and lots of bugs related to incremental compilation, and perhaps
one of the most impactful realizations was that the model Cargo started
with fundamentally doesn't handle interrupting Cargo halfway through and
resuming the build later.
The previous model relied upon implicitly propagating "dirtiness" based
on whether the one of the dependencies of a build was rebuilt or not.
This information is not available, however, if Cargo is interrupted and
resumed (or performs a subset of steps and then later performs more).
We've fixed this in a number of places historically but the purpose of
this commit is to put a nail in this coffin once and for all.
Implicit propagation of whether a unit is fresh or dirty is no longer
present at all. Instead Cargo should always know, irrespective of it's
in-memory state, whether a unit needs to be recompiled or not. This
commit actually turns up a few bugs in the test suite, so later commits
will be targeted at fixing this.
Note that this required a good deal of work on the
fingerprintmoduleto fix some longstanding bugs (like #6780) and some serious hoops had to
be jumped through for others (like #6779). While these were fallout from
this change they weren't necessarily the primary motivation, but rather
to help make
fingerprintsa bit more straightforward in what's analready confusing system!
Closes #6780