"cargo check" (incorrectly) shows compiler errors that are not present when running "cargo build"

[RedBreadcat]

Problem
I have a workspace of several crates. Two such crates are "common" (a library crate) and "game_lib", another library that depends on "common".
"common" contains several functions that are pass-by-value that were once pass-by-reference.

// Current
pub fn rotate_point_about_origin(pt: Vector2, rot: Radians, origin: Vector2) -> Vector2 {
    rotate_point(pt - origin, rot) + origin
}

// Old
pub fn rotate_point_about_origin(pt: &Vector2, rot: Radians, origin: &Vector2) -> Vector2 {
    rotate_point(pt - origin, rot) + origin // Some of this implementation was different. I can't recall how. It does not seem relevant though. In any case, it compiled.
}

However, when I run cargo c in game_lib, I'm going getting errors that suggest that the compiler is seeing the old function signature. These errors occur in multiple places in game_lib, not just the example below. They occur for multiple functions inside common too, not just geometry::rotate_point_about_origin.

error[E0308]: mismatched types
   --> game_lib\src\systems\foo\bar.rs:107:53
    |
107 |                 geometry::rotate_point_about_origin(bazz_points[2], -main_camera_rot, view_centre),
    |                                                     ^^^^^^^^^^^^^^
    |                                                     |
    |                                                     expected reference, found struct `Matrix`
    |                                                     help: consider borrowing here: `&bazz_points[2]`
    |
    = note: expected reference `&Matrix<f32, Const<2_usize>, Const<1_usize>, ArrayStorage<f32, 2_usize, 1_usize>>`
                  found struct `Matrix<f32, Const<2_usize>, Const<1_usize>, ArrayStorage<f32, 2_usize, 1_usize>>`

Running cargo b, there is no such false error. rust-analyzer does not see the false error. When I made this change a few days ago, I'm pretty sure I would have run cargo c multiple times, and did not see the error then.
I've encountered this issue a couple of other times in the past. It pops up maybe once every couple of weeks. Running cargo clean always seems to resolve the issue, but I haven't done so yet in case there's valuable information that may be gleamed.

Steps

1. Have a workspace of multiple crates that depend on each other
2. Make some changes in a dependent crate.
3. [I don't know what the direct cause of the bug is.]
4. Run cargo c and see a compiler error.

Possible Solution(s)
This feels like some kind of caching issue. Could cargo be caching some of the intermediate data used by cargo c?

Notes
Output of cargo version --verbose:

cargo 1.55.0 (32da73ab1 2021-08-23)
release: 1.55.0
commit-hash: 32da73ab19417aa89686e1d85c1440b72fdf877d
commit-date: 2021-08-23

rustc 1.55.0 (c8dfcfe04 2021-09-06)
Windows 10

Output of rustup show:

Default host: x86_64-pc-windows-msvc
rustup home:  C:\Users\[snip]\.rustup

installed toolchains
--------------------

stable-x86_64-pc-windows-msvc
nightly-x86_64-pc-windows-msvc

active toolchain
----------------

stable-x86_64-pc-windows-msvc (default)
rustc 1.55.0 (c8dfcfe04 2021-09-06)

[RedBreadcat]

In target/debug/.fingerprint there are many directories. But of note, there are:

• common-22eb78fa992cde90
• common-af40c81771bb33e7
• common-d542ff3ffb485b8a
• common-d820f8239521f0d3
• common-f4d12f0d3a29e09e

I backed up my (apparently buggy) target directory, but then began deleting some of these directories in .fingerprint. I think deleting common-22eb78fa992cde90, common-af40c81771bb33e7, and common-d542ff3ffb485b8a had no effect. Deleting the others resolved the issue.
However, I may have messed something up, as every time I run cargo c now, even with the (buggy) original target directory, I have no issue. I may have been careless with my backup of the directory, and accidentally modified it somehow.

I've attached the directories, in case they contain some useful information.
common.zip

Could these directories contain some timestamp information? I dual-boot with Linux, and whenever returning to Windows my system time has a multi-hour offset.

[ehuss]

Sorry you're running into that. Unfortunately without the full original target, we probably can't know exactly what is happening.

Could these directories contain some timestamp information? I dual-boot with Linux, and whenever returning to Windows my system time has a multi-hour offset.

Yea, that is frustrating, I run into the same problem when dual booting and have never found a solution. This is almost certainly the cause, though. Cargo works with filesystem mtime timestamps. If it gets into a scenario where the timestamp for a fingerprint is in the future, it won't try to recompile something, or if the timestamp for the altered file ended up being far into the past. If you forgot to resync the clock (which I often do) and ran cargo just once (maybe rust-analyzer triggered a build when it started), then the timestamps can get out of whack.

If you run into it again, you can set the CARGO_LOG=cargo::core::compiler::fingerprint=trace environment variable when running cargo. It will tell you why it thinks something should or should not be recompiled. The output is a little verbose, and can be hard to read. However, here is a guide to how it works:

There are two mtime comparisons:

1. The mtime of output files (.rmeta in this case) is compared against the mtimes of its dependencies' output. If any dependency is newer, it gets rebuilt.

   The relevant log messages here are:

   • max output mtime for {:?} is {:?} {}
   • max dep mtime for {:?} is {:?} {}
   • dependency on {} is newer than we are {} > {} {:?}
   • filesystem up-to-date {:?}

2. The mtime of the "dep-info" file in the .fingerprint directory is compared against the mtime of all the source files of a crate. If any source file is newer, it gets rebuilt.

   The relevant log messages here are:

   • all paths up-to-date relative to {:?} mtime={}
   • any message that starts with stale:

The log messages will tell you the mtime values of all these things. You can then compare those mtimes with what you think should be valid. For example, if the dep-info mtime is in the future, or maybe a source file looks to be much older than expected.

One thought I had while writing this is that cargo could check for timestamps that are far in the future compared to the system clock. This can be a little tricky because the timestamps on network filesystems can be set by the server which may not be in sync with the client, but I think cargo has problems with that anyway.

[RedBreadcat]

It's a bit tricky that both past and future timestamps can cause issues. This seems to be a common issue for build systems. There are plenty of stackoverflow questions about make's "clock skew detected" warning, after all. Perhaps cargo could display some kind of warning too?

This article (despite having a pretty opinionated title!) seems to have some reasonable suggestions: https://apenwarr.ca/log/20181113
By using mtime in combination with other filesystem metadata like the size and inode number, it seems that file changes can be detected with higher confidence without needing to resort to calculating a checksum on everything.

Next time I run into the issue I'll collect a trace.

[Eh2406]

We hope to move in the direction suggested in that article when we have a content hash system that works (cc #8623)

[ehuss]

Yea, relying on just mtime is known to be limiting in some cases. I'm going to close as a duplicate of #6529. If you are able to reproduce in such a way that doesn't involve mtime issues, feel free to reopen.

[nerdo]

Years later, this still seems to be an issue.

I'm having a very similar problem with a similar setup. A crate with a few libs and inter-dependencies and for some reason, I kept getting an error in my ide (neovim) about a function not existing, but it does exist and the code compiles and executes cleanly otherwise.

While it's in this state, I copied the directory to another and attempted to zip it but it's too large to share (19GB! I'm assuming, because of dependencies - it's really not a huge program).

I tried running cargo clean in my main workspace, and it seemed to half fix it? I had a few errors but now it's only showing one.

It's quite annoying as I can't trust what cargo check is saying and need to fully compile much more frequently as a result.

It may be worth noting that I'm using macos, and that this project's files are sitting on an external APFS volume that I use to switch back and forth between two macos-based machines... although I'm almost certain I've come across this issue before on projects that were on an internal SSD.

Some output to demonstrate. These two commands were run back-to-back:

reaper/eat_chili/surreal_digestive on  main [$?] is 📦 v0.1.0 via 🦀 v1.69.0
❯ cargo check
    Checking surreal_digestive v0.1.0 (/Volumes/2TB/reaper/eat_chili/surreal_digestive)
error[E0599]: no method named `new_determinate` found for mutable reference `&'life2 mut ProgressProvider` in the current scope
   --> src/lib.rs:124:36
    |
124 |         let issues_step = progress.new_determinate(num_issues as u64);
    |                                    ^^^^^^^^^^^^^^^ help: there is a method with a similar name: `new_indeterminate`

For more information about this error, try `rustc --explain E0599`.
error: could not compile `surreal_digestive` due to previous error

reaper/eat_chili/surreal_digestive on  main [$?] is 📦 v0.1.0 via 🦀 v1.69.0
❯ cargo build
   Compiling futures-sink v0.3.28
   Compiling futures-channel v0.3.28
   Compiling progress_provider v0.1.0 (/Volumes/2TB/reaper/eat_chili/progress_provider)
   Compiling flume v0.10.14
   Compiling sqlx-core v0.6.3
   Compiling futures-util v0.3.28
   Compiling futures-executor v0.3.28
   Compiling tokio-tungstenite v0.18.0
   Compiling futures v0.3.28
   Compiling surrealdb v1.0.0-beta.9+20230402
   Compiling sqlx-macros v0.6.3
   Compiling sqlx v0.6.3
   Compiling chili_provider v0.1.0 (/Volumes/2TB/reaper/eat_chili/chili_provider)
   Compiling surreal_digestive v0.1.0 (/Volumes/2TB/reaper/eat_chili/surreal_digestive)
    Finished dev [unoptimized + debuginfo] target(s) in 14.16s

reaper/eat_chili/surreal_digestive on  main [$?] is 📦 v0.1.0 via 🦀 v1.69.0 took 14s
❯

[peterblockman]

In my case, cargo clean alone did not resolve the problem. I had to install cargo-cache and execute cargo cache -a to fix it.

[irvingoujAtDevolution]

In my case, cargo clean alone did not resolve the problem. I had to install cargo-cache and execute cargo cache -a to fix it.

you are my life saver!

[Barca545]

Unfortunately, cargo cache -a did not fix the problem for me. I additionally tried cargo clean, restarting the server and restarting my computer. I am on Windows 11, VSC, I'm not sure if the only solution I have is switching editors? I'm hoping for a less drastic solution.