Components occasionally become corrupted after rustup upgrade

[XAMPPRocky]

Problem

At Embark we usually upgrade to using the latest stable rust compiler soon after its release. We've noticed a consistent spurious issue where your components become corrupted after being upgraded. This happens across different platforms, with different components, and it does not happen to the same install consistently, but we do usually have it happen to at least one or two people on the team every upgrade cycle.

This bug is usually fixed by just re-installing the component or toolchain altogether, so it's not the worst bug, but it is pretty annoying.

Steps

1. rustup upgrade
2. Run a command with the new toolchain.

Errors

the 'cargo.exe' binary, normally provided by the 'cargo' component, is not applicable to the '1.50.0-x86_64-pc-windows-msvc' toolchain

error: the 'cargo-clippy' binary, normally provided by the 'clippy' component, is not applicable to the '1.51.0-aarch64-apple-darwin' toolchain

[rbtcollins]

So this is happening on different OS's?

Fascinating bug. This is BinaryNotProvidedByComponent with description "binary should be provided by component but isn't in current toolchain"

Next time it happens, lets check the following:

• the directly executed binaries (e.g. cargo) should be hardlinks to rustup(.exe on windows)
• the indirectly executed binaries within the toolchain dir, which is what Rustup adds to PATH) should be on disk: check by cding to the toolchain directory in ~/.rustup/toolchains/<toolchain-name>/bin/ , and looking around.
• the components should be in the manifest:
  ~/.rustup/toolchains/<toolchain-name>/lib/rustlib/multirust-config.toml.

That said, I strongly suspect the problem is: #988 (and you may see / #2417 sometimes as well).

[me-diru]

I got the same bug in macOS and it is only happening in one particular codebase terminal.

The error:

error: the 'cargo' binary, normally provided by the 'cargo' component, is not applicable to the '1.51.0-x86_64-apple-darwin' toolchain

I tried updating using rustup update, I get this:

info: syncing channel updates for 'stable-x86_64-apple-darwin'
info: checking for self-updates

  stable-x86_64-apple-darwin unchanged - rustc 1.51.0 (2fd73fabe 2021-03-23)

info: cleaning up downloads & tmp directories

[rbtcollins]

Please perform the debugging steps I requested.

[me-diru]

Steps followed:

• I verified this with find ~/.cargo/bin -samefile ~/.cargo/bin/rustup and cargo is hard-linked to rustup
• cargo is directly executed, so this is not relevant (but I didn't find cargo in my tool-chain)
• I think cargo is present in the toml file you mentioned

  config_version = "1"
  
  [[components]]
  pkg = "cargo"
  target = "x86_64-apple-darwin"
  

Thank you for responding

[rbtcollins]

cargo is not directly executed: you run cargo, but cargo is rustup, rustup the runs the real cargo which is located in the path in the toolchain dir that I gave. If its not there, (and you say it isn't), then the component is either incorrectly unpacked, unpacked but has had elements deleted (e.g. by a virus scanner), or isn't unpacked at all, irrespective of the toolchain config.

See if the following repairs it:
rustup component remove cargo
rustup component add cargo

If so thats pretty strong evidence for the bug with concurrency. How are you normally running rustup? Are you using an editor with rust integration of some sort?

[me-diru]

It worked, cargo is running now :)

I use VSCode, I don't recall doing any special rust integration while installing rust.

[rbtcollins]

@diru1100 how did you trigger the bug? Did you run an update to rust from the CLI? Or from some vscode extension?

[rbtcollins]

@XAMPPRocky any further insight on this? All the evidence so far is the known concurrency limitations as I mention above. I'd like to close this as a dupe, but I don't want to if there is a good chance its a different cause (e.g. some IO corruption on MacOS X).

[me-diru]

@diru1100 how did you trigger the bug? Did you run an update to rust from the CLI? Or from some vscode extension?

hey @rbtcollins, I got the bug in a specific project directory. Rust was working fine in other terminal windows afaik. First I tried doing rustup update, which temporarily fixed the problem until I did what you suggested.

I do have these 3 in the toolchains folder if that helps:

1.49.0-x86_64-apple-darwin
stable-x86_64-apple-darwin
1.51.0-x86_64-apple-darwin 

[XAMPPRocky]

All the evidence so far is the known concurrency limitations as I mention above. I'd like to close this as a dupe

@rbtcollins Feel free to close as dupe.

[rbtcollins]

@diru1100 did you have vscode or another IDE open at the time you ran rustup ?
Oh I think I might have a lead on it - do you have a toolchain fail in that directory? @XAMPPRocky do you use toolchain files?

[me-diru]

@diru1100 did you have vscode or another IDE open at the time you ran rustup ?

Yup, the terminal which gave the error was in VSCode.

Oh I think I might have a lead on it - do you have a toolchain fail in that directory?

Not sure about this

[XAMPPRocky]

@XAMPPRocky do you use toolchain files?

Yeah, we do in our largest projects, and the errors I gave in the original post was from one of those projects.

[rbtcollins]

@diru1100 I think you may have a different bug then, if you don't have a toolchain file.

@XAMPPRocky toolchain files trigger implicit installation. I wonder if something like this is possible:
IDE runs several comands at once: each -> triggers implicit install, concurrency corruption bug occurs.

[tlinford]

@rbtcollins not sure if this is useful, but I came across this when I opened a folder with a rust-toolchain file inside in vs code with rust-analyzer and rust-analyzer downloaded the toolchain.
Fixed it with:
rustup component remove cargo
rustup component add cargo

Also tried to reproduce by removing the toolchain, and letting rust-analyzer fetch it again, but it doesn't seem to happen every time

[rbtcollins]

@XAMPPRocky do your teams use an IDE and rust-analyzer by chance?

[XAMPPRocky]

Yes, 100% I believe most people are using VS Code with rust-analyzer.

[rbtcollins]

Duplicate of #988 for sure at this point.

[Fanisus]

The error is still not solved even in rust 1.73.0

I get error when using VS Code + rust-analyzer

It occurs which i try to compile or build a project and suddenly rust-analyzer decides to do something

[matthiaskrgr]

I think I just ran into this while my disk had some problems while doing a cargo update.
what was very weird was that I rustup uninstalld and reinstall the nightly toolchain several times but that would not fix anything. The rustup component remove, rustup component add trick did work then (for cargo at least)
#2704 (comment)
which I don't understand because I would think removing the entire toolchain should have the same effect 😅

[matthiaskrgr]

Right, manually rustup component remove $i ; rustup component add $i every single component one by one fixed everything in the end...

[ChrisDenton]

Maybe rustup could have a toolchain verify subcommand that checks the health of and fixes corruption in the toolchain and components? Of course it'd be better if the corruption could be avoided in the first place but it might be good to have some obvious method of recovery for when the worst does happen.

Bonus points if it could diagnose the problem and ask the user to send a bug report with that information, but that's maybe a bit extra 🙂.