reqwest http handler causes silent failure on AAarch64 Windows

[naixx]

Problem

When I run rustup-init.exe it doesn't download any toolchains.

Steps

1. rustup default stable

C:\Users\a\.cargo\bin>rustup default stable
info: syncing channel updates for 'stable-x86_64-pc-windows-msvc'

C:\Users\a\.cargo\bin>

2. No output on explicit install

C:\Users\a\.cargo\bin>rustup toolchain install stable
info: syncing channel updates for 'stable-x86_64-pc-windows-msvc'

C:\Users\a\.cargo\bin>

3. No downloads in logging

C:\Users\a\.cargo\bin>rustup --verbose update stable
verbose: read metadata version: '12'
verbose: installing toolchain 'stable-x86_64-pc-windows-msvc'
verbose: toolchain directory: 'C:\Users\a\.rustup\toolchains\stable-x86_64-pc-windows-msvc'
info: syncing channel updates for 'stable-x86_64-pc-windows-msvc'
verbose: creating temp file: C:\Users\astra\.rustup\tmp\cizkbcsni3xp7r1c_file
verbose: downloading file from: 'https://static.rust-lang.org/dist/channel-rust-stable.toml.sha256'
verbose: downloading with reqwest

C:\Users\a\.cargo\bin>

Possible Solution(s)

Kinnison from discord suggested to use RUSTUP_USE_CURL=1
And that helped
His comments are:

OK that's interesting, it implies something is wrong with how reqwest is looking for stuff on your system and thus it fails
(a) it shouldn't silently fail
and (b) eventually RUSTUP_USE_CURL is going away

Notes

No response

Rustup version

1.26

Installed toolchains

no toolchains

[Alovchin91]

Thanks for the workaround @naixx @kinnison! I've just faced it on Windows aarch64. I'm gonna take a look if I can nail the issue down.

[djc]

@Alovchin91 @naixx would either you be game to try to chase this down? The way I would do it is just by injecting a bunch of dbg!() spew in the download code (printf-style debugging) (starting from download_to_path_with_backend(), but I don't have any AArch64 Windows machines available to me. If it helps, I can make a branch with a bunch of spew on it and then you can compile + run it to see what happens?

[Alovchin91]

@djc Sure, I would be happy to help! Sorry, I was busy with a new job so I didn’t get to it just yet. If you’d have a branch, I could clone it and run it this week.

I’m not sure though if @naixx was facing this issue on an Arm64 machine though? 🤔 If not, I don’t have an x64 machine to test on, so I can only help with the Arm64 part. It does reproduce on Arm64 for me quite reliably though.

[djc]

@Alovchin91 no need to apologize! Let's dig out the AArch64 issue, please try the branch from #3462.

[Alovchin91]

@djc Sorry for a late reply; I was trying to reproduce the issue with a Rustup version built from your branch to no avail 🤷‍♂️ I could however reproduce it again (after quite a number of attempts) using Rustup 1.26.0 and record a crash dump using Procdump.

The issue seems to be a null pointer somewhere in the reqwest-internal-sync-runtime thread. If you're interested, I could share the crash dump with you privately.

[Alovchin91]

One thing to note though is that with the build from your branch, downloads do sometimes become very slow. In those cases I see debug messages like the following:

...
[download\src\lib.rs:118] data.len() = 16384
[download\src\lib.rs:317] bytes_read = 16384
[download\src\lib.rs:118] data.len() = 16384
[download\src\lib.rs:317] bytes_read = 128
[download\src\lib.rs:118] data.len() = 128
[download\src\lib.rs:317] bytes_read = 16384
[download\src\lib.rs:118] data.len() = 16384
...

-- with Rustup waiting for a few seconds after reading those 128 bytes before proceeding to download. So maybe in a build without dbg! there's a race condition that triggers when the download becomes slow.

[djc]

The issue seems to be a null pointer somewhere in the reqwest-internal-sync-runtime thread. If you're interested, I could share the crash dump with you privately.

Huh, that seems pretty surprising! Would be happy to inspect the crash dump. You should be able to find my email address online, can you send me an email with it?

[Alovchin91]

@djc I have sent you an email with 3 types of dumps I've recorded, and here's an error message I've caught once at some point while trying to do Time-Travel Debugging:

error: could not download file from 'https://static.rust-lang.org/dist/channel-rust-1.71.1.toml.sha256' to 'C:\Users\a\.rustup\tmp\e68g3byz2dhv7pfp_file': failed to make network request: error sending request for url (https://static.rust-lang.org/dist/channel-rust-1.71.1.toml.sha256): connection error: The specified data could not be decrypted. (os error -2146893008): connection error: The specified data could not be decrypted. (os error -2146893008): The specified data could not be decrypted. (os error -2146893008)

-2146893008 is SEC_E_DECRYPT_FAILURE and I can see bcrypt.dll loaded in the 1st chance exception dump. So it might or might not be related to the issue at hand.

[Alovchin91]

@djc A heads-up: I haven't encountered this issue for a while now. Could have been fixed in Windows or in reqwest.

[djc]

@Alovchin91 thanks for letting us know, and I feel very bad for having failed to follow up on your email with debug stuff. Note that, given that the error makes it look like this failure might have been related to the use of native-tls on Windows, rustup 1.28 will default to using rustls which makes it even less likely to trigger issues like this.

[rami3l]

A heads-up: I haven't encountered this issue for a while now. Could have been fixed in Windows or in reqwest.

@Alovchin91 Thanks for the update! Would you mind if I close this as stale until the issue resurfaces?

[Alovchin91]

Would you mind if I close this as stale until the issue resurfaces?

@rami3l Sure, feel free to! Thanks for asking 😊