Proxy: Cannot download files #2523

pJunger · 2020-10-16T10:20:43Z

Problem
Any command like update, check. etc. fails with operation timed out if a proxy is set with a leading http://.

Steps

Set proxy with export http_proxy=http://gateway.something.net:10402 (& export https_proxy=http://gateway.something.net:10402)
Run rustup update
Receive

error: could not download file from 'https://static.rust-lang.org/rustup/release-stable.toml' to '/tmp/rustup-updateMGVR4p/release-stable.toml'
error: caused by: failed to make network request
error: caused by: error sending request for url (https://static.rust-lang.org/rustup/release-stable.toml): operation timed out
error: caused by: operation timed out

Possible Solution(s)
Workaround is possible by removing the http://

export http_proxy=gateway.something.net:10402
export https_proxy=gateway.something.net:10402
rustup update

Notes
OS:

Ubuntu 20.04.1 (in WSL2)

Output of rustup --version:

rustup 1.22.1 (b01adbb 2020-07-08)

Output of rustup show:

Default host: x86_64-unknown-linux-gnu
rustup home: /home/me/.rustup

stable-x86_64-unknown-linux-gnu (default)
rustc 1.47.0 (18bf6b4f0 2020-10-07)

Edit: The * was meant only for obfuscation, replaced it with "something".

The text was updated successfully, but these errors were encountered:

kinnison · 2020-10-16T10:59:50Z

I would guess that this would likely be a problem in env_proxy if it's anywhere. @inejge what do you think it might be?

dzhang-b · 2020-10-16T17:36:16Z

HI, I have the similar problem that using rustup in a environment required proxy.

Here is the debug output:

[0] # rustup --version
rustup 1.22.1 (b01adbbc3 2020-07-08)

Verbose Output

[0] # rustup --verbose update
verbose: read metadata version: '12'
info: no updatable toolchains installed
info: checking for self-updates
error: could not download file from 'https://static.rust-lang.org/rustup/release-stable.toml' to '/tmp/rustup-updateag7RVu/release-stable.toml'
error: caused by: failed to make network request
error: caused by: error sending request for url (https://static.rust-lang.org/rustup/release-stable.toml): error trying to connect: unexpected EOF
error: caused by: error trying to connect: unexpected EOF
error: caused by: unexpected EOF

My current environment has these variables set

http_proxy=http://proxy...
ftp_proxy=ftp://proxy-dmz...
ALL_PROXY=socks5://proxy...
socks_proxy=proxy-dmz....
https_proxy=https://proxy...

Using the RUSTUP_USE_CURL=1 given the following output:

[0] # RUSTUP_USE_CURL=1 rustup --verbose update
verbose: read metadata version: '12'
info: no updatable toolchains installed
info: checking for self-updates
error: could not download file from 'https://static.rust-lang.org/rustup/release-stable.toml' to '/tmp/rustup-updatedc1VH6/release-stable.toml'
error: caused by: error during download
error: caused by: [35] SSL connect error

The related issues so far:

inejge · 2020-10-16T17:46:29Z

I can't reproduce the reported behavior when running the following program:

// env_proxy = "0.4"
// url = "2"

use env_proxy::for_url;
use url::Url;

fn main() {
    let manifest = "https://static.rust-lang.org/rustup/release-stable.toml";
    let proxy = for_url(&Url::parse(manifest).unwrap());
    dbg!(proxy.to_url());
}

The program mimics rustup's usage of env_proxy. Both

https_proxy=http://gateway.*.net:10402 ./proxy-test

and

https_proxy=gateway.*.net:10402 ./proxy-test

produce the same output:

[src/main.rs:10] proxy.to_url() = Some(
    "http://gateway.*.net:10402/",
)

I may try with rustup itself later, but I don't have access to WSL2, so I can't recreate the exact environment. As an aside, gateway.*.net is one very funky hostname, but it shouldn't matter if it's properly defined in /etc/hosts or the local resolver.

X-21 · 2020-12-23T07:51:39Z

I would guess that this would likely be a problem in env_proxy if it's anywhere. @inejge what do you think it might be?

I have a issue when install rust and I guess it's like this problem about proxy.

$ curl --proto '=https' --tlsv1.2 https://sh.rustup.rs -sSf | sh

error: could not download file from 'https://static.rust-lang.org/dist/channel-rust-stable.toml.sha256' to '/home/edc/.rustup/tmp/e6kk8puoa1s49au8_file'
error: caused by: failed to make network request
error: caused by: error sending request for url (https://static.rust-lang.org/dist/channel-rust-stable.toml.sha256): operation timed out
error: caused by: operation timed out

$ echo $http_proxy
http://192.168.10.1:10899/

# edc @ ubuntu in ~ [15:45:06] 
$ echo $https_proxy
https://192.168.10.1:10899/

And I turn off the Ubuntu's system proxy, restart and the installation script would have successed.

#1328

kinnison · 2020-12-23T08:45:07Z

@X-21
If you're in a position to reproduce the issue, I'd appreciate it if you could. If you copy your rustup binary as rustup-init then it will save you constantly redownloading the installer during tests. For example if you do:

cp $(which rustup) /tmp/rustup-init
strace -f -s1024 -o /tmp/trace.log /tmp/rustup-init

And then once it has failed as above, you can attach trace.log to a comment (do not paste it in, attach it please, it'll be quite big) and we can look to see if we can see what's going on.

X-21 · 2020-12-23T08:55:00Z

@X-21
If you're in a position to reproduce the issue, I'd appreciate it if you could. If you copy your rustup binary as rustup-init then it will save you constantly redownloading the installer during tests. For example if you do:
cp $(which rustup) /tmp/rustup-init
strace -f -s1024 -o /tmp/trace.log /tmp/rustup-init
And then once it has failed as above, you can attach trace.log to a comment (do not paste it in, attach it please, it'll be quite big) and we can look to see if we can see what's going on.

Sure, I am pleasure to help that.

cp $(which rustup) /tmp/rustup-init
strace -f -s1024 -o /tmp/trace.log /tmp/rustup-init

Then it was failed as expected and generated the trace.log which I attached.
trace.log

kinnison · 2020-12-23T09:05:27Z

Okay, so from the looks of that trace, it connected to your proxy and sent the clienthello etc. Your proxy then did not respond. Are you sure the https proxy is actually working? Could you also do the following for me to see how curl deals with things?

strace -f -s1024 -o /tmp/curltrace.log curl https://sh.rustup.rs > /dev/null

Then attach curltrace.log please?

X-21 · 2020-12-23T09:13:32Z

Okay, so from the looks of that trace, it connected to your proxy and sent the clienthello etc. Your proxy then did not respond. Are you sure the https proxy is actually working? Could you also do the following for me to see how curl deals with things?
strace -f -s1024 -o /tmp/curltrace.log curl https://sh.rustup.rs > /dev/null
Then attach curltrace.log please?

Yes I am sure the proxy is working. I can't visit some entertainment website without the proxy.

And the curltrace.log:
curltrace.log

kinnison · 2020-12-23T09:32:03Z

Okay, curl looks like it used HTTP not HTTPs to talk to the proxy on your system. Fascinating. If I set similar environment variables and run curl, it uses HTTPs to connect, not HTTP, same as rustup does.

On the other hand, wget interestingly uses HTTP by default when it tries to use a proxy.

Your curl is 7.47.0 whereas mine is 7.64.0. I wonder if this is simply the case that older versions of curl, and wget, always use http to connect to https proxies, whereas newer ones (and rustup) use https, and whatever proxying software is on that server can't cope with the choice between the two? Sadly your proxy doesn't announce itself in the 'connection established' response which makes it harder to guess.

Do you know what proxy software is in use?

X-21 · 2020-12-23T09:56:34Z

I use proxy by ubuntu's Network proxy like that:

so all request(http/https/socks5/ftp) will be transmitted to another machine in local network, and this machine will handle these request to a proxy software named shadowsocks what is a proxy work in Transport Layer.

I try curl https://www.google.com and capture it:

X-21 · 2020-12-23T10:11:03Z

I use proxy by ubuntu's Network proxy like that:

so all request(http/https/socks5/ftp) will be transmitted to another machine in local network, and this machine will handle these request to a proxy software named shadowsocks what is a proxy work in Transport Layer.

I try curl https://www.google.com and capture it:

and I capture it when download file from 'https://static.rust-lang.org/dist/channel-rust-stable.toml.sha256' :

I guess the problem is casued by the Server Name of request for download file from static.rust-lang.org, which is doesn't indicate the destination domain.

curl google:

download form rust-lang.org:

kinnison · 2020-12-23T11:06:54Z

The SNI is not relevant here, it's that we're doing SSL to the proxy, which the proxy is not responding to properly.

Is the machine 192.168.10.1 running shadowsocks itself? If so, which version of the shadowsocks server?

X-21 · 2020-12-24T01:12:26Z

The SNI is not relevant here, it's that we're doing SSL to the proxy, which the proxy is not responding to properly.

Is the machine 192.168.10.1 running shadowsocks itself? If so, which version of the shadowsocks server?

Yes I know it was doing SSL to proxy, but I think it should do SSL to static.rust-lang.org via shadowsocks.
Yes the machine in 192.168.10.1 is running shadowsocks and it's version is lastest - https://github.com/shadowsocks/shadowsocks-windows/releases/tag/4.3.3.0

kinnison · 2020-12-24T08:34:45Z

I agree that it should do SSL to static.rust-lang.org. It would achieve that by establishing the link to the proxy and then asking it to connect through to port 443 on static.rust-lang.org and then it'd establish SSL over that. Having read through shadowsocks' code, it uses privoxy as its underlying http proxy. Having been through privoxy's source code it has no SSL capability at all. Given that, your https_proxy environment variable should be set to http://server:port/ not to https://server:port/ Given that shadowsocks doesn't support anything but HTTP and SOCKS proxying, you also shouldn't be setting it in the ftp proxy slot in your config most likely.

I'd suggest not setting it in the https proxy slot, clients should fall back to http_proxy from there IIRC.

pJunger · 2020-12-24T09:00:12Z

I also added a strace log from running rustup self update for my configuration: rustup-trace-update.log

X-21 · 2020-12-24T09:26:26Z

I agree that it should do SSL to static.rust-lang.org. It would achieve that by establishing the link to the proxy and then asking it to connect through to port 443 on static.rust-lang.org and then it'd establish SSL over that. Having read through shadowsocks' code, it uses privoxy as its underlying http proxy. Having been through privoxy's source code it has no SSL capability at all. Given that, your https_proxy environment variable should be set to http://server:port/ not to https://server:port/ Given that shadowsocks doesn't support anything but HTTP and SOCKS proxying, you also shouldn't be setting it in the ftp proxy slot in your config most likely.

I'd suggest not setting it in the https proxy slot, clients should fall back to http_proxy from there IIRC.

Thanks, it is a socks5 proxy as you said. So I change the https_proxy to 'http://192.168.10.1:10899' make rustup and other commands working. I also have the confusion why rustup can't work but others can.
I only set socks slot and http slot initially, but some commands like
curl https://www.google.com/ -o 1.html
would not be working and
curl http://www.google.com/ -o 1.html
is ok.
So I set https slot to 'https://192.168.10.1:10899' and those would be fine.
Actully I should have set https_proxy to 'http://192.168.10.1:10899' not to ‘https://ip:port’ , both two setting were working therefore I did not care these setting anymore.

Piping · 2020-12-24T19:02:34Z

I can confirm the behavior and right fix for rustup 1.23.1.

The fix is to change https_proxy environment variable to use http://proxy_version_url.

Then rustup is able talk to static.rust-lang.org.

Is it a bug that rustup does not use https://proxy_url correctly?

kinnison · 2020-12-28T11:13:25Z

No, this is a bug that your proxy doesn't support https:// proxy urls properly, and you've not encountered it otherwise because your OS is running old enough versions of curl that it doesn't support them properly either. Is there a chance you can unset the https proxy variable? Most things will then fall back to the all proxy anyway, which should be set to an http:// url which will then work.

monk279 · 2022-08-07T08:54:31Z

@pJunger what does "something" exactly mean?

pJunger added the bug label Oct 16, 2020

kinnison closed this as completed Jun 8, 2021

loynoir mentioned this issue Sep 22, 2023

increase rustup fetch default retries and default timeout #3486

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Proxy: Cannot download files #2523

Proxy: Cannot download files #2523

pJunger commented Oct 16, 2020 •

edited

Loading

kinnison commented Oct 16, 2020

dzhang-b commented Oct 16, 2020 •

edited

Loading

inejge commented Oct 16, 2020

X-21 commented Dec 23, 2020 •

edited

Loading

kinnison commented Dec 23, 2020

X-21 commented Dec 23, 2020

kinnison commented Dec 23, 2020

X-21 commented Dec 23, 2020

kinnison commented Dec 23, 2020

X-21 commented Dec 23, 2020

X-21 commented Dec 23, 2020

kinnison commented Dec 23, 2020

X-21 commented Dec 24, 2020

kinnison commented Dec 24, 2020

pJunger commented Dec 24, 2020

X-21 commented Dec 24, 2020

Piping commented Dec 24, 2020

kinnison commented Dec 28, 2020

monk279 commented Aug 7, 2022

Proxy: Cannot download files #2523

Proxy: Cannot download files #2523

Comments

pJunger commented Oct 16, 2020 • edited Loading

kinnison commented Oct 16, 2020

dzhang-b commented Oct 16, 2020 • edited Loading

inejge commented Oct 16, 2020

X-21 commented Dec 23, 2020 • edited Loading

kinnison commented Dec 23, 2020

X-21 commented Dec 23, 2020

kinnison commented Dec 23, 2020

X-21 commented Dec 23, 2020

kinnison commented Dec 23, 2020

X-21 commented Dec 23, 2020

X-21 commented Dec 23, 2020

kinnison commented Dec 23, 2020

X-21 commented Dec 24, 2020

kinnison commented Dec 24, 2020

pJunger commented Dec 24, 2020

X-21 commented Dec 24, 2020

Piping commented Dec 24, 2020

kinnison commented Dec 28, 2020

monk279 commented Aug 7, 2022

pJunger commented Oct 16, 2020 •

edited

Loading

dzhang-b commented Oct 16, 2020 •

edited

Loading

X-21 commented Dec 23, 2020 •

edited

Loading