rustbuild seems to deal badly with poor internet connections

[durka]

I tried to build rust with rustbuild.

./configure --enable-ccache --enable-rustbuild
make

But after a few seconds it fails downloading the nightly with errno 104, which is connection reset by peer. If I run it again it sees the file is there and fails saying the checksum is wrong. I guess it needs to retry the download.

[alexcrichton]

Ah we probably need to download to a temporary location and only when done move it over into where it's expected to be, that way we'll restart any failed download.

[cyplo]

Hi ! Anyone interested in taking this ? If not I'd like to try.

[GuillaumeGomez]

@cyplo: Sure. Take it.

[alexcrichton]

If you have any questions, feel free to ask me @cyplo!

[cyplo]

Hi @alexcrichton, using your invite to ask here ;)

I took a quick look there and it seems that the initial download is done via bootstrap.py and it seems that the fix can be done entirely within that file.

Do you maybe know if there any other downloads, not done during the bootstrap.py phase that would need this treatment or other reasons to touch something beyond bootstrap.py in the fix for this issue ?

Thanks a lot !

[alexcrichton]

Yeah I believe that's the only location where we download anything. There's other downloads that Cargo does for the actual compilation of the build system itself but Cargo handles bad internet and things like that internally already.

[cyplo]

Hi ! I see that #32926 is merged now, I'm thinking on how to integrate the download resume/retry with it. @alexcrichton / @caipre Let me know what do you think of the following:

• try downloading to a temp location
• check checksum there
• if invalid - remove and retry max N times
• if valid - move to destination

[GuillaumeGomez]

Isn't it possible to check checksum before downloading? It seems to be overkill otherwise (downloading something to just get "no, you're up to date finally" is really annoying). Or at least check the date or something equivalent which could prevent to download for nothing?

[cyplo]

Yes, totally, it would probably be like this:

• if there is no file - download
• check checksum
• invalid - retry from scratch
• valid - proceed

thanks !

[caipre]

The logic to check whether the file exists and needs updated is already present:

rust/src/bootstrap/bootstrap.py

Lines 92 to 93 in 6fa61b8

	if self.rustc().startswith(self.bin_root()) and \
	(not os.path.exists(self.rustc()) or self.rustc_out_of_date()):

My PR added the checksumming within the get() method, so you shouldn't have any work on that point. I'm not sure we need a retry mechanism. I prefer to just report the problem and let the user decide what to do next.

For this issue, you need to change the path that the tarball is downloaded to to use some temporary path, then move that to the correct path after the download is complete.

An analogy to shell:

curl -o file.tmp https://... && mv file.tmp file

I suppose you might want to update the checksumming to be done on the temporary file, and only if that succeeds do the move.

[alexcrichton]

Yeah @caipre's thoughts are exactly my own, let's download to a temp location, verify the checksum, and once both are done we move it to the final location.

[cyplo]

Hey ! I need some more time to get to PR, as I'm quite tight on time lately. If it's not critical/needed quickly - I will come up with PR eventually. If it is - feel free to grab it.

[cyplo]

Hi ! @durka: the fix should be merged now, could you see if this helps ? thanks !

[alexcrichton]

Closing as #33288 has landed, but feel free to reopen if troubles still happen!

(thanks again @cyplo!)