Parallelize initial Rust download in bootstrap

[Noratrieb]

Parallelize the initial download of Rust in bootstrap.py

time ./x.py --help after rm -r build
Before: 33s
After: 27s

[rustbot]

r? @albertlarsan68

(rustbot has picked a reviewer for you, use r? to override)

[Noratrieb]

Apparently we want to be python 2 compatible here, I'll fix that tomorrow.

[jyn514]

I think we might be able to drop python2 compat in bootstrap.py proper now that the x.py shim reexecs itself with python3.

[Noratrieb]

I wasn't able to see this on my fast internet connection, but testing this on a slower one shows that the progress bars get pretty broken by the parallel download which is not ideal.

[Noratrieb]

I made it Python 2.7 compatible and tested it manually using python2.7 x.py (patching x.py to not exec Python 3).

[albertlarsan68]

Is the progress bar mess fixed?

[Noratrieb]

Not really, I'm unsure about the best approach to it. I can try whether parallel downloads are faster on a fast internet connection (I wouldn't expect them to be faster on a slow one) and if no, just keep this part serial. If they are am advantage, I'll try find something.

[Noratrieb]

Inserting an exit() before building bootstrap

with a fast (480MiB/s) internet connection
serial download, parallel unpacking: 11s
parallel download, parallel unpacking: 10s
serial download, serial unpacking: 16s

with a less fast (60MiB/s) internet connection
serial download, parallel unpacking: 16s
parallel download, parallel unpacking: 22s (the terminal output from the download progress bars is completely broken)
serial download, serial unpacking: 26s

so the parallel download is actually worse for slow connections, probably because it reduces parallelism for the extraction phase since the downloads don't all complete at the same time

[albertlarsan68]

LGTM, just one question/nit/potential improvement.

[albertlarsan68]

Thanks for the PR!
@bors r+ rollup=iffy (touches core bootstrap code)

[bors]

📌 Commit e829adc819c55c0736bcdce948c53f2920adf5b1 has been approved by albertlarsan68

It is now in the queue for this repository.

[bors]

⌛ Testing commit e829adc819c55c0736bcdce948c53f2920adf5b1 with merge 0a92164116464c739fa27fe5e2b820e598bdd7ca...

[Noratrieb]

@bors r-

[Noratrieb]

I quickly tested it locally again and it worked.
@bors r=albertlarsan68

[bors]

📌 Commit a98968e has been approved by albertlarsan68

It is now in the queue for this repository.

[bors]

⌛ Testing commit a98968e with merge 9ecda8d...

[bors]

☀️ Test successful - checks-actions
Approved by: albertlarsan68
Pushing 9ecda8d to master...

[rust-timer]

Finished benchmarking commit (9ecda8d): comparison URL.

Overall result: no relevant changes - no action needed

@rustbot label: -perf-regression

Instruction count

This benchmark run did not return any relevant results for this metric.

Max RSS (memory usage)

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-1.5%	[-1.5%, -1.5%]	1
All ❌✅ (primary)	-	-	0

Cycles

This benchmark run did not return any relevant results for this metric.

[Zalathar]

This seems to be causing bootstrap to get stuck in a loop and unable to proceed on some systems.

I'm seeing problems on macOS, and #111046 reports similar problems on Windows.

[Zalathar]

It seems that multiprocessing requires the program's entry point to have been guarded by the if __name__ == '__main__': idiom.

(bootstrap.py uses that idiom, but x.py itself currently does not.)

If I edit x.py to wrap all the code in that if statement, bootstrap seems to work fine again.