Refactor dist tarballs generation

[pietroalbini]

Before this PR, each tarball we ship as part of a release was generated by manually creating the directory structure and invoking rust-installer generate. This means each tarball was slightly different, adding new ones meant copy-pasting the code generating another tarball and removing the useless parts, and more importantly refactoring how tarballs are generated is extremely time-consuming.

This PR introduces a new abstraction in rustbuild, Tarball. The Tarball struct provides a trivial API to generate simple tarballs, and can get out of the way when more complex tarballs have to generate. For example, the whole code to generate the build-manifest tarball is now the following:

let tarball = Tarball::new(builder, "build-manifest", &self.target.triple);
tarball.add_file(&build_manifest, "bin", 0o755);
tarball.generate()

One notable change between the old tarballs and the new ones is that the "overlay" (README.md, COPYRIGHT, LICENSE-APACHE and LICENSE-MIT) is now available in every produced tarball, while before each tarball inconsistently had or didn't have those files. Tarballs that need a different overlay have a way to change which files to include (with the set_overlay method):

let mut tarball = Tarball::new(builder, "rustfmt", &target.triple);
tarball.set_overlay(OverlayKind::Rustfmt);
tarball.add_file(rustfmt, "bin", 0o755);
tarball.add_file(cargofmt, "bin", 0o755);
tarball.add_legal_and_readme_to("share/doc/rustfmt");
Some(tarball.generate())

The PR should be reviewed commit-by-commit, as each commit migrated a separate tarball to use Tarball. During development i made sure every tarball can still be generated, and for the most compex tarballs I manually ensured the list of files between the old and new tarballs did not have unexpected changes.

r? @Mark-Simulacrum

[Mark-Simulacrum]

I'm probably not going to get a chance to review this until the weekend at the earliest -- is this blocking some work? If so I could potentially fit it in but it'd be a bit rough.

[pietroalbini]

It's not blocking any work, the next task for me on this project is to change rust-installer, and that can happen in parallel of this PR. Also I'm probably as busy as you :)

[Mark-Simulacrum]

This looks amazing to me -- I noted one typo, and I'm sure I might've missed others, but I don't think that should block (we can patch things up if we notice them later). It seems good to get this landed quickly so we're not landing it around the release (so we can fix things on master without worrying about backports).

r=me with typo fixed -- up to you on @Keruspe's comment.

[pietroalbini]

@bors r=Mark-Simulacrum

[bors]

📌 Commit 272838e3f60487a40e25363e6f681dfd011234dc has been approved by Mark-Simulacrum

[bors]

⌛ Testing commit 272838e3f60487a40e25363e6f681dfd011234dc with merge de7ce5c3f8869f3288d1fb9480fdcd985362f15a...

[bors]

💔 Test failed - checks-actions

[pietroalbini]

@bors try

[bors]

⌛ Trying commit 272838e3f60487a40e25363e6f681dfd011234dc with merge 80acf4fa64fccb397fb4cec1d663dd473f492507...

[Mark-Simulacrum]

FWIW I think git absorb or similar should make it easy to avoid the "fix review comments" commit, but since this would still have 21 commits - which seems right - I am not going to block on that.

[bors]

💔 Test failed - checks-actions

[pietroalbini]

Ok should've fixed the CI failure (I also amended the commit history).

@bors try

[bors]

⌛ Trying commit 6a66aa4a263a3cd7586b723ec589c2f739e233ef with merge bf52e3fae8d78a18529a0c726ce78238217c20e8...

[bors]

☀️ Try build successful - checks-actions
Build commit: bf52e3fae8d78a18529a0c726ce78238217c20e8 (bf52e3fae8d78a18529a0c726ce78238217c20e8)

[Mark-Simulacrum]

@bors r+

[bors]

📌 Commit 8736731 has been approved by Mark-Simulacrum

[bors]

⌛ Testing commit 8736731 with merge 7f9c43c...

[bors]

☀️ Test successful - checks-actions
Approved by: Mark-Simulacrum
Pushing 7f9c43c to master...

[semarie]

if I didn't mess myself, with this PR, the current nightly tarball extracts inside image/ directory instead of previously rustc-nightly-src/ directory.

does such name will stick on release too (instead of rustc-1.50-src/) ?

[Mark-Simulacrum]

Yeah, at least rustc-nightly-src changed. Looking into a fix now.

[Mark-Simulacrum]

Fix should be up in #80397, and seems to work locally.

[pietroalbini]

Sorry about that!