1100% increase (!) in x86_64-unknown-linux release compile times

[maxburke]

Hi,

We've noticed a significant performance regression in compiling our top-level crate in release. It seems to have started as of nightly-2022-09-23:

$ touch src/main.rs && cargo build --release
   Compiling ulv2 v0.1.0 (/home/max/src/ul/services/ulv2)
    Finished release [optimized] target(s) in 17m 10s

With nightly-2022-09-22 the builds are significantly faster:

$ touch src/main.rs && cargo build --release
   Compiling ulv2 v0.1.0 (/builds/UrbanLogiq/ul/services/ulv2)
    Finished release [optimized] target(s) in 1m 22s

In trying to identify where this issue began we see that it's happening as far forward as 2022-11-14. Unfortunately other issues with toolchains have hindered us testing any further forward.

Compilation tests were done on an AMD 5950x (Ubuntu 22.04) with the following config.toml options:

[target.x86_64-unknown-linux-gnu]
rustflags = ["-C", "target-feature=+avx2,+fma", "-C", "link-arg=-fuse-ld=lld"]

We've seen a similar ratio in our CI builds (Azure F8s instances building in Ubuntu 20.04), but Apple M1 (aarch64 darwin) builds are unaffected.

[Mark-Simulacrum]

Bisecting to a particular commit with https://github.com/rust-lang/cargo-bisect-rustc would help, and if you can provide a GitHub repository or similar that would also help (so we can reproduce).

[Noratrieb]

You could also profile the build with the -Ztime-passes or -Zself-profile rustc flags to find the cause of the slowness (which I assume will be in some specific place).

[Noratrieb]

Also, simple profiling with something like perf could help (if you could build rustc from source with debug symbols that could be even more useful).

[maxburke]

@Nilstrieb I've created a repository here with the profiling data obtained with -Ztime-passes and -Zself-profile: https://github.com/maxburke/rustc-104842

There are two directories, one for 2022-09-22 which doesn't exhibit the performance issue and 2022-09-23 which does

@Mark-Simulacrum Unfortunately I'm not able to share the repository, but I will try bisecting to the affecting commit.

[maxburke]

@Mark-Simulacrum Bisected the performance regression to change 8ab71ab59fd17a1c51d23b68eced935b92431b70

[Mark-Simulacrum]

Cc #100980 @compiler-errors

Note that this means this regression will hit stable with 1.66, so rather soon, unless we end up with a backportable fix.

[compiler-errors]

This is reverted and backported on beta via #103509 -- if the current beta contains this PR and it's still slow, then I can investigate, but seems to be dealt with.

[Mark-Simulacrum]

Oh, hm, we should have reverted that PR on beta and nightly now. It would be great to see if we can verify it doesn't reproduce on beta perhaps? (Testing a more recent nightly would also be great).

[compiler-errors]

@maxburke do you mind checking on the latest nightly?

edit: oh "Unfortunately other issues with toolchains have hindered us testing any further forward."

Can you check that your beta tool chain is up to date and re-test?

[maxburke]

@compiler-errors I solved the issues we were getting with moving forward and after trying the latest nightly (2022-11-24), performance seems to be back to normal.