Advisory: Miscompilation with opt-level="z"

[jamesmunns]

Summary

LLVM17, which was introduced in nightly-2023-08-09, and stable 1.73.0, contains a regression in an
optimization pass called "machine outlining", which reduces code size by merging repeated segments of
machine code, creating an "outlined function" which replaces the repeated code.

In some cases, this optimization pass does not correctly consider side effects of calling the outlined
function, causing the Link Register (LR) to become corrupted, and for the resulting program to act incorrectly,
including unexpected crashes, or jumping to incorrect locations, including RAM data regions.

This regression is subject to certain combinations of factors, and does not always reliably trigger. However,
this regression is possible for all Cortex-M targets, when the "machine outlining" optimization pass is active.
We believe this optimization pass is only active when opt-level = "z" is selected.

It is advised that ALL users of rustc stable >= 1.73.0, or rustc nightly >= 2023-08-09 apply one of the
workarounds specified below. As of 2023-12-21, the regression has not been fixed in rustc.

Technical details

Links to context:

• Description of Machine Outlining
• rustc issue
• upstream LLVM fix, not yet applied to rustc

Potentially Affected Configurations

• Any of the following versions of rustc:
  • stable versions >= 1.73, including the soon-to-be-released 1.75
  • Some betas? Not sure, probably everything >= 1.73, to be confirmed
  • ALL nightlies >= nightly-2023-08-09
• AND any of the following Cortex-M targets:
  • thumbv6m-none-eabi
  • thumbv7m-none-eabi
  • thumbv7em-none-eabi
  • thumbv7em-none-eabihf
  • thumbv8m.base-none-eabi
  • thumbv8m.main-none-eabi
  • thumbv8m.main-none-eabihf
• AND any of the following:
  • opt-level = "z" in any of your profiles
  • Manually enabling the machine outliner, e.g setting RUSTFLAGS=-Cllvm-args=--enable-machine-outliner=...
    to any value other than "never"

Workarounds

There are currently three potential workarounds, any of which are sufficient to avoid this regression:

• Downgrade to a version of rustc not affected. See the affected configurations above for affected versions.
• Use an optimization level other than "z", such as 2, 3, or "s".
• Add the following RUSTFLAG to your configuration: -Cllvm-args=--enable-machine-outliner=never.
  • Please refer to the cargo docs for how to set RUSTFLAGS persistently

Determining if you may be affected

Determining whether you are affected requires looking at your rustc version, and your optimization settings.

Compiler Version

You can determine your currently active version of rust by running the rustup show command.

$ rustup show

... output omitted ...

active toolchain
----------------

stable-aarch64-apple-darwin (default)
rustc 1.73.0 (cc66ad468 2023-10-03)

You may also see the version overridden for some reason, such as a rust-toolchain file:

$ rustup show

... output omitted ...

active toolchain
----------------

nightly-2023-11-01-aarch64-apple-darwin (overridden by '/Users/.../rust-toolchain.toml')
rustc 1.75.0-nightly (9d83ac217 2023-10-31)

If you see a version matching:

• rustc 1.xx.y, where the 1.xx.y version is >= 1.73.0
• rustc 1.xx.y-beta, where the 1.xx.y version is >= 1.73.0
• rustc 1.xx.y-nightly (SHA1 yyyy-mm-dd), where the yyyy-mm-dd date is >= 2023-08-09

Your compiler version is affected. Note that as overrides may be set on a folder/project basis, you may
need to check all potentially affected projects.

Optimization Settings

Optimization settings are set in the Cargo.toml of your binary project.

By default, the debug/dev profile, used by cargo build, uses opt-level=0, which is not affected.

By default, the release/optimized profile, used by cargo build --release, uses opt-level=3,
which is not affected.

If your profiles contain a section containing opt-level = "z":

[profile.NAME]
debug = 2
codegen-units = 1
incremental = false
opt-level = "z" # <---
lto = "fat"

Then you will be affected in relevant compiler versions. note that profile.NAME can be any profile,
including profile.dev, profile.release, and others.

Thanks

Thanks to @peter9477 for finding the bug and providing a reproduction case.

[jamesmunns]

rust-lang/rust#119802 is likely to fix this, as it updates LLVM to include the expected fix.

[Sympatron]

Adding this to your global ~/.cargo/config.toml is probably the safest option for now:

[target.'cfg(all(target_arch = "arm", target_os = "none"))']
rustflags = ["-C", "llvm-args=--enable-machine-outliner=never"]

[jamesmunns]

Potentially, but do look at the linked docs for rustflag setting methods. They don't always compose, and your suggestion would override build settings for example, or would be overridden by env vars.

I avoided making a specific recommendation on purpose, but perhaps should have drawn more attention to that point.

[Sympatron]

I my project I also have a [target.'cfg(all(target_arch = "arm", target_os = "none"))'] section in ./.cargo/config.toml with additional rustflags. I built with -v to check which flags are passed to rustc and all flags from both config files were passed, so I assumed they are merged by cargo. But I have to admit that I am no expert on thoses things...

[jamesmunns]

There are four mutually exclusive sources of extra flags. They are checked in order, with the first one being used:

1. CARGO_ENCODED_RUSTFLAGS environment variable.
2. RUSTFLAGS environment variable.
3. All matching target..rustflags and target..rustflags config entries joined together.
4. build.rustflags config value.

From https://doc.rust-lang.org/cargo/reference/config.html#buildrustflags, Linked in the advisory.

[Sympatron]

Ok, then opt-level = "s" is probably the better option if you use RUSTFLAGS or CARGO_ENCODED_RUSTFLAGS .

[tomkris]

What is the right place to monitor status of this issue and fix release? All linked issues are "closed/merged". Does it mean that next release 1.76 will have fix included?

[jamesmunns]

Yes, it appears so: rust-lang/rust#119861

The LLVM fix was backported to the current beta, so the current beta should have the fix, and 1.76 should have it as well. I haven't gotten confirmation from @peter9477 that his code no longer crashes, and I haven't had time to test the repro either.

[peter9477]

I'm currently pinned to channel = "nightly-2023-12-28" (in rust-toolchain.toml) and had added llvm-args=--enable-machine-outliner=never to my rustflags to avoid the issue.

I'm not sure what I would have to change to verify the "current beta" (not really sure what that is), and I believe I have dependencies that can't build without nightly so I assume I can't just directly test 1.76.

Happy to do a confirmation test if someone can guide me on what I can do to accomplish that.

[jamesmunns]

@peter9477 you should be able to pick a nightly after rust-lang/rust#119802 landed (so Jan 13th? maybe the 14th or 15th, or just whatever the most recent one is), and remove the llvm args.

If the current nightly no longer reproduces the bug: we're probably fine.

If you DO want to test the beta, you can use the newest beta, you'll also have to set the RUSTC_BOOTSTRAP=1 env var to force the compiler to allow use of nightly features on a beta compiler.

If THAT no longer reproduced the bug: we're fairly certain the fix was backported to 1.76 correctly and will be fixed in the next stable.

[jonathanpallant]

Unless it gets back ported to beta, it won't be in stable until 1.77, in eight weeks time.

[jamesmunns]

It was backported to beta.

[jamesmunns]

Citation: rust-lang/rust#119861