Expose algebraic floating point intrinsics

[calder]

Problem

A stable Rust implementation of a simple dot product is 8x slower than C++ on modern x86-64 CPUs. The root cause is an inability to let the compiler reorder floating point operations for better vectorization.

See https://github.com/calder/dot-bench for benchmarks. Measurements below were performed on a i7-10875H.

C++: 10us ✅

With Clang 18.1.3 and -O2 -march=haswell:

C++	Assembly
float dot(float *a, float *b, size_t len) { #pragma clang fp reassociate(on) float sum = 0.0; for (size_t i = 0; i < len; ++i) { sum += a[i] * b[i]; } return sum; }

Nightly Rust: 10us ✅

With rustc 1.86.0-nightly (8239a37) and -C opt-level=3 -C target-feature=+avx2,+fma:

Rust	Assembly
fn dot(a: &[f32], b: &[f32]) -> f32 { let mut sum = 0.0; for i in 0..a.len() { sum = fadd_algebraic(sum, fmul_algebraic(a[i], b[i])); } sum }

Stable Rust: 84us ❌

With rustc 1.84.1 (e71f9a9) and -C opt-level=3 -C target-feature=+avx2,+fma:

Rust	Assembly
fn dot(a: &[f32], b: &[f32]) -> f32 { let mut sum = 0.0; for i in 0..a.len() { sum += a[i] * b[i]; } sum }

Proposed Change

Add core::intrinsics::f*_algebraic wrappers to f16, f32, f64, and f128 gated on a new float_algebraic feature.

Alternatives Considered

#21690 has a lot of good discussion of various options for supporting fast math in Rust, but is still open a decade later because any choice that opts in more than individual operations is ultimately contrary to Rust's design principles.

In the mean time, processors have evolved and we're leaving major performance on the table by not supporting vectorization. We shouldn't make users choose between an unstable compiler and an 8x performance hit.

References

• Imprecise floating point operations (fast-math) #21690
• ACP: Expose algebraic floating point intrinsics libs-team#532
• Tracking Issue for algebraic floating point methods #136469
• https://github.com/calder/dot-bench
• https://www.felixcloutier.com/x86/vfmadd132ps:vfmadd213ps:vfmadd231ps

try-job: x86_64-gnu-nopt
try-job: x86_64-gnu-aux

[rustbot]

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @thomcc (or someone else) some time within the next two weeks.

Please see the contribution instructions for more information. Namely, in order to ensure the minimum review times lag, PR authors and assigned reviewers should ensure that the review label (S-waiting-on-review and S-waiting-on-author) stays updated, invoking these commands when appropriate:

• @rustbot author: the review is finished, PR author should check the comments and take action accordingly
• @rustbot review: the author is ready for a review, this PR will be queued again in the reviewer's queue

[rustbot]

Some changes occurred to the intrinsics. Make sure the CTFE / Miri interpreter
gets adapted for the changes, if necessary.

cc @rust-lang/miri, @rust-lang/wg-const-eval

[calder]

Thanks for the speedy review @saethlin! I had a couple questions:

• Should we add these methods to f16 and f128 as well?
• Are intrinsic wrappers tested explicitly? I see tests for the intrinsics themselves but not for other wrappers.

[RalfJung]

Thanks for the PR! Going through the usual process, the first step would be a t-libs-api ACP to gauge the team's thinking on whether and how this should be exposed. This PR cannot land before a corresponding FCP has been accepted.

[RalfJung]

In terms of tests, usually we have doc tests. Also, given that the semantics are far from obvious (these operations are nondeterministic!), they need to be documented more carefully - probably in some central location, which is then referenced from everywhere.

[calder]

@RalfJung: Thanks for the quick response!

• Created ACP here: ACP: Expose algebraic floating point intrinsics libs-team#532
• Do you have a suggestion for a good central place to document these methods?

[RalfJung]

https://doc.rust-lang.org/nightly/std/primitive.f32.html seems like a good place, that's where we already document everything special around NaNs. So a new section on algebraic operations there would probably be a good fit.

[calder]

Changes

• Added f16 and f128 methods as requested by @tgross35 here.
• Renamed to algebraic_*() to match checked_*() as requested by @tgross35 here.
• Added central documentation with verbage suggested by @RalfJung here.

[tgross35]

I don't see an existing codegen test for the intrinsics so these should probably get one. https://github.com/rust-lang/rust/tree/3f33b30e19b7597a3acbca19e46d9e308865a0fe/tests/codegen/float would be a reasonable home.

For reference, this would just be a file containing functions like this for each of the new methods, in order to verify the flags that we expect are getting set:

// CHECK-LABEL: float @f32_algebraic_add(
#[no_mangle]
pub fn f32_algebraic_add(a: f32, b: f32) -> f32 {
    // CHECK: fadd reassoc nsz arcp contract float %a, %b
    a.algebraic_add(b)
}

[rustbot]

The Miri subtree was changed

cc @rust-lang/miri

[calder]

The addition and multiplication cases both end up as %b, %a rather than %a, %b which surprised me but isn't incorrect. I opted to allow either in case behavior changes in the future.

[tgross35]

This looks pretty reasonable to me but all the public functions should get some examples. I think it may also be good to give a small demo of how this may work at the end of the new "Algebraic operators" section, e.g.:

For example, the below:

```
x = x.algebraic_add(x, a.algebraic_mul(b));
```

May be rewritten as either of the following:

```
x = x + (a * b); // As written
x = (a * b) + x; // Reordered to allow using a single `fma`
```

[calder]

Per function examples

Did you have specific examples in mind? I'm struggling to think of ones that aren't repetitive / low signal-to-noise (i.e. assert this algebraic add is approximately equal to a normal add). Should we instead link to the central documentation and test approximate equality of each of the ops in normal tests like this so they don't clutter the documentation?

Central example

Made a couple small edits for brevity. How's this look?

[bors]

💔 Test failed - checks-actions

[tgross35]

@calder would you be able to update this? The thresholds likely need to be increased

[calder]

@tgross35: Fixed tolerances, sorry for the delay!

[tgross35]

@bors try

It looks like this PR includes an accidental backtrace downgrade, not sure why rustbot didn't flag that but it should be dropped.

[bors]

⌛ Trying commit acc3d85 with merge 55ea7cb...

[bors]

💔 Test failed - checks-actions

[calder]

@bors try

Tweaked tolerances some more. Is there an easy way to run these MIRI tests locally?

[bors]

@calder: 🔑 Insufficient privileges: not in try users

[tgross35]

@bors try

I think you should be able to run ./x miri library

[bors]

⌛ Trying commit 0151a01 with merge dabe80a...

[bors]

💔 Test failed - checks-actions

[tgross35]

@bors delegate+ for @bors try, just don't merge it (I'll take a final look once things pass)

[tgross35]

@bors try

It's a bit better to use --keep-base when you rebase locally to squash, that way GH shows a diff of only relevant things that changed. Unfortunately GH has the limitation that if the push includes an actual rebase onto latest (sometimes unavoidable ofc), then the "compare" link is pretty unhelpful e.g. https://github.com/rust-lang/rust/compare/53944e53dca5f93f1559b3914470c61dda2c350c..dfd4b6cddf5b2a3936492fe5c7cb97198fec3761

[bors]

⌛ Trying commit dfd4b6c with merge 2d8ed6b...

[tgross35]

What is actually going on that causes these tests to fail? a + b should be 579.0, but the result is 579.0011. Afaict none of the algebraic effects can come into play here, so shouldn't the result be exact?

[bors]

☀️ Try build successful - checks-actions
Build commit: 2d8ed6b (2d8ed6b4bf07905ce9e8b131b366708f273717c0)

[leonardo-m]

By they way, in my codebase I use something minimal like this:

#[derive(Copy, Clone, Default)]
#[repr(transparent)]
struct Af64(pub f64);

impl Af64 {
    const ZERO: Af64 = Af64(0.0);
    const ONE: Af64 = Af64(1.0);
    fn from(x: i32) -> Self { Self(f64::from(x)) }
}

impl Add for Af64 {
    type Output = Self;

    #[inline]
    fn add(self, other: Self) -> Self {
        Self(fadd_algebraic(self.0, other.0))
    }
}

impl Sub for Af64 {
    type Output = Self;

    #[inline]
    fn sub(self, other: Self) -> Self {
        Self(fsub_algebraic(self.0, other.0))
    }
}

impl Mul for Af64 {
    type Output = Self;

    #[inline]
    fn mul(self, other: Self) -> Self {
        Self(fmul_algebraic(self.0, other.0))
    }
}

impl Div for Af64 {
    type Output = Self;

    #[inline]
    fn div(self, other: Self) -> Self {
        Self(fdiv_algebraic(self.0, other.0))
    }
}

impl Rem for Af64 {
    type Output = Self;

    #[inline]
    fn rem(self, other: Self) -> Self {
        Self(frem_algebraic(self.0, other.0))
    }
}