nontemporal_store: make sure that the intrinsic is truly just a hint #128149

RalfJung · 2024-07-24T18:59:23Z

The !nontemporal flag for stores in LLVM sounds like it is just a hint, but actually, it is not -- at least on x86, non-temporal stores need very special treatment by the programmer or else the Rust memory model breaks down. LLVM still treats these stores as-if they were normal stores for optimizations, which is highly dubious. Let's avoid all that dubiousness by making our own non-temporal stores be truly just a hint, which is possible on some targets (e.g. ARM). On all other targets, non-temporal stores become regular stores.

~~Blocked on rust-lang/stdarch#1541 propagating to the rustc repo, to make sure the _mm_stream intrinsics are unaffected by this change.~~

Fixes #114582
Cc @Amanieu @workingjubilee

rustbot · 2024-07-24T18:59:31Z

r? @fmease

rustbot has assigned @fmease.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

rustbot · 2024-07-24T18:59:33Z

Some changes occurred in compiler/rustc_codegen_cranelift

cc @bjorn3

Some changes occurred in compiler/rustc_codegen_gcc

cc @antoyo, @GuillaumeGomez

compiler/rustc_codegen_gcc/src/builder.rs

compiler/rustc_codegen_cranelift/src/intrinsics/mod.rs

Kobzol · 2024-07-24T21:22:38Z

It's a bit sad if this becomes an actual no-op on x86, nontemporal stores can be (very rarely) very useful. Is this usable through anything else than mm_stream_* and friends? These functions already contain a safety note that a fence has to be used after them.

workingjubilee · 2024-07-24T21:26:26Z

It's a bit sad if this becomes an actual no-op on x86, nontemporal stores can be (very rarely) very useful. Is this usable through anything else than mm_stream_* and friends? These functions already contain a safety note that a fence has to be used after them.

Those intrinsic functions are, in the latest commits to std::arch, internally assembly. They should behave identically to how they did before, please file a bug if they emit notably worse codegen. The distinction is that, in general, LLVM does not try to perform store deduplication into assembly code.

Kobzol · 2024-07-25T05:56:28Z

So mm_stream will still emit nontemporal instructions on x86? Thanks for clarifying!

RalfJung · 2024-07-25T06:18:15Z

Yes it will, that's why rust-lang/stdarch#1541 needs to propagate first.

Kobzol · 2024-07-25T06:20:32Z

Ok, great, then I misunderstood this change, sorry!

workingjubilee · 2024-07-25T06:51:10Z

Thank you, @RalfJung.

compiler/rustc_codegen_llvm/src/builder.rs

RalfJung · 2024-08-05T08:58:12Z

rust-lang/stdarch#1541 has propagated, so this should be good to go. We haven't heard from @fmease though so let's re-roll the reviewer dice.

r? compiler

jieyouxu

Thanks! The changes themselves look good based on the prior discussions. Just a question about the codegen test with regards to target coverage.

jieyouxu · 2024-08-11T09:42:02Z

compiler/rustc_codegen_llvm/src/builder.rs

+                const WELL_BEHAVED_NONTEMPORAL_ARCHS: &[&str] =
+                    &["aarch64", "arm", "riscv32", "riscv64"];


Question [REV1 (2/2)]: If I understand it correctly, the codegen test is modified such that it only codegens for aarch64-unknown-linux-gnu target, but here we have 4 "well-behaved" architectures that we do emit !nontemporal for.

tests/codegen/intrinsics/nontemporal.rs

jieyouxu · 2024-08-11T09:50:46Z

r? jieyouxu

tests/codegen/intrinsics/nontemporal.rs

jieyouxu

Thanks, feel free to r=me,Amanieu,Jubilee after maybe fixing the nit :3

tests/codegen/intrinsics/nontemporal.rs

RalfJung · 2024-08-12T09:36:14Z

@bors r=jieyouxu,Amanieu,Jubilee

bors · 2024-08-12T09:36:16Z

📌 Commit 5be6c7d has been approved by jieyouxu,Amanieu,Jubilee

It is now in the queue for this repository.

tests/codegen/intrinsics/nontemporal.rs

RalfJung · 2024-08-12T09:43:08Z

@bors r=jieyouxu,Amanieu,Jubilee

bors · 2024-08-12T09:43:11Z

📌 Commit 75743dc has been approved by jieyouxu,Amanieu,Jubilee

It is now in the queue for this repository.

…llaumeGomez Rollup of 10 pull requests Successful merges: - rust-lang#128149 (nontemporal_store: make sure that the intrinsic is truly just a hint) - rust-lang#128394 (Unify run button display with "copy code" button and with mdbook buttons) - rust-lang#128537 (const vector passed through to codegen) - rust-lang#128632 (std: do not overwrite style in `get_backtrace_style`) - rust-lang#128878 (Slightly refactor `Flags` in bootstrap) - rust-lang#128886 (Get rid of some `#[allow(rustc::untranslatable_diagnostic)]`) - rust-lang#128929 (Fix codegen-units tests that were disabled 8 years ago) - rust-lang#128937 (Fix warnings in rmake tests on `x86_64-unknown-linux-gnu`) - rust-lang#128978 (Use `assert_matches` around the compiler more) - rust-lang#128994 (Fix bug in `Parser::look_ahead`.) r? `@ghost` `@rustbot` modify labels: rollup

Rollup merge of rust-lang#128149 - RalfJung:nontemporal_store, r=jieyouxu,Amanieu,Jubilee nontemporal_store: make sure that the intrinsic is truly just a hint The `!nontemporal` flag for stores in LLVM *sounds* like it is just a hint, but actually, it is not -- at least on x86, non-temporal stores need very special treatment by the programmer or else the Rust memory model breaks down. LLVM still treats these stores as-if they were normal stores for optimizations, which is [highly dubious](llvm/llvm-project#64521). Let's avoid all that dubiousness by making our own non-temporal stores be truly just a hint, which is possible on some targets (e.g. ARM). On all other targets, non-temporal stores become regular stores. ~~Blocked on rust-lang/stdarch#1541 propagating to the rustc repo, to make sure the `_mm_stream` intrinsics are unaffected by this change.~~ Fixes rust-lang#114582 Cc `@Amanieu` `@workingjubilee`

…ouxu,Amanieu,Jubilee nontemporal_store: make sure that the intrinsic is truly just a hint The `!nontemporal` flag for stores in LLVM *sounds* like it is just a hint, but actually, it is not -- at least on x86, non-temporal stores need very special treatment by the programmer or else the Rust memory model breaks down. LLVM still treats these stores as-if they were normal stores for optimizations, which is [highly dubious](llvm/llvm-project#64521). Let's avoid all that dubiousness by making our own non-temporal stores be truly just a hint, which is possible on some targets (e.g. ARM). On all other targets, non-temporal stores become regular stores. ~~Blocked on rust-lang/stdarch#1541 propagating to the rustc repo, to make sure the `_mm_stream` intrinsics are unaffected by this change.~~ Fixes rust-lang#114582 Cc `@Amanieu` `@workingjubilee`

rustbot assigned fmease Jul 24, 2024

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. T-libs Relevant to the library team, which will review and decide on the PR/issue. labels Jul 24, 2024

This comment has been minimized.

Sign in to view

RalfJung force-pushed the nontemporal_store branch from 05e50ef to 8d0d26e Compare July 24, 2024 19:40

workingjubilee reviewed Jul 24, 2024

View reviewed changes

compiler/rustc_codegen_gcc/src/builder.rs Outdated Show resolved Hide resolved

compiler/rustc_codegen_cranelift/src/intrinsics/mod.rs Outdated Show resolved Hide resolved

RalfJung force-pushed the nontemporal_store branch 2 times, most recently from e2efa65 to d1e92af Compare July 24, 2024 20:03

Amanieu reviewed Jul 25, 2024

View reviewed changes

compiler/rustc_codegen_llvm/src/builder.rs Outdated Show resolved Hide resolved

workingjubilee mentioned this pull request Aug 3, 2024

feat(rust): Add NontemporalArithmeticChunked pola-rs/polars#17708

Closed

RalfJung added 2 commits August 5, 2024 10:57

nontemporal_store: make sure that the intrinsic is truly just a hint

28e0907

RISC-V also has sane nontemporal stores

697787a

RalfJung force-pushed the nontemporal_store branch from 0135bb1 to 697787a Compare August 5, 2024 08:57

rustbot assigned TaKO8Ki and unassigned fmease Aug 5, 2024

jieyouxu approved these changes Aug 11, 2024

View reviewed changes

rustbot assigned jieyouxu and unassigned TaKO8Ki Aug 11, 2024

jieyouxu reviewed Aug 11, 2024

View reviewed changes

tests/codegen/intrinsics/nontemporal.rs Show resolved Hide resolved

jieyouxu approved these changes Aug 12, 2024

View reviewed changes

tests/codegen/intrinsics/nontemporal.rs Outdated Show resolved Hide resolved

RalfJung force-pushed the nontemporal_store branch from b1eca7b to 5be6c7d Compare August 12, 2024 09:35

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Aug 12, 2024

jieyouxu reviewed Aug 12, 2024

View reviewed changes

tests/codegen/intrinsics/nontemporal.rs Outdated Show resolved Hide resolved

make the codegen test also cover an ill-behaved arch, and add links

75743dc

RalfJung force-pushed the nontemporal_store branch from 5be6c7d to 75743dc Compare August 12, 2024 09:42

GuillaumeGomez mentioned this pull request Aug 12, 2024

Rollup of 10 pull requests #129008

Merged

bors merged commit 095ca33 into rust-lang:master Aug 12, 2024
6 checks passed

rustbot added this to the 1.82.0 milestone Aug 12, 2024

RalfJung deleted the nontemporal_store branch August 14, 2024 12:15

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

nontemporal_store: make sure that the intrinsic is truly just a hint #128149

nontemporal_store: make sure that the intrinsic is truly just a hint #128149

RalfJung commented Jul 24, 2024 •

edited

Loading

rustbot commented Jul 24, 2024

rustbot commented Jul 24, 2024

This comment has been minimized.

Kobzol commented Jul 24, 2024

workingjubilee commented Jul 24, 2024

Kobzol commented Jul 25, 2024

RalfJung commented Jul 25, 2024

Kobzol commented Jul 25, 2024

workingjubilee commented Jul 25, 2024

RalfJung commented Aug 5, 2024

jieyouxu left a comment •

edited

Loading

jieyouxu Aug 11, 2024

jieyouxu commented Aug 11, 2024

jieyouxu left a comment

RalfJung commented Aug 12, 2024

bors commented Aug 12, 2024

RalfJung commented Aug 12, 2024

bors commented Aug 12, 2024

		const WELL_BEHAVED_NONTEMPORAL_ARCHS: &[&str] =
		&["aarch64", "arm", "riscv32", "riscv64"];

nontemporal_store: make sure that the intrinsic is truly just a hint #128149

nontemporal_store: make sure that the intrinsic is truly just a hint #128149

Conversation

RalfJung commented Jul 24, 2024 • edited Loading

rustbot commented Jul 24, 2024

rustbot commented Jul 24, 2024

This comment has been minimized.

Kobzol commented Jul 24, 2024

workingjubilee commented Jul 24, 2024

Kobzol commented Jul 25, 2024

RalfJung commented Jul 25, 2024

Kobzol commented Jul 25, 2024

workingjubilee commented Jul 25, 2024

RalfJung commented Aug 5, 2024

jieyouxu left a comment • edited Loading

Choose a reason for hiding this comment

jieyouxu Aug 11, 2024

Choose a reason for hiding this comment

jieyouxu commented Aug 11, 2024

jieyouxu left a comment

Choose a reason for hiding this comment

RalfJung commented Aug 12, 2024

bors commented Aug 12, 2024

RalfJung commented Aug 12, 2024

bors commented Aug 12, 2024

RalfJung commented Jul 24, 2024 •

edited

Loading

jieyouxu left a comment •

edited

Loading