Add alignment parameter to `simd_masked_{load,store}` #147355

sayantn · 2025-10-04T23:41:43Z

This PR adds an alignment parameter to the SIMD intrinsics simd_masked_load and simd_masked_store. This parameter is the (byte) alignment of the ptr parameter, so this is kind of a generalization from the previous signature to allow under-aligned and over-aligned pointers.

The main motive for this is stdarch - most vector loads are either fully aligned (to the vector size) or unaligned (byte-aligned), so the previous signature doesn't cut it.

I introduced a const parameter instead of a const-generic parameter because portable-simd uses pointers aligned to the element type, and so needs to pass align_of::<T>() as the alignment, but this isn't possible with const-generic parameters without GCE.

cg_llvm
cg_clif (not planned in this PR)
miri/const_eval (will implement once Implement most SIMD intrinsics in const-eval #146568 is merged, otherwise there will be nasty merge conflicts)

Alternatives

Using a const-generic parameter, with 0 having the special meaning of using the element type's alignment. This will be useful in the common case of using the element type's alignment, and also offer enough flexibility to use in stdarch

cc @workingjubilee @RalfJung @BoxyUwU

rustbot · 2025-10-04T23:41:46Z

The Miri subtree was changed

cc @rust-lang/miri

Portable SIMD is developed in its own repository. If possible, consider making this change to rust-lang/portable-simd instead.

cc @calebzulawski, @programmerjake

Some changes occurred to the platform-builtins intrinsics. Make sure the
LLVM backend as well as portable-simd gets adapted for the changes.

cc @antoyo, @GuillaumeGomez, @bjorn3, @calebzulawski, @programmerjake

Some changes occurred in compiler/rustc_codegen_ssa

cc @WaffleLapkin

Some changes occurred to the intrinsics. Make sure the CTFE / Miri interpreter
gets adapted for the changes, if necessary.

cc @rust-lang/miri, @RalfJung, @oli-obk, @lcnr

rustbot · 2025-10-04T23:41:48Z

r? @lcnr

rustbot has assigned @lcnr.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

programmerjake · 2025-10-05T02:35:21Z

src/tools/miri/tests/pass/intrinsics/portable-simd.rs

    let default = i32x4::splat(0);
    let mask = i32x4::from_array([!0, !0, !0, 0]);
-    let vals = unsafe { intrinsics::simd_masked_load(mask, buf.as_ptr(), default) };
+    let vals = unsafe { intrinsics::simd_masked_load(mask, buf.as_ptr(), default, 4) };


i32 doesn't always have alignment 4, so this should use align_of::<i32>()

Oh! Didn't know that, is that true for all primitive types too?

basically all primitive types, though 1 byte types must have alignment 1 due to rust's rule that size is always a multiple of alignment.

e.g. on avr-none, integers/floats have alignment 1, and on msp430-none-elf most types have alignment 2.

programmerjake · 2025-10-05T02:38:32Z

compiler/rustc_codegen_llvm/src/intrinsic.rs

+        // The fourth argument is the alignment, must be a power of two integer constant
+        let alignment = bx
+            .const_to_opt_u128(args[3].immediate(), false)
+            .expect("typeck should have ensure that this is a const");


Maybe I'm missing it because I'm not familiar with typeck's handling of intrinsics, but I do not see where you actually add a typeck check that this is actually a constant. It looks like you're just telling typeck that it's a parameter of u32.

Yeah the expect message here is wrong.

For simd_insert/extract/shuffle, we have some ad-hoc checks in typeck that ensure this. But for intrinsics it's also fine to just ICE when they are used wrong. Rust has no concept of const arguments so this is a bad hack anyway.

programmerjake · 2025-10-05T02:43:18Z

tests/codegen-llvm/simd-intrinsic/simd-intrinsic-generic-masked-store.rs

-    // CHECK: call void @llvm.masked.store.v4p0.p0(<4 x ptr> {{.*}}, ptr {{.*}}, i32 {{.*}}, <4 x i1> [[B]])
-    simd_masked_store(mask, pointer, values)
+    // CHECK: call void @llvm.masked.store.v4p0.p0(<4 x ptr> {{.*}}, ptr {{.*}}, i32 8, <4 x i1> [[B]])
+    simd_masked_store(mask, pointer, values, 8)


this looks like it assumes the alignment of a pointer is 8, but it should be fine even if that alignment is wrong.

I am not assuming anything here, just trying to test if the alignment is propagated to LLVM

RalfJung · 2025-10-05T09:09:26Z

If we only need normally-aligned and unaligned loads, IMO it'd be better to just have a const generic boolean indicating which of them we want for any particular operation. That avoids ad-hoc hacks such as const parameters in intrinsics.

bjorn3 · 2025-10-05T09:19:15Z

compiler/rustc_codegen_llvm/src/intrinsic.rs


+        // The fourth argument is the alignment, must be a power of two integer constant
+        let alignment = bx
+            .const_to_opt_u128(args[3].immediate(), false)


Why not use a const generic? That is a lot easier to implement in some other codegen backends.

programmerjake · 2025-10-05T09:56:25Z

If we only need normally-aligned and unaligned loads, IMO it'd be better to just have a const generic boolean indicating which of them we want for any particular operation. That avoids ad-hoc hacks such as const parameters in intrinsics.

for portable-simd I think we should default to element-level-alignment since I expect that to be more efficient than unaligned ops on some targets (GPUs? maybe RISC-V V?)

RalfJung · 2025-10-05T11:41:04Z

If we only need normally-aligned and unaligned loads, IMO it'd be better to just have a const generic boolean indicating which of them we want for any particular operation. That avoids ad-hoc hacks such as const parameters in intrinsics.

for portable-simd I think we should default to element-level-alignment since I expect that to be more efficient than unaligned ops on some targets (GPUs? maybe RISC-V V?)

Yes, so...?

IIUC we either want element-level alignment or no alignment, so we can just have a const bool generic controlling that.

programmerjake · 2025-10-05T12:21:22Z

If we only need normally-aligned and unaligned loads, IMO it'd be better to just have a const generic boolean indicating which of them we want for any particular operation. That avoids ad-hoc hacks such as const parameters in intrinsics.

for portable-simd I think we should default to element-level-alignment since I expect that to be more efficient than unaligned ops on some targets (GPUs? maybe RISC-V V?)

Yes, so...?

I thought you meant full-simd-type alignment or unaligned, since that's what x86 uses for simd instructions it calls aligned.

RalfJung · 2025-10-05T12:45:04Z

This is about simd_masked_load/store which are currently documented as

/// Unmasked values in `T` must be readable as if by `<ptr>::read` (e.g. aligned to the element
/// type).

sayantn · 2025-10-05T12:47:49Z

As a summary, we need 3 types of alignments

element alignment (used in portable simd)
simd type alignment (x86 aligned)
fully unaligned (x86 unaligned)

So a bool flag won't cut it, at best we can use a const generic parameter, with 0 meaning element size aligned (because that is the most used, and can't be specified using const generics (requires gce))

RalfJung · 2025-10-05T12:53:19Z

As a summary, we need 3 types of alignments

Now you are expanding the scope of the PR. So far the motivation has been, we'd like an unaligned version of the existing intrinsics. If you also want SIMD type aligned variants, the PR description needs to be expanded to argue for this.

IIRC, last time this was looked into, the SIMD type alignment option wasn't necessary -- LLVM was more than able to use surrounding info on reference types to deduce the right alignment for the desired codegen. So please show some concrete undesirable codegen if you want to motivate a form of this intrinsic that requires SIMD type alignment.

sayantn · 2025-10-05T13:06:14Z

I apologise if I was unclear, but the motivation was always adding these 3 types of loads. LLVM will always generate an unaligned (byte-aligned) load/store if we pass any alignment less that the vector type size (because it is guaranteed to be safe). But for _mm_mask_load_ intrinsics, the pointer needs to be aligned to the vector size, so LLVM won't generate aligned loads unless we pass the vector size as alignment.

RalfJung · 2025-10-05T13:31:38Z

LLVM will always generate an unaligned (byte-aligned) load/store if we pass any alignment less that the vector type size

I don't think the "always" here is correct. If we are loading from an &__m128 LLVM should be able to use that fact to generate the aligned intrinsics.

However, I guess stdarch uses raw pointers in its API. So yeah this definitely needs to be explained properly in the PR description, currently it is at best confusing.

RalfJung · 2025-10-05T13:36:39Z

If we need 3 different alignment modes (Cc @Amanieu for the stdarch part here), that can still be done using const generics with a new 3-variant enum (similar to the enum we have for atomic memory access orderings).

workingjubilee · 2025-10-05T17:44:43Z

Yes, std::arch tries to reflect the type signatures used by the C vendor functions... it's not exactly just "bindgen for vendor functions", but it kinda is bindgen for vendor functions.

Amanieu · 2025-10-05T22:39:51Z

I'm happy with the current API that takes a constant (either as an argument or a const generic). An enum doesn't really provide much of an advantage when the desired alignment can just be explicitly provided.

sayantn · 2025-10-06T03:16:45Z

If everyone agrees, I can substitute the const argument for a const-generic u32 alignment. But then portable-simd would face problems because it has to pass align_of::<T> to the const-generic argument somehow, so I propose to add a special meaning to 0 - if 0 is passed as the alignment it is interpreted as the element type's alignment. This doesn't affect stdarch, all invocations of this intrinsic there will use literal (for x86 at least, I don't know much about other archs)

RalfJung · 2025-10-06T07:09:17Z

I'm happy with the current API that takes a constant (either as an argument or a const generic). An enum doesn't really provide much of an advantage when the desired alignment can just be explicitly provided.

The enum provides the big advantage that we don't need more ad-hock "constant argument" hacks.

What I was hoping to get from you is confirmation on which forms of the intrinsic are needed for stdarch.

RalfJung · 2025-10-06T07:10:43Z

If everyone agrees, I can substitute the const argument for a const-generic u32 alignment. But then portable-simd would face problems because it has to pass align_of::<T> to the const-generic argument somehow, so I propose to add a special meaning to 0 - if 0 is passed as the alignment it is interpreted as the element type's alignment. This doesn't affect stdarch, all invocations of this intrinsic there will use literal (for x86 at least, I don't know much about other archs)

I would prefer that over the "constant argument" hack. Not sure if it's better than a 3-value enum but 🤷 .

lcnr · 2025-10-06T13:35:11Z

r? @RalfJung though feel free to reassigned

rustbot assigned lcnr Oct 4, 2025

This comment has been minimized.

Sign in to view

Add alignment parameter to simd_masked_{load,store}

9afbfb1

sayantn force-pushed the masked-loads branch from 4cec11b to 9afbfb1 Compare October 5, 2025 00:27

programmerjake reviewed Oct 5, 2025

View reviewed changes

bjorn3 reviewed Oct 5, 2025

View reviewed changes

rustbot assigned RalfJung and unassigned lcnr Oct 6, 2025

Add alignment parameter to simd_masked_{load,store} #147355

Are you sure you want to change the base?

Add alignment parameter to simd_masked_{load,store} #147355

Conversation

sayantn commented Oct 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Alternatives

Uh oh!

rustbot commented Oct 4, 2025

Uh oh!

rustbot commented Oct 4, 2025

Uh oh!

This comment has been minimized.

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

RalfJung commented Oct 5, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

programmerjake commented Oct 5, 2025

Uh oh!

RalfJung commented Oct 5, 2025

Uh oh!

programmerjake commented Oct 5, 2025

Uh oh!

RalfJung commented Oct 5, 2025

Uh oh!

sayantn commented Oct 5, 2025

Uh oh!

RalfJung commented Oct 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sayantn commented Oct 5, 2025

Uh oh!

RalfJung commented Oct 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

RalfJung commented Oct 5, 2025

Uh oh!

workingjubilee commented Oct 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Amanieu commented Oct 5, 2025

Uh oh!

sayantn commented Oct 6, 2025

Uh oh!

RalfJung commented Oct 6, 2025

Uh oh!

RalfJung commented Oct 6, 2025

Uh oh!

lcnr commented Oct 6, 2025

Uh oh!

Uh oh!

Add alignment parameter to `simd_masked_{load,store}` #147355

Add alignment parameter to `simd_masked_{load,store}` #147355

sayantn commented Oct 4, 2025 •

edited

Loading

RalfJung commented Oct 5, 2025 •

edited

Loading

RalfJung commented Oct 5, 2025 •

edited

Loading

workingjubilee commented Oct 5, 2025 •

edited

Loading