Mesh Shaders

[inodentry]

Mesh Shaders are an exciting new kind of rendering pipeline for modern hardware, that directly combines compute and rasterization.

KHRONOS recently officially released an extension for Vulkan, which is now quickly gaining support in different drivers. DX12 has had them for a while now. Metal (to my knowledge) also has them. Therefore, it should now be available on at least all the desktop platforms (given compatible hardware) and iOS.

Would it be possible for wgpu to provide access to this functionality?

There is a WebGPU tracking issue for it: gpuweb/gpuweb#3015 , but looks like it is not a priority and unlikely to be added to the WebGPU standard soon.

Perhaps wgpu could support it as a "native-only" extension?

[expenses]

Mesh shaders as a native-only extension should be simple enough to implement, at least for vulkan. I'd like to look into this but I don't have a huge amount of free time to do so (similar to #1040 (comment)).

For anyone else interested, https://github.com/nvpro-samples/gl_vk_meshlet_cadscene is a vulkan example that uses the new VK_EXT_mesh_shader extension.

[heavyrain266]

Mesh shader extension in WGSL could be really helpful for maintaining e.g. PSSL support in naga and PS5 API in wgpu, current implementation avoids usage of wgsl and requires rewriting all shaders in PSSL which is rather annoying to maintain.

[zmarlon]

Since there are now mesh shaders on DX12, Metal and Vulkan, it would at least be possible on all modern APIs. DX11, OpenGL ES and WebGPU would of course not work. Would there be any interest in seeing mesh shaders in WGPU? If so, I might try to implement it.

[cwfitzgerald]

There definitely is interest! It's going to be a big project though, so do join us in the chat room to chat about the design before starting work.

[JunkuiZhang]

Any progress have been made? Its a very exciting feature and maybe the future of game engine dev.

[lylythechosenone]

Indeed, I would like to know this too. Has work actually started yet, or is this still just an idea?

@cwfitzgerald

[cwfitzgerald]

No one has taken lead in it - it's a rather involved thing, needing both improvements to Naga and wgpu.

If someone was so motivated, we could walk through potential implementation plans.

[ZoopOTheGoop]

Tag me in, I'd need a bit of a push in the right direction, but I'm willing to do the legwork.

[JMS55]

I'm not a wgpu maintainer, and can't help implement it, but I'd be happy to help test it out if you do implement mesh shaders. It's a feature I've been very much wanting.

In terms of implementation you would need to:

• Figure out the common denominator between metal/vulkan/dx12 for an API to implement
• Figure out all the edge cases and things that could go wrong, and how you would add validation for it
• Add mesh shader support to naga's WGSL frontend
• Add mesh shader support to naga's spirv/hlsl/msl backends

I definitely encourage you to join the matrix channel linked in the wgpu readme and talk to the wgpu devs there if you want to implement mesh shaders.

[Bromles]

Looks like mesh shaders may be added to the WebGPU standard itself after 1.0

gpuweb/gpuweb#3015

[SupaMaggie70Incorporated]

Overview & problem

I've been looking at the issue of mesh shading recently, and I have some ideas for how it could be implemented into wgsl. I have so far only looked at DirectX and Vulkan, but I would imagine Metal would be somewhat similar. The only features added to the CPU side are

• Creating mesh-shaded render pipelines
• Drawing using these pipelines(including indirect draws)

Creating the mesh pipeline seems to be very similar to creating standard pipelines in vulkan and directx, at least in terms of what is required from the public facing API. I expect metal to be similar. The draw calls are very simple and pretty much identical between vulkan and directx, and from looking at metal examples they seem to be similar there(though metal does differentiate between drawing with threads and threadgroups).
The GPU side of things is where all of the complexity comes in. Below is a non-exhaustive list of what would need to be added:

• A mesh shader enable extension
• Task & mesh shader entry points
  • Not a real issue but should we call task shaders task shaders or amplification shaders? I will go with task shaders for now
• Allow most compute operations in these stages
• setMeshOutputs and emitMeshTasks functions/operations, for mesh and task functions respectively
  • emitMeshTasks would probably take an argument of a task_payload variable for compatibility with HLSL even though GLSL automatically determines the task_payload variable to use and SPIR-V doesn't require this as an argument at all.
• A way to mark fragment inputs as per primitive
• Possibly a new address space, task_payload, for carrying data from the task shader to the mesh shader
  • HLSL avoids this by simply using workgroup storage and setting it to task_payload if used as an emitMeshTasks parameter
• Builtins
  • Allow most compute stage builtins to be used in task and mesh shaders
  • Allow most vertex stage output builtins to be output by mesh shaders
  • Allow primitive_id to be written to by mesh shader
  • cull_primitive: bool - output of the mesh shader per primitive
  • point_index: u32, line_indices: vec2<u32>, triangle_indices: vec3<u32> for outputting the indices of a specific primitive
• The most important problem is that of how to actually output the data from mesh shaders.

While a decent amount of work, most of this could probably be done in at most a few days. The main problem is with the handling of complex outputs from the mesh shader. To that end, I have 3 main ideas for how this could be done in WGSL. I will show code snippets and explain each idea. Note that all of the below are just prototypes. The names of builtins or functions or the exact syntax could change at any time.

Main proposal - Metal inspired

struct VertexOutput {
	@builtin(position) position: vec4<f32>,
	@location(0) color: vec4<f32>,
}
struct PrimitiveOutput {
	@builtin(triangle_indices) index: vec3<f32>,
	@builtin(cull_primitive) cull: bool,
	@location(1) colorMask: vec4<f32>,
}

@mesh
@workgroup_size(1)
fn ms_main<VertexOutput, PrimitiveOutput>(@builtin(local_invocation_index) index: u32, @builtin(global_invocation_id) id: vec3<u32>) {
	setMeshOutputs(3, 1);
	setVertex(0, VertexOutput { ... });
	...
	setPrimitive(0, PrimitiveOutput { ... });
}

This would be the easiest to implement while requiring few features to the WGSL language itself. The main issue would be with the logic around which types are used for vertex and primitive output. However, once implemented, it would function relatively well, work with all major APIs(that I'm aware of), and not have any potential performance drawbacks.

Proposal #​2

struct VertexOutput {
	@builtin(position) position: vec4<f32>,
	@location(0) color: vec4<f32>,
}
struct PrimitiveOutput {
	@builtin(triangle_indices) index: vec3<f32>,
	@builtin(cull_primitive) cull: bool,
	@location(1) colorMask: vec4<f32>,
}
struct MeshOutput {
	@per_vertex vertices: array<VertexOutput, 3>,
	@per_primitive primitives: array<PrimitiveOutput, 1>,
}

@mesh
@workgroup_size(1)
fn ms_main(@builtin(local_invocation_index) index: u32, @builtin(global_invocation_id) id: vec3<u32>) -> MeshOutput {
	...
}

This is probably the most WGSL-like method. While on the surface it seems reasonable, the problem arises with how naga handles function outputs in SPIR-V(and probably other targets). Currently, naga will create a struct, fill in values, and then copy those values to the actual output, which means you are doubling the number of writes required. Doing that for mesh shaders would involve many copies in loops, with the number of copies determined by the parameters to setMeshOutputs. While this is probably optimized away by many drivers, and probably poses little performance problems, it leads to unnecessarily long and complicated code that could perform worse on vulkan implementations that don't make this optimization. If we could make substantial internal improvements to naga or verify that this doesn't significantly impact performance, this could be the method of choice.

We would also need to make several changes to WGSL, such as allowing array outputs and in general a more complex output system. This approach would mean a lot of work.

Proposal #​3

struct VertexOutput {
	@builtin(position) position: vec4<f32>,
	@location(0) color: vec4<f32>,
}
struct PrimitiveOutput {
	@builtin(triangle_indices) index: vec3<f32>,
	@builtin(cull_primitive) cull: bool,
	@location(1) colorMask: vec4<f32>,
}
@vertices
var<out> vertices: array<VertexOutput>;
@primitives
var<out> primitives: array<PrimitiveOutput>;

@mesh(triangles, 3, 1)
@workgroup_size(1)
fn ms_main(@builtin(local_invocation_index) index: u32, @builtin(global_invocation_id) id: vec3<u32>) {
	...
}

This is the option that aligns best with GLSL.

Full example

This is a full example of a WGSL shader using proposal #​1 showcasing most features and how I expect them to be implemented.

enable mesh_shading;

const positions = array(
	vec4(0.,-1.,0.,1.),
	vec4(-1.,1.,0.,1.),
	vec4(1.,1.,0.,1.)
);
const colors = array(
	vec4(0.,1.,0.,1.),
	vec4(0.,0.,1.,1.),
	vec4(1.,0.,0.,1.)
);

struct TaskPayload {
	colorMask: vec4<f32>,
	visible: bool,
}
var<task_payload> taskPayload: TaskPayload;
var<workgroup> workgroupData: f32;

struct VertexOutput {
	@builtin(position) position: vec4<f32>,
	@location(0) color: vec4<f32>,
}
struct PrimitiveOutput {
	@builtin(triangle_indices) index: vec3<f32>,
	@builtin(cull_primitive) cull: bool,
	@location(1) colorMask: vec4<f32>,
}

@task
@workgroup_size(1)
fn ts_main() {
	workgroupData = 1.0;
	taskPayload.colorMask = vec4(1.0, 1.0, 0.0, 1.0);
	taskPayload.visible = true;
	emit_mesh_tasks(3u, 1u, 1u, &taskPayload);
}

@mesh
@workgroup_size(1)
fn ms_main(@builtin(local_invocation_index) index: u32, @builtin(global_invocation_id) id: vec3<u32>) {
	set_mesh_outputs(3u, 1u);
	workgroupData = 2.0;
	setVertex(0, VertexOutput {
		position: positions[0],
		color: colors[0] * taskPayload.colorMask,
	});
	setVertex(1, VertexOutput {
		position: positions[1],
		color: colors[1] * taskPayload.colorMask,
	});
	setVertex(2, VertexOutput {
		position: positions[2],
		color: colors[2] * taskPayload.colorMask,
	});
	setPrimitive(0, PrimitiveOutput {
		index: vec3<u32>(0, 1, 2),
		cull: !taskPayload.visible,
		colorMask: vec4<f32>(1.0, 0.0, 1.0, 1.0),
	});
}
@fragment
fn fs_main(vertex: VertexOutput, primitive: PrimitiveOutput) -> @location(0) vec4<f32> {
	return vertex.color * primitive.colorMask;
}

Summary

In summary, I believe this could be accomplished in a relatively short amount of time, with the best option for the WGSL implementation being in my opinion proposal #​1. I am asking for input from anybody else who has recommendations or ideas or sees a potential problem, before I actually start working on this. On a completely separate note, this might also make it easier to implement other improvements that people want(like tesselation shaders), though I don't know enough about those features to know how one would go about implementing them.

Also, sorry for the long drawn out(and obviously unpracticed) format. I am asking for advice and recommendations here, and just throwing out some ideas. I haven't really done many open source contributions before or written any RFCs.

[cwfitzgerald]

Hey! I just wanted to say thank for putting this proposal together! It'll be a few days until I can digest it, but this is great!

[Vecvec]

point_index: u32

I think points may need to be an extra extension as DirectX seems not to support it

from https://www.khronos.org/blog/mesh-shading-for-vulkan#portability

	DirectX 12	VK_EXT_mesh_shader
Supported primitives	triangles, lines	triangles, lines, points

I'm not sure if 0 length lines are rendered (it might depend on the hardware) but if they are then that could probably be used as a point. DirectX doesn't appear to mention if this approach would work, so I suspect it could take some testing.

[SupaMaggie70Incorporated]

That's an interesting idea. I worry that there isn't a good way to render non one width lines with directx, and a one pixel point probably isn't desirable. Granted I've never used directx myself, so I may be wrong about any number of things. It does seem that at least initially, point rendering should be put behind a feature flag(or omitted entirely).

[SupaMaggie70Incorporated]

One other question I have regarding the pipelines is whether they should be just standard RenderPipeline's or whether they should have their own mesh shader pipeline type. Since they have different commands, it may be useful to differentiate the two to mitigate bugs or user error, but all 3 major APIs have decided not to.