Raytracing support

[teaglin]

Hi, I was curious if there is any support for ray tracing. I am interested in prototyping raytracing applications on macOS with Metal 2 and then scaling them up for production on Nvidia's RTX GPUs.

[cdavis5e]

Theoretically, it should be possible to support the VK_NV_ray_tracing extension on Mac, using the Metal Performance Shaders framework which uses a very similar technique to accelerate ray tracing. At this time, though, ray tracing isn't supported.

[billhollings]

If anyone wants to take a run at implementing this...it would be welcome.

[teaglin]

I might be interested, but I probably won't be free for a few weeks to a month for other priorities.

[Lichtso]

Why not go straight for the newest (vendor independent) VK_KHR_ray_tracing extension?
https://www.khronos.org/news/press/khronos-group-releases-vulkan-ray-tracing

[billhollings]

Why not go straight for the newest (vendor independent) VK_KHR_ray_tracing extension?

This is an older thread. But yes...we should now aim for the KHR version.

[Lichtso]

One of the dependencies of VK_KHR_ray_tracing is VK_KHR_buffer_device_address. Which requires the host to know the physical addresses (pointers) inside the device. However, it seems that Metal does not expose that information to the user.

[oscarbg]

Came here searching for VK_KHR_buffer_device_address references, as is a recommended extension for current VKD3D Proton 2.0..
let me point out that now raytracing on Metal (on iOS 14.0 and Mac 11.0) seems more similar to VK_KHR_ray_tracing having added support for raytracing shaders (intersection, etc..) (i.e. integrated into Metal shading language)..
seems this new support requires on Mac:
*RDNA GPU or later (no Vega or earlier support)
*Intel Skylake GPU or later
and on IOS: A13 or later..
recalling from memory so might be wrong..

[dictoon]

Any news on this?

[oscarbg]

@krupitskas so you will work on spirv cross raytracing metal support?
also may be a good idea to look at https://github.com/ConfettiFX/The-Forge crossplatform raytracing support they support DXR Metal and Vulkan Rt.

[oscarbg]

seeing AMD RDNA2 6xxx GPUs now supported on MacOS 11.4 betas.. would be nice if someone can test if Metal Performance Shaders Raytracing demos get some "huge" speedups on these GPUs:

https://developer.apple.com/documentation/metalperformanceshaders/metal_for_accelerating_ray_tracing
https://developer.apple.com/documentation/metalperformanceshaders/animating_and_denoising_a_raytraced_scene

this may indicate it's being mapped to new RDNA2 raytracing ISA..

anyway WWDC 21 is near and we (hope)surely will get some update on raytracing on Metal on MacOS..

[billhollings]

NVIDIA Vulkan sample KhronosGroup/Vulkan-Samples#274 both include use of Ray Queries

[kode54]

Metal Performance Shaders Raytracing is also available on Apple Silicon, with the added benefit of being able to run in a single rendering pass.

[zmarlon]

Since buffer device address now available in Metal 3, VK_KHR_buffer_device_address has been included in the latest version of MoltenVK. Now you should actually be able to implement VK_KHR_ray_tracing? Is there any information in this direction?

[cdavis5e]

Since buffer device address now available in Metal 3, VK_KHR_buffer_device_address has been included in the latest version of MoltenVK. Now you should actually be able to implement VK_KHR_ray_tracing? Is there any information in this direction?

I have plans to work on this over the next six months. (Full disclosure: I'm being paid to do this.)

[natevm]

@cdavis5e I'm working on a Vulkan ray tracing library which uses most all features of Vulkan RT (SBT record data, AABB trees, etc). I'd be very interested to build off your work if I can. :)

[fknfilewalker]

What is the status on this?

[LeeTeng2001]

Also curious about the status

[mannewalis]

Any updates on the status of ray tracing support on Mac with MoltenVK?

[natevm]

Since buffer device address now available in Metal 3, VK_KHR_buffer_device_address has been included in the latest version of MoltenVK. Now you should actually be able to implement VK_KHR_ray_tracing? Is there any information in this direction?

I have plans to work on this over the next six months. (Full disclosure: I'm being paid to do this.)

Seeing as it’s been nearly a year since this comment was made, it might be worth hunting for another developer to work on this.

[AntarticCoder]

To implement raytracing, we need the dependencyVK_KHR_acceleration_structure first, and that extension needs VK_KHR_deferred_host_operations. If we were to implement acceleration structures, we could also implement VK_KHR_ray_query, so that could be some what easier.

I want to try a hand at implementing it, but I have no idea where to start, could someone give me some pointers. Thanks

Edit: I meant VK_KHR_deferred_host_operations not VK_KHR_deferred_operations

[natevm]

That would be a great start. VK_KHR_ray_query would also let you test things out without thinking about shader binding table details right away. Ultimately we'd want both queries and the full pipeline for portability reasons, but it's a good idea to break it all down.

I've never actually heard of VK_KHR_deferred_operations before. I see it's mentioned as VK_KHR_deferred_operation in some Intel RT slides, but other than that it's a bit of a dead end... Is this related to VK_KHR_deferred_host_operations? I seem to get more results for that.

https://registry.khronos.org/vulkan/specs/1.3-extensions/man/html/VK_KHR_deferred_host_operations.html
I've talked to Josh Barczak from that man page before on twitter, he's nice and iirc open to DMs. You might reach out to him @JoshuaBarczak

[AntarticCoder]

Thanks for the info, I'll try to contact @JoshuaBarczak.

[cdavis5e]

@gpx1000 ^^

[devshgraphicsprogramming]

To implement raytracing, we need the dependencyVK_KHR_acceleration_structure first, and that extension needs VK_KHR_deferred_host_operations. If we were to implement acceleration structures, we could also implement VK_KHR_ray_query, so that could be some what easier.

I want to try a hand at implementing it, but I have no idea where to start, could someone give me some pointers. Thanks

Edit: I meant VK_KHR_deferred_host_operations not VK_KHR_deferred_operations

You'd need to implement KHR_buffer_device_address first, at least the non-shader side of it.

[devshgraphicsprogramming]

Since buffer device address now available in Metal 3, VK_KHR_buffer_device_address has been included in the latest version of MoltenVK. Now you should actually be able to implement VK_KHR_ray_tracing? Is there any information in this direction?

I have plans to work on this over the next six months. (Full disclosure: I'm being paid to do this.)

Seeing as it’s been nearly a year since this comment was made, it might be worth hunting for another developer to work on this.

I've been offering to implement since 2021, but seems there's no funding available.

[AntarticCoder]

@devshgraphicsprogramming

I believe VK_KHR_buffer_address has been implemented already according to MoltenVK/Layers/MVKExtensions.def on the line

MVK_EXTENSION(KHR_buffer_device_address, KHR_BUFFER_DEVICE_ADDRESS, DEVICE, 13.0, 16.0)

[natevm]

@AntarticCoder is VK_KHR_deferred_host_operations actually required for VK_KHR_acceleration_structure?

I noticed that if I remove VK_KHR_deferred_host_operations, that my vulkan ray tracing application still works. I'm only using the device-side acceleration structure builds though, ie vkCmdBuildAccelerationStructuresKHR rather than vkBuildAccelerationStructuresKHR. But that's probably the most common usage.

vkBuildAccelerationStructuresKHR does consume a VkDeferredOperationKHR for the host-side accel build, but perhaps so long as no host commands are actually deferred, then perhaps this extension isn't required for VK_KHR_acceleration_structure?

Edit: ah, nvm, I get validation errors if I remove that extension.

[AntarticCoder]

@natevm You seem to be right, but would that not be going off the specification? Or are we allowed to deviate from the specification like this?

Some applications may use deferred operations as well as that, especially if they're building on the host, so that would also break some compatabillity, I believe.

[natevm]

@AntarticCoder if it violates spec, then it's likely required to be supported.

The extension seems simple enough itself. Would require defining an opaque VkDeferredOperationKHR handle which I believe is allocated by a vkCreateDeferredOperationKHR. Then, if this type is passed to a deferrable command (eg vkBuildAccelerationStructuresKHR) then that command might return VK_OPERATION_DEFERRED_KHR rather than the normal VK_SUCCESS or etc.

From there, the command can be executed using vkDeferredOperationJoinKHR. There's a vkGetDeferredOperationResultKHR, a vkDestroyDeferredOperationKHR, and then finally a vkGetDeferredOperationMaxConcurrencyKHR which could iiuc could just return the number of available CPU cores perhaps? Seems to depend on the operation passed in to be deferred...

Might be worth making a separate issue on this, then indicating that this issue depends on that one.

[AntarticCoder]

Sure, I'll create an issue.

[rcaridade145]

A related issue from spirv-cross KhronosGroup/SPIRV-Cross#2115

[Try]

A related issue from spirv-cross KhronosGroup/SPIRV-Cross#2115

One more issue to note: spirv-cross also have no support for cullMask, for MSL backed. Also I'm not aware of any good way to implement cullMask in MSL ray-query workflow.

[devshgraphicsprogramming]

A related issue from spirv-cross KhronosGroup/SPIRV-Cross#2115

One more issue to note: spirv-cross also have no support for cullMask, for MSL backed. Also I'm not aware of any good way to implement cullMask in MSL ray-query workflow.

slap more limits into KHR_portability_subset , like that mask always must be 0xFF

[AntarticCoder]

Opened a PR: #1967

[oscarbg]

@AntarticCoder nice work.. seems next step will be implementing VK_KHR_pipeline_library?, before tackling the main raytracing extension?

[AntarticCoder]

Thanks @oscarbg. Not necessarily, only raytracing pipelines need that. Ray queries are probaly going to be implemented first due to their simpler nature.

[FunMiles]

With the M3 chips now supporting hardware accelerated Ray Tracing, this issue is becoming more pressing. Is there any current work being done on this?

[gpx1000]

We have nothing to announce just yet.

[loopervfx]

Any update? it's coming up on a year. even just ray queries (inline raytracing) would be great for countless auxiliary use cases that we have been prototyping even and especially with underpowered hardware. Current limitations on the target platform makes it a difficult proposition. Now the Apple M4 chips have been announced with even better raytracing performance! Exciting times. Thanks much for all your hard work, happy Halloween 🎃

[cdavis5e]

I know about as much as you do. Honestly, at this point, I don't think the contract I was supposed to be hired for to implement this is ever going to go through. I'm not even sure if Holochip still exists, or if @gpx1000 is even still alive--last I heard from him, he had his cancer in remission, but...

[gpx1000]

Hi, we're still here. The contract to implement this doesn't look like it's going to materialize; so it's standing in wait of finding new funding. Nothing else new to report.

[natevm]

If I understand it right, the current limiting issue is that Metal’s BLAS acceleration structures cannot be easily referenced through VkDeviceAddresses, which are required in the TLAS primitives for both ray queries as well as the ray tracing pipeline.

We have nothing to announce just yet.

@gpx1000 this doesn’t really inspire much confidence…

edit: ah, makes sense if funding is not available.

double edit: 😵‍💫 oof, I didnt know about the personal health situation. That’s definitely more important.

Is it possible for MoltenVK to officially assign a funded developer to look into at least ray query support?

[gpx1000]

I don't have anything I can provide beyond saying that there is currently no customer, nor funding for this for us so there's no one I can assign out of Holochip's developers. Unless that situation changes; not much I can say other than sorry I can't do more.

And thanks; with luck I will get to cancer cured at the 5 year mark, for now I'm still cancer free.

[natevm]

One thing I’ve been watching, DX12 recently announced they would be dropping DXIR in favor of SPIR-V. That will affect Apple’s game porting toolkit…

That might mean more funding goes into tooling for SPIR-V on Mac.

There’s also an effort on supporting Metal through NVIDIA’s Slang compiler, and their wrapper around the different host-side API (formerly called GFX, I believe that naming is changing though). I don’t think ray tracing is supported there just yet, but it’s something I know they’re thinking about, and might have the resources to support.

[gpx1000]

I would suggest watching this effort as well: https://gitlab.freedesktop.org/mesa/mesa/-/issues/11990 It is an exciting direction to monitor.

[devshgraphicsprogramming]

If I understand it right, the current limiting issue is that Metal’s BLAS acceleration structures cannot be easily referenced through VkDeviceAddresses, which are required in the TLAS primitives for both ray queries as well as the ray tracing pipeline.

We have nothing to announce just yet.

@gpx1000 this doesn’t really inspire much confidence…

edit: ah, makes sense if funding is not available.

double edit: 😵‍💫 oof, I didnt know about the personal health situation. That’s definitely more important.

Is it possible for MoltenVK to officially assign a funded developer to look into at least ray query support?

VK_KHR_portability_subset comes to rescue, lets make VK_KHR_portability_subset2 and require the TLASes only be accessed from descriptor sets (you can declare special bindings for TLASes btw, you don't need to make them from BDA)

[natevm]

VK_KHR_portability_subset comes to rescue, lets make VK_KHR_portability_subset2 and require the TLASes only be accessed from descriptor sets (you can declare special bindings for TLASes btw, you don't need to make them from BDA)

I haven’t been looking to closely, but I don’t think that solves the problem.

This constraint isn’t device-side, this is a host-side problem. This isn’t about accessing TLAS either, it’s about storing BLAS in TLAS

This is a problem in translating all VKRT code, where users today populate the BLAS device address in an instance primitive (either on the CPU or GPU) which are then passed into to the accel build calls to create the TLAS. Changing this would require a Vulkan spec change, which I don’t personally think makes much sense.

This paradigm cannot be easily translated to Metal, since Metal accels cannot be referenced by a VKDeviceAddress.

@AntarticCoder knows more than me about this.

[devshgraphicsprogramming]

VK_KHR_portability_subset comes to rescue, lets make VK_KHR_portability_subset2 and require the TLASes only be accessed from descriptor sets (you can declare special bindings for TLASes btw, you don't need to make them from BDA)

I haven’t been looking to closely, but I don’t think that solves the problem.

This constraint isn’t device-side, this is a host-side problem. This isn’t about accessing TLAS either, it’s about storing BLAS in TLAS

This is a problem in translating all VKRT code, where users today populate the BLAS device address in an instance primitive (either on the CPU or GPU) which are then passed into to the accel build calls to create the TLAS. Changing this would require a Vulkan spec change, which I don’t personally think makes much sense.

This paradigm cannot be easily translated to Metal, since Metal accels cannot be referenced by a VKDeviceAddress.

@AntarticCoder knows more than me about this.

BTW Afaik, METAL 3 has something similar to BDA.

For any host command, you could use a BDA to Metal BLAS and TLAS handle mapping.

Then the only problem for you is translating BDA to BLAS handles when doing device-side builds, this could be done with a hashmap / small translation dispatch before you proceed to build TLASes device side.