Implement MSAA per-sample shading option to improve rendering quality

[Calinou]

Compared to setting Scaling 3D > Scale above 1.0, this provides better quality (thanks to MSAA's superior sampling pattern) with better performance and lower VRAM utilization.

This also works on 2D rendering, which can be useful to maximize sharpness of downscaled 2D textures without graininess when not using mipmaps. In 2D, this also provides antialiasing for hard Light2D shadows.

Per-sample shading is mainly targeted at Movie Maker mode usage, although modern high-end GPUs can handle per-sample shading with 2× or 4× MSAA during gameplay while staying above 60 FPS.

This is supported in Forward+ and Forward Mobile. However, most mobile GPUs don't support per-sample shading as a hardware limitation (this includes Apple Silicon).

• This closes Implement MSAA per-sample shading option to improve rendering quality godot-proposals#6698.
• This also closes Vulkan: 2D msaa does not AA clipped children #66005 independently of Cascaded canvas groups and canvas group shader revamp #74859.
• Full per-sample shading also addresses Visible seams between GridMap tiles (sometimes only with MSAA enabled) #16337, but we should find a faster solution for that particular problem.

TODO

• Check for hardware per-sample shading support before trying to enable it to avoid crashes on unsupported hardware (as done for the MSAA level).
• Implement in Direct3D 12 if an equivalent exists there. (To my knowledge, a Metal equivalent to this feature does not exist.)
• Make setting changes effective without requiring a restart. Right now, it will sometimes work in the editor (likely because a shader recompilation is triggered somewhere), but not in the running project and not in 2D.
• Use separate project settings for 2D and 3D, so you can control per-sample shading with more granularity (and with a lower performance impact if you only need it on 2D or 3D).
  • Relevant conversation on the Godot Contributors Chat (backup)

Performance

Tested at https://github.com/Calinou/godot-reflection's spawn location.

OS: Fedora 37
CPU: Intel Core i9-13900K
GPU: NVIDIA GeForce RTX 4090

Antialiasing type	1152×648	3840×2160
No AA	868 FPS (1.15 mspf)1	324 FPS (3.09 mspf)
Render Scale 2.0 (old supersampling approach)	644 FPS (1.55 mspf)1	93 FPS (10.75 mspf)
4× MSAA + 4× SSAA (this PR)	876 FPS (1.14 mspf)1	209 FPS (4.78 mspf)

Preview

1152×648

No AA	Render Scale 2.0 (old supersampling approach)	4× MSAA + 4× SSAA (this PR)	Ground truth2

3840×2160

No AA	Render Scale 2.0 (old supersampling approach)	4× MSAA + 4× SSAA (this PR)

Footnotes

1. Performance wildly fluctuates at the default resolution, likely due to a CPU bottleneck. ↩ ↩² ↩³

2. Render in a window twice as large, then downsample in an image editor. ↩

[jcostello]

Really nice improvement in quality and performance. I notice some lost in texture quality. The wall bricks looks blurier like in FXAA, Is that expected?

Edit: Only in 1152×648

[Calinou]

Really nice improvement in quality and performance. I notice some lost in texture quality. The wall bricks looks blurier like in FXAA, Is that expected?

I'm not sure why this is happening. This likely has to do with SDFGI reflections looking different depending on the actual viewport resolution, but not per-sample shading sample count (which is strange). I've tried using a mipmap LOD bias of -1.0 when using per-sample shading, but that didn't improve the situation.

I've added a "ground truth" example to OP to show what would be the best possible appearance.

[clayjohn]

I'm not sure this is worth supporting. It is limited to Vulkan, and hardware support is not very good. It is a cool feature, but the limitations make it pretty hard to justify supporting in a cross platform engine.

[izarii-dev]

about the support on DirectX 12, I found that NVIDIA can do per-sample shading at driver level for DirectX11 games. I'm not sure how they did this but it can be helpful, they called it "variable rate supersampling"
the minimum GPU for this feature is GTX 1650 Super

https://developer.nvidia.com/vrworks/graphics/variablerateshading#:~:text=Variable%20Rate%20Supersampling%20(VRSS),-FIGURE%203%3A%20VRSS&text=Variable%20Rate%20Supersampling%20(VRSS)%20leverages%20NVIDIA%20Variable%20Rate%20Shading%20(,increase%20image%20quality%20and%20performance.

[Calinou]

@izarii-dev That feature is only meant to be used in VR.

There also exist driver-specific MSAA settings that act as supersampling (SGSSAA or "16x MSAA"), but these are not available at an application level without using proprietary APIs.¹ 16x MSAA (which was typically 4x SSAA + 4x MSAA) was available in OpenGL, but not in Vulkan or Direct3D 12 🙁

Footnotes

1. Only a handful of games have an in-game SGSSAA option for this reason, the most notable examples being some PC ports developed by PH3 such as Ys X: Nordics. ↩

[alyssarosenzweig]

unless I'm very confused, per-sample shading is a standard feature supported even with gles3.2 and metal... although you may need to trigger it implicitly with a gl_SampleID read (or equivalent) rather than being able to set minSampleShading?

[alyssarosenzweig]

glMinSampleShading is available in gles3.2 at least

[clayjohn]

unless I'm very confused, per-sample shading is a standard feature supported even with gles3.2 and metal... although you may need to trigger it implicitly with a gl_SampleID read (or equivalent) rather than being able to set minSampleShading?

It seems like I was the one confused, I thought we were talking about VK_KHR_fragment_shading_rate, not sampleRateShading.

We already expose the sampleShadingEnable and minSampleShading members of VkPipelineMultisampleStateCreateInfo in the Vulkan backend through our RDPipelineMultisampleState class. We do still need to verify that sampleRateShading is true though.