Implement MSAA or supersampling in ReflectionProbes for a smoother result

[BracerJack]

Godot version

3.3.2

System information

Windows 10, 1650, same problem even for Android Tablets

Issue description

I have had a lot of good feedback and good will from:
https://www.reddit.com/r/godot/comments/o4b5z1/so_after_freaking_my_godot_is_only_used_for_pixel/
Currently the top post at 679 upvotes within 8 hours and I feel maybe this should give me enough confident to ask this question least I look too much like a noob.

Actual Demo - Godot's HTML 5 export is awesome ! I think a lot of people overlooked the realism that Godot can achieve in the HTML 5 department dominated by low poly retro simple js demos:
https://www.bracerjack.com/godot/NDE_Symposium/

I managed to make it look realistic thanks to Godot's rendering tech but I couldn't get the edges inside the reflection and ONLY inside the reflection to anti-alias, the edges on everywhere else OUTSIDE of the reflection zone looks great, just the edges inside the reflection looks really jagged.

I have turned on Msaa to 4x and it makes all the edges looks anti-alias, which is good and beautiful, but no effect on the reflection, the edges inside the reflection continues to look non-aliased.

I have set the Atlas Subdiv to 1 because I am using only one reflection probe, though the reflection image clears up good, the edges inside the reflections are still non-aliased.

I have tried increasing the Atlas Size all the way to 8K, the reflection data clears up but the edges inside the reflections are still non-aliased.

Steps to reproduce

Model a Building in Godot to scale, texture it.
Put in one reflection probe, set Atlas Subdiv to 1.

You will find that increasing Atlas Size will not anti-alias edges in the reflection.
You will find that enabling Msaa, no matter how high you go, will not anti-alias edges in the reflection.

[Calinou]

MSAA does not impact aliasing that is caused by fragment shaders (such as specular aliasing). This is not a Godot limitation – any engine that implements MSAA will have this issue. To combat specular aliasing, you need to use supersampling (which is very expensive) or enable a form of post-processing antialiasing such as FXAA (which is fast, but blurs the image). Since Godot 3.3, FXAA can be enabled in the project settings or on specific Viewports.

Also, reflection probes have a limited resolution, which means that filtering artifacts will be visible when you are up close. This could be fixed by rendering reflection probes at a higher resolution then downscaling them (effectively acting as supersampling), but it would make reflection probes in 3.x even slower than they currently are. As a workaround, you can increase roughness on the reflective material to make the reflection blurrier.

That said, it may be the case that MSAA is ignored when reflection probes are rendered. However, this should be exposed as an optional property rather than always enabled, since it would make reflection probe rendering slower. cc @clayjohn

[BracerJack]

Thank you for the education !
I tried FXAA, there are some changes, but it never hit the subject head on.

But your second statement clears up EVERYTHING !
That explains it !
So...as of right now Godot version 3.3.2, this is actually something that is actually an issue then because the option is not exposed, thank you !

I will now update the Reddit post with a link to the explanation here as it is current still the number one post at 679 upvotes and people are starting to ask the same questions after seeing the demo.

Thank you !

[Calinou]

So...as of right now Godot version 3.3.2, this is actually something that is actually an issue then because the option is not exposed, thank you !

I don't know if this is possible to expose. I haven't seen a reference to render targets in the GLES3 reflection probe code, and you likely need render targets to enable MSAA. And I'm not even talking about GLES2 here 🙂

[BracerJack]

But thanks to you, the awareness for a need for this is now at the surface.
I am just a user, I don't know this, I thought it was my fault.
So Thank you !

[clayjohn]

This is likely a result of the upper limit on cubemap resolution used for rendering reflections

godot/drivers/gles3/rasterizer_scene_gles3.cpp

Line 5084 in 2966084

int max_reflection_cubemap_sampler_size = 512;

We recently removed the similar limit from shadow maps in #47160 and #48059 we can do the same for reflection cubemaps

[BracerJack]

@clayjohn, thank you for trying to tackle the issue, much appreciated.
However, I am under the understanding that the issue is not merely a matter of resolution but that the reflection probe does its "capturing" before anti-aliasing occurs (look at picture on the third post).
Would "removing the limit" cause the probe to be able to capture the anti-aliased version of the environment ?
I apologize if I am talking nonsense here.

[clayjohn]

@BracerJack There are two distinct issues in your screenshots above:

1. The reflections are low resolution so blocky artifacts are very visible
2. the blocky artifacts aren't smoothed out by anti-aliasing

To solve 1. we can do as I suggested above and change the limit on cubemap sizes (optionally, with a project setting). This will result in much smoother edges and will decrease how noticeable the aliasing is. This also allows you to supersample your reflections (as Calinou suggests above)

To solve 2, we would need to completely rewrite how reflection probes work under the hood to use MSAA compatible renderbuffers. However, using MSAA would allow us to smooth out the aliasing regardless of the resolution used. The render buffers used for reflections are different from the one used to draw the main screen, so setting MSAA on your main viewport has no effect on the reflection probes.

To get perfect reflections, we need both 1 and 2.

[Calinou]

the blocky artifacts aren't smoothed out by anti-aliasing

One way to solve this without multisampling would be to use supersampling, which means rendering at double the resolution then halving the resolution by resizing the cubemap with linear interpolation. However, due to the high cost of doing so, this is only viable for probes that are updated Once, not probes that are Always updated.

I would personally pick a blurrier but supersampled probe over a very sharp but aliased one. Sharper is not always better 🙂

[BracerJack]

"The render buffers used for reflections are different from the one used to draw the main screen, so setting MSAA on your main viewport has no effect on the reflection probes."
That helps me to understand the situation clearly, thank you @clayjohn, if and when number 2 is implemented would be a very exciting day for me ;p

"I would personally pick a blurrier but supersampled probe over a very sharp but aliased one. Sharper is not always better"
Agree 100% @Calinou, I am not into sharp reflections, it actually looks quite artificially lame even if possible, I would rather smooth anti-aliased edges on the reflection, even if the reflections are blurred.

[Calinou]

I've toyed around with the reflection probe roughness blur filter in 3.x. It's possible to use a minimum value for the roughness and effectively trick it into generating slightly blurry mipmaps for perfectly smooth (non-rough) surfaces. This is done here:

godot/drivers/gles3/rasterizer_scene_gles3.cpp

Line 730 in c550033

storage->shaders.cubemap_filter.set_uniform(CubemapFilterShaderGLES3::ROUGHNESS, rpi->render_step / 5.0);

Replace rpi->render_step / 5.0 with MAX(0.05, rpi->render_step / 5.0) and recompile. 0.05 still looks fairly sharp, but is much less aliased compared to 0.0 (the default for perfectly smooth surfaces).

We can expose this as a project setting, so that you can choose between having sharp but aliased reflections or smoother but blurrier reflections. We should discuss what the default value should be, but I think we can make it 0.05 by default since you can't expect ReflectionProbe reflections to be always accurate all times. In this case, a slightly blurrier reflection generally looks better.

This option can likely be provided in the master branch as well, and possibly even GLES2 in 3.x.

[BracerJack]

I don't know how to do the whole re-compile thing but as soon as this 0.05 is implemented I will update that demo :D
Thank you @Calinou for taking a try at it :D