GPULightmapper: skip smoothen positions for flat triangles

[williamd67]

Smoothening positions for flat, non-smoothened, triangles is unnecessary and
caused positions to move outside their triangle which caused side-effects as
rays from those positions intersected with triangles which could not be
reached from the original triangle.

This is solved by skipping smoothening of positions for flat triangles.
A triangle is determined to be flat as its vertex normals are equal.

This PR replaces #53264 which will be closed.
This PR fixes the artifact of strange blob that was mentioned in #51638.

After removing the smoothening of positions:

[jcostello]

@williamd67 The issue seems to be fixed. How ever in the Test.tscn scene there is a weird thing when baking in diferent qualities. Not sure if its related with the fix

Low

Medium

High

Ultra

[williamd67]

@jcostello I have an idea what causes this. If so, it should not happen with a single bounce. In that case it is also not related to this fix.

With the introduction of ignoring back faces, I also added an 'ActiveRay' counter to ensure that only rays that actively contribute to the light of a texel count. In the bounces after the first, the rays that reach the sky (no hit) are not counted. This may cause this effect as a single ray that hits the lighted wall may add a lot of light to this texel (all other rays reach the sky and are not counted). I expect that due to the higher quality more rays are traced and there is a higher chance that one may hit the lighted wall. The only thing I cannot explain is why medium is lighter than high. Yet it may be caused by a ray under a specific angle that is only traced in medium and not in high.

BTW: if this is the case there will be an easy fix (increment 'ActiveRay' counter for rays that reach the sky in all bounces). I will test this when I have some time and create a new PR.

[jcostello]

Very nice. Thank you for the hard work. I enjoy testing your PRs

[jcostello]

Since this resolve the original issue and @williamd67 confirms that my comment is not related with the fix. I think this would be ready for merge once the code is reviewed

[JFonS]

Since it seems to be a precision issue, I'm a bit concerned with using any(notEqual(norm_a, norm_b)) to compare the normal vectors. The same issue will most likely also happen with normals that are very similar but not exactly the same, so I would use dot(norm_a, norm_b) < 0.99 instead. The comparison value can be adjusted, but 0.99 corresponds to ~8 degrees between vectors, which seems reasonable to me. Very flat faces won't have many shadow termination issues anyway, so we can leave a safe margin.

[williamd67]

Good point, I will update this PR. And as the flatness of a triangle is independent of the fragment, I moved the code to the vertex-shader.

[akien-mga]

Thanks!