Tighter shadow culling - fix light colinear to frustum edge

[lawnjelly]

In rare situations if a light is placed near colinear to a frustum edge, the extra culling plane derived can have an inaccurate normal due to floating point error. This PR detects colinear triangles, and prevents adding a culling plane in this situation.

Fixes #89702

Notes

• Epsilons were measured with typical scales (assuming 32 bit real, there's little to gain by making them smaller for 64 bit). These should hopefully be fine but if there are any more reports they can be adjusted.
• There are other alternative ways of checking for co-linearity (some slightly faster, e.g. dot product of two normalized sides), but this seems pretty foolproof and is called only once per edge per light per frame, so accuracy is more important than speed.

[Calinou]

Tested locally (rebased on top of master 7a42afb), this PR doesn't resolve the issue from the original MRP (#89702 (comment)):

simplescreenrecorder-2024-04-04_15.53.14.mp4

Even if the above video was recorded at 60 FPS, the Max FPS project setting was set to 20 for demonstration purposes.

Disabling the Tighter Shadow Caster Culling project setting resolves the issue. I'm using a 16:9 aspect ratio which is identical to the base window size, so I assume it's what mrjustaguy tested this with.

[lawnjelly]

Ah I'll try again, it could be the epsilon needs adjusting. 👍

I was using a modified version of the scene as it didn't require all the models to show the bug.

UPDATE:
Hmm, I can't seem to replicate it here now, even with the original project. Also both my modified versions work fine.

But given the MRP runs in fullscreen, there could be some small difference given that the bug is to do with float error and epsilons. I'll keep trying to replicate it.

It's possible that I could expose the epsilon in a test build to confirm it fixes for you / tweak the value, as it's pretty empirical if I remember right.

Or you could just bump it up in the source file, it should be the second epsilon currently set to 0.00001f. If you bump it up to say 0.001f and it eliminates the problem then we know it just needs a slightly higher setting.

(Also just to eliminate the obvious, it might be worth double checking you were running the PR build. It's pretty easy to e.g. run a release build after compiling dev build, etc, I've done this many times. 😁 )

Anything special you could mention about your build is useful, compiler, screen resolution, 32 or 64 bit build etc.

UPDATE:
Been trying for an hour now, various resolutions, can't get it to replicate. The frustum is determined by the aspect ratio, so that could affect things, but I tried to give a fairly conservative epsilon.

So I guess @Calinou it would need you to try a few different epsilons and see at what point it is getting fixed on your system. But definitely double check it is running the correct build,

[Calinou]

But definitely double check it is running the correct build,

This was indeed an outdated build issue, for the same reason as in #70766 (comment). Apologies.

I've tested the PR locally and it works now.

[akien-mga]

Thanks!