Add raytraced shadows

[mrjustaguy]

Disclaimer: This is Just an idea, I'm no rendering expert and the feasibility (On Current Hardware and Future Hardware) and possible performance of this is highly questionable and as such is not expected to be Seriously Implemented, especially in any near future, but is here instead more for providing inspiration for anyone in the field trying to deal with this issue. Should this end up being doable and implemented, that'd be great to see, but I doubt that'll be the case as to my understanding GPUs have a very Rigid Rendering Pipeline that wouldn't play nice with the idea (as in just wouldn't want to do it)

Describe the project you are working on

A game requiring Good Shadows

Describe the problem or limitation you are having in your project

Shadow Maps are a Pain to get to look Good and without issues like Peter-panning, Shadow Acne etc...

Describe the feature / enhancement and how it helps to overcome the problem or limitation

The Idea's to Not use Shadow maps, but instead, to detect for each pixel, which lights would normally affect it, and to see if said lights are occluded or not (so under shadow or not)

Reasoning:

• There is no point in Shading stuff outside of what will be on screen
• The method wouldn't suffer from Peter-panning and Shadow Acne to the same degree, possibly even eliminating the issues
• Drastically improves Directional Lights, by not having any more distance based fading, splitting and too few pixels in the shadow map to give a good quality shadow

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

1. Render an array of pixels, like a Depth map, but instead gathering exactly the Opaque point in the 3D World that is being rendered by a given pixel.
2. For each of the points, look up which lights it might be affected (which omni lights are in range, which spot lights are in range and looking in a direction that'd shine light on that point, and for directional lights just make sure that the light isn't shining behind the point)
3. For all lights that might affect a given point test by shooting a ray from the point to each light source to test for occlusion (makes sense for directional lights to add a user option to define how long the ray can be as there's no point in having a super long ray testing for occlusions far out where there is nothing, maybe it can be done automatically)
4. If Light isn't occluded, Apply said light to said pixel, if it is occluded do nothing.

If this enhancement will not be used often, can it be worked around with a few lines of script?

No

Is there a reason why this should be core and not an add-on in the asset library?

Rendering is Core

[Calinou]

This sounds like screen-space contact shadows, which are supported in 3.x but were removed in master because of their glitchiness and low performance.

Also, most games can't live with perfectly hard shadows nowadays. They need soft shadows somehow.

If you want to use fully baked lightmapping and have dynamic objects still cast shadow maps, #2354 is likely a better solution.

[mrjustaguy]

Judging by the Way SSCS glitches, my method clearly avoids some of the issues, as SSCS clearly don't get each individual pixel's world point and then ray cast to all possible affecting lights to look for occlusion between the point and the lights, something that SSCS obviously isn't doing on a per-pixel level or it'd have a continuous, perfectly hard shadow without that white gap.

Perfect, Hard shadows versus Imperfect (due to things like Shadow map resolution limitations, PP, and Shadow acne), Soft shadows that are also more complicated to use right.. it's a trade off, and also depends on the style.

#2354 is a good proposal, however it has very little to do with this proposal, aside from improving Shadows..

My proposal would probably suffer from a significant performance drop when used (atleast on older, pre RT hardware), and would probably require RTX to work in a playable form, as it is asking for 120 million rays per light (if each light affects whole screen) per second for 1080p 60 fps, and considering that an RTX 2060 is supposedly capable of shooting 5 Billion rays per second (according to https://en.wikipedia.org/wiki/List_of_Nvidia_graphics_processing_units#GeForce_20_series), that'd amount to a total of ~40 Lights taking up full screen for a standard 16:9 1080p 60 fps XP, however that is only IF RTX can be used for the occlusion tests like intended, which I mean, probably, but might not be the case...

[clayjohn]

@mrjustaguy What you described above is essentially a high level overview of ray traced shadows. Which, as you know, is really only feasible on the highest-end graphics cards right now. RTX hardware provides two improvements necessary:

1. RTX hardware provides optimized ray-triangle intersections (because you will be calculating millions of these)
2. RTX hardware provides efficient ways to store geometry and access it during the ray cast.

Without RTX hardware you are limited to using data the renderer already has available like shadowmaps, or, as Calinou pointed out above, the depth map.

[mrjustaguy]

Ok, I managed to make a Proof of Concept for a way to get the desired effect for Directional Lights, but without RT, However it is unclear how GPU friendly it is, and would probably at the very least need Compute

POC.zip (made in current stable)

Idea Behind it:

1. Get Depth Texture rendered by the Camera, and then use it to get what point in the world each pixel corresponds to (roughly at least)
2. Create a Depth Test for all of the points obtained in step 1, from the Directional Light's point of view
3. For every point, if the Depth Test shows that it is not the closest to the Directional Light's point of view, the point is under shadow, because only the lowest depth (closest to camera) are not occluded, any higher depths are occluded by the point with the lowest depth
4. Render Final Image, and apply the Shadow data

Proof of Concept Notes:
Sphere is the occluder
Position3D is the position of a tested Pixel
Simplified - Directional light is such that Global Z is Depth, which doesn't really matter as the test for Position3D doesn't test depth, just that it is occluded in 2D by a triangle of the sphere, so if the Position 3D is moved in Global Z, it will not affect the result, and as such the result will print true if in XY Position3D is inside the sphere, and false if it is not, further more the Sphere is taking non transformed triangles in the test, so moving, rotating and scaling the sphere in the editor will not cause any differences to the outcome, only changing the mesh itself will.

This means the things that aren't done that need to be for real world application are:

1. No Depth Testing, just all projected onto a plane (so like Z Buffer being just 1 plane, not more)
2. No transformations are applied (Light comes as shown by the Directional Light Node, modifications to Sphere such as move, rotate and scale will have no effect on everything, because the vertices aren't transformed in any way)
3. Lazy 3d-2d projection transform, moving Vector3 to Vector2 (the Light Plane) is just removing Z, this is why it's essentially just testing if Position3D is inside the sphere on the XY axis and moving it in Z won't do anything because it's all just tested on 1 plane
4. Rendering, it just says if the Position 3D would be in shadow, so if it is or isn't inside the Sphere in XY coordinates, nothing else, and project has to be re-ran every time you want to preform a test

This would likely outperform Ray Traced directional Shadows in the OP because this is just 2 Depth Rasterizations, one being the Camera, and other being the Directional Light for all the points the 1st Depth map provides, and would result with the same thing as the OP desired, and would also possibly allow for having Soft shadows by in the 2nd Depth map rendering, by depth testing around the point it's testing.

[mrjustaguy]

seems like duplicate of #3073, and figured out how to get Directional Shadows to look fairly good and RT Should probably be discussed on a broader level (Shadows, Reflections, AO, things like that), with a higher quality proposal.