Tracking issue: Parallax mapping improvements

[nicopap]

This issue tracks the progress on the "Future works" section of the parallax mapping PR #5928.

Except distance fading and depth buffer update, bevy's parallax mapping implementation is on par with similar features from other general game engines. But it is far from being state of the art.

Contributing

Want to check off one of those checkboxes? Nothing easier!

1. Open a feature request with, as title, "Add <feature> for parallax mapping", you can copy/paste the description if you feel lazy!
2. Implement the feature and open a PR!

All related issues will be listed in this tracking issue.

Low hanging fruits

Some features weren't added in the original parallax mapping PR in order to avoid scope creep, not because they are particularly hard to implement. Those are low hanging fruits that could be implemented.

• Distance fading: user-configurable limit on the view range in which parallax mapping is applied. Parallaxing doesn't make sense on distant objects as the difference of view position is small enough to be insignificant, yet it still incurs a significant cost. This video motivates distance fading.
• Improved mipmap sampling: Mipmapping is another way to reduce perf cost of parallaxing on distant materials. The current implementation chooses to sample at LOD level 0 for performance and practicality. It also works around the naga default Dx12 compiler. However, there is nothing preventing us from pre-computing a gradient and using this set gradient when sampling the depth map. This paper discusses the perf characteristics of various sampling methods in POM.
• Offset limiting method: This is the ancestor of steep parallax mapping. It is not as accurate, but has the advantage of requiring no iteration, only a simple sampling of the depth map. Therefore, it is a great method to use for weak devices. It is described in the original article used as reference for the implementation and an original paper.

Important but requires moderate efforts

Those are features that would be really nice to have but are not implemented because it would require a somewhat consequent time investment.

• Update the depth buffer: Updating the depth buffer according to the depth map would allow accurate silhouetting in intersecting geometry. it would also allow shadows cast into and from parallaxed surfaces to follow the depth-mapped surface, rather than just the mesh surface. However, it's both costly (need to run parallaxing for all source of light!) and error prone (need to compute the additional depth relative to viewport). WIP: Update depth buffer with parallax mapping result #8486
• Depth maps glTF extension: We currently do not load depth maps from glTF because glTF doesn't have such a feature! A proposal is open but seems heavily stalled. However, glTF is extensible and you can add a depth map through the extras field on materials. This would require:
  1. Changing the glTF loading code from bevy_gltf to support that extension.
  2. Defining the extension formally so that it's possible to modify authoring software to include the depth information in the glTFs it exports (say for example through the blender gltf-io plugin).
  3. Modify at least one authoring tool to support such an export option.
• Generate the depth maps from normal maps: A work around the lack of depth map is to simply deduce it from the normal map. Implemented in https://github.com/devildahu/n2dmap.

More complex but fun improvements

Finally, we are left with the "state of the art" features. The last improvements in parallax mapping. Although, since parallax mapping is an old and tried algorithm, "state of the art" is already quite old. Regardless, implementing those would elevate bevy's implementation of parallax mapping as one of the very best.

• Cone mapping / relaxed cone stepping method: This GPU gems article explains a variant of parallax mapping using additional information stored in the depth map texture to greatly reduce the cost of the initial ray-marching step, and improved accuracy. It is known to be used in SimCity 2013. It seems not that hard to implement, however would require pre-processing the depth map asset.
• Ray-quadratic distance: This barbaric name hides an extremely wishable feature: extending parallax mapping to curved surfaces. The current implementation doesn't play well with curvature. By adding quadratic surface parameter attributes to mesh vertices, it is possible to compute a more accurate ray that account for mesh surfaces and allow "missing" the material when the parallaxed surface is not hit, for true silhouetting. This paper describes in detail the method.
• Quadtree displacement mapping method: is another parallax mapping variant that aims to reduce the ray-marching steps, based on carefully built mipmaps. However, the performance gains are contested. It is discussed in this series of slides.

[nicopap]

• Fix pbr shader breaking on missing UVs #8412