GenerateEnvironmentMapLight #9414

JMS55 · 2023-08-10T20:00:35Z

Objective

Closes Automatic Skybox -> EnvironmentMapLight generation #9380

Solution

Adds the GenerateEnvironmentMapLight component.
Implements "Fast Filtering of Reflection Probes" for the specular filtering.
- Special thanks to Godot - their implementation was an extremely valuable reference.
Implements cosine-weighted hemisphere importance sampling for the diffuse filtering.

Changelog

Added GenerateEnvironmentMapLight for automatically generating an EnvironmentMapLight component from a Skybox component. This can be used instead of KhronosGroup's glTF-IBL-Sampler.

github-actions · 2023-08-10T20:12:54Z

Example no_renderer failed to run, please try running it locally and check the result.

github-actions · 2023-08-12T06:55:10Z

Example no_renderer failed to run, please try running it locally and check the result.

github-actions · 2023-08-12T07:32:02Z

Example no_renderer failed to run, please try running it locally and check the result.

github-actions · 2023-08-13T03:30:46Z

Example no_renderer failed to run, please try running it locally and check the result.

github-actions · 2023-08-13T04:50:59Z

Example no_renderer failed to run, please try running it locally and check the result.

github-actions · 2023-08-13T04:53:44Z

Example no_renderer failed to run, please try running it locally and check the result.

superdump · 2023-11-28T22:34:36Z

Is filter_coefficents.rs still needed now that the coefficients have been serialised into filter_coefficents.bin?

Also, coefficents -> coefficients.

JMS55 · 2023-11-28T23:21:11Z

The rust file is not needed, no. It's unused, but I left it as a reference. I can remove it if we're ok with an opaque bin file. Up to you/cart.

Also damn, I thought I had double checked my spelling :(

pcwalton

I haven't fully digested the guts of the shader yet but here are a bunch of comments.

pcwalton · 2023-12-05T02:19:34Z

crates/bevy_pbr/src/environment_map/diffuse_convolution.wgsl

+        case 0u: { return vec3(1.0, v, -u); }
+        case 1u: { return vec3(-1.0, v, u); }
+        case 2u: { return vec3(u, 1.0, -v); }
+        case 3u: { return vec3(u, -1.0, v); }


These two are flipped compared to https://github.com/KhronosGroup/glTF-IBL-Sampler/blob/master/lib/source/shaders/filter.frag, is that intentional?

I used the same get_dir function from https://www.activision.com/cdn/research/filter_using_table_128.txt that I used in the other shaders

pcwalton · 2023-12-05T02:37:38Z

crates/bevy_pbr/src/environment_map/generate_from_skybox.rs

+/// * The first frame this component is added to the skybox entity, an [`EnvironmentMapLight`]
+/// component will be generated and added to the skybox entity.


You could just say "this is generally a quick operation".

crates/bevy_pbr/src/environment_map/generate_from_skybox.rs

crates/bevy_pbr/src/environment_map/filter.wgsl

pcwalton · 2023-12-05T05:44:32Z

crates/bevy_pbr/src/environment_map/filter.wgsl

+    var color = vec4(0.0);
+    for (var axis = 0u; axis < 3u; axis++) {
+        let other_axis0 = 1u - (axis & 1u) - (axis >> 1u);
+        let other_axis1 = 2u - (axis >> 1u);


I feel like if you're switching over axis below anyway you might as well just set other_axis0 and other_axis1 explicitly in a switch statement instead of using these bit tricks.

pcwalton · 2023-12-05T05:45:50Z

crates/bevy_pbr/src/environment_map/diffuse_convolution.wgsl

+    var color = vec3(0.0);
+    for (var sample_i = 0u; sample_i < 32u; sample_i++) {
+        // R2 sequence - http://extremelearning.com.au/unreasonable-effectiveness-of-quasirandom-sequences
+        let r = fract(0.5 + f32(sample_i) * vec2<f32>(0.75487766624669276005, 0.5698402909980532659114));


Factor this out into a rand function :)

It's 1 line, I'm inclined not to 😅

crates/bevy_pbr/src/environment_map/diffuse_convolution.wgsl

pcwalton

LGTM

JMS55 · 2023-12-29T20:04:06Z

Diffuse filtering is suffering from not enough samples, and fireflys. I should look at https://www.shadertoy.com/view/4c2GRh.

atlv24 · 2023-12-30T03:15:24Z

crates/bevy_pbr/src/environment_map/diffuse_convolution.wgsl

+        // R2 sequence - http://extremelearning.com.au/unreasonable-effectiveness-of-quasirandom-sequences
+        let r = fract(0.5 + f32(sample_i) * vec2<f32>(0.75487766624669276005, 0.5698402909980532659114));
+
+        let cos_theta = sqrt(1.0 - f32(r.y));


Suggested change

let cos_theta = sqrt(1.0 - f32(r.y));

// map uniformly distributed [0..1)^2 into hemisphere with cosine importance (Lambertian distribution)

let cos_theta = sqrt(1.0 - f32(r.y));

atlv24 · 2023-12-30T03:29:26Z

crates/bevy_pbr/src/environment_map/filter.wgsl

+fn main(@builtin(global_invocation_id) global_id: vec3<u32>) {
+    var id = global_id;
+    var level = 0u;
+    if id.x < 128u * 128u {


I think it has to do the biggest mip first for other reasons, maybe because all the mips after it depend on it. but wgsl workgroups are not guaranteed to run in any particular order so idk

atlv24 · 2023-12-30T03:53:13Z

crates/bevy_pbr/src/environment_map/filter.wgsl

+    id.x -= id.y * res;
+
+    let u = (f32(id.x) * 2.0 + 1.0) / f32(res) - 1.0;
+    let v = -(f32(id.y) * 2.0 + 1.0) / f32(res) + 1.0;


it maps to the centers of every other 2x2 texel patch, because the lower mip level must represent those texels as one larger one that covers the area of all 4 original ones.

atlv24 · 2023-12-30T03:58:13Z

crates/bevy_pbr/src/environment_map/filter.wgsl

+    id.y = id.x / res;
+    id.x -= id.y * res;
+
+    let u = (f32(id.x) * 2.0 + 1.0) / f32(res) - 1.0;


Suggested change

let u = (f32(id.x) * 2.0 + 1.0) / f32(res) - 1.0;

// remap integers [0..res-1]^2 to the centers of every 2x2 texel patch we are mipping down to one texel, in (-1..1)^2

let u = (f32(id.x) * 2.0 + 1.0) / f32(res) - 1.0;

JMS55 · 2023-12-30T04:24:59Z

This PR has artifacts unfortunately. It can't be merged as-is.

atlv24 · 2023-12-30T17:34:56Z

crates/bevy_pbr/src/environment_map/filter.wgsl

+
+    id.z = id.y;
+    let res = 128u >> level;
+    id.y = id.x / res;


could save an integer division here with

Suggested change

id.y = id.x / res;

id.y = id.x >> (7 - level);

This commit introduces a new type of camera, the *omnidirectional* camera. These cameras render to a cubemap texture, and as such extract into six different cameras in the render world, one for each side. The cubemap texture can be attached to a reflection probe as usual, which allows reflections to contain moving objects. To use an omnidirectional camera, create an [`OmnidirectionalCameraBundle`]. Because omnidirectional cameras extract to six different sub-cameras in the render world, render world extraction code that targets components present on cameras now needs to be aware of this fact and extract components to the individual sub-cameras, not the root camera component. They also need to run after omnidirectional camera extraction, as only then will the sub-cameras be present in the render world. New plugins, `ExtractCameraComponentPlugin` and `ExtractCameraInstancesPlugin`, are available to assist with this. Each side of an omnidirectional camera can be individually marked as active via the `ActiveCubemapSides` bitfield. This allows for the common technique of rendering only one (or two, or three) sides of the cubemap per frame, to reduce rendering overhead. It also allows for on-demand rendering, so that an application that wishes to optimize further can choose sides to refresh. For example, an application might wish to only rerender sides whose frusta contain moving entities. In addition to real-time reflection probes, this patch introduces much of the infrastructure necessary to support baking reflection probes from within Bevy as opposed to in an external program such as Blender, which has been the status quo up to this point. Even with this patch, there are still missing pieces needed to make this truly convenient, however: 1. Baking a reflection probe requires more than just saving a cubemap: it requires pre-filtering the cubemap into diffuse and specular parts in the same way that the [glTF IBL Sampler] does. This is not yet implemented in Bevy; see bevyengine#9414 for a previous attempt. 2. The cubemap needs to be saved in `.ktx2` format, as that's the only format that Bevy presently knows how to load. There's no comprehensive Rust crate for this, though note that my [glTF IBL Sampler UI] has code to do it for the specific case of cubemaps. 3. An editor UI is necessary for convenience, as otherwise every application will have to create some sort of bespoke tool that arranges scenes and saves the reflection cubemaps. The `reflection_probes` example has been updated in order to add an option to enable dynamic reflection probes, as well as an option to spin the cubes so that the impact of the dynamic reflection probes is visible. Additionally, the static reflection probe, which was previously rendered in Blender, has been changed to one rendered in Bevy. This results in a change in appearance, as Blender and Bevy render somewhat differently. Partially addresses bevyengine#12233. [glTF IBL Sampler]: https://github.com/KhronosGroup/glTF-IBL-Sampler [glTF IBL Sampler UI]: https://github.com/pcwalton/gltf-ibl-sampler-egui

JMS55 · 2024-11-08T07:21:46Z

godotengine/godot#43763 (comment)

# Objective This PR implements a robust GPU-based pipeline for dynamically generating environment maps in Bevy. It builds upon PR #19037, allowing these changes to be evaluated independently from the atmosphere implementation. While existing offline tools can process environment maps, generate mip levels, and calculate specular lighting with importance sampling, they're limited to static file-based workflows. This PR introduces a real-time GPU pipeline that dynamically generates complete environment maps from a single cubemap texture on each frame. Closes #9380 ## Solution Implemented a Single Pass Downsampling (SPD) pipeline that processes textures without pre-existing mip levels or pre-filtered lighting data. Single Pass Downsampling (SPD) pipeline: - accepts any square, power-of-two cubemap up to 8192 × 8192 per face and generates the complete mip chain in one frame; - copies the base mip (level 0) in a dedicated compute dispatch (`copy_mip0`) before the down-sampling pass; - performs the down-sampling itself in two compute dispatches to fit within subgroup limits; - heavily inspired by Jasmine's prototype code. Pre-filtering pipeline: - generates the specular Radiance Map using bounded-VNDF GGX importance sampling for higher quality highlights and fewer fireflies; - computes the diffuse Irradiance Map with cosine-weighted hemisphere sampling; - mirrors the forward-/reverse-tonemap workflow used by TAA instead of exposing a separate *white-point* parameter; - is based on the resources below together with the “Bounded VNDF Sampling for Smith-GGX Reflections” paper. The pre-filtering pipeline is largely based on these articles: - https://placeholderart.wordpress.com/2015/07/28/implementation-notes-runtime-environment-map-filtering-for-image-based-lighting/ - https://bruop.github.io/ibl/ - https://gpuopen.com/download/Bounded_VNDF_Sampling_for_Smith-GGX_Reflections.pdf > The forward-/reverse-tonemap trick removes almost all fireflies without the need for a separate white-point parameter. Previous work: #9414 ## Testing The `reflection_probes.rs` example has been updated: - The camera starts closer to the spheres so the reflections are easier to see. - The GLTF scene is spawned only when the reflection probe mode is active (press Space). - The third display mode (toggled with Space) shows the generated cubemap chain. - You can change the roughness of the center sphere with the Up/Down keys. ## Render Graph Composed of two nodes and a graph edge: ``` Downsampling -> Filtering ``` Pass breakdown: ``` dowsampling_first_pass -> dowsampling_second_pass -> radiance_map_pass -> irradiance_map_pass ``` <img width="1601" height="2281" alt="render-graph" src="https://github.com/user-attachments/assets/3c240688-32f7-447a-9ede-6050b77c0bd1" /> --- ## Showcase <img width="2564" height="1500" alt="image" src="https://github.com/user-attachments/assets/56e68dd7-9488-4d35-9bba-7f713a3e2831" /> User facing API: ```rust commands.entity(camera) .insert(GeneratedEnvironmentMapLight { environment_map: world.load_asset("environment_maps/pisa_specular_rgb9e5_zstd.ktx2"), ..default() }); ``` ## Computed Environment Maps To use fully dynamic environment maps, create a new placeholder image handle with `Image::new_fill`, extract it to the render world. Then dispatch a compute shader, bind the image as a 2d array storage texture. Anything can be rendered to the custom dynamic environment map. This is already demonstrated in PR #19037 with the `atmosphere.rs` example. We can extend this idea further and run the entire PBR pipeline from the perspective of the light probe, and it is possible to have some form of global illumination or baked lighting information this way, especially if we make use of irradiance volumes for the realtime aspect. This method could very well be extended to bake indirect lighting in the scene. #13840 should make this possible! ## Notes for reviewers This PR no longer bundles any large test textures. --------- Co-authored-by: atlas <email@atlasdostal.com>

JMS55 added 15 commits August 7, 2023 22:04

WIP

cd3e90f

Rename EnvironmentMapPlugin to EnvironmentMapLightPlugin

f682078

Small WIP for shader setup

186283b

WIP: Add most of the bind group setup

a6ad58d

WIP: Add node

5608501

Export GenerateEnvironmentMapLight

fd955e9

More WIP

212ec68

Rename

22573cf

Small fixes

fe75bba

Small fix

cf9b330

WIP

3ebbbdb

Add further attribution

368272d

Bunch of fixes

53feb96

Port downsample shader

a92ab3c

Misc

b0b23b3

JMS55 added C-Feature A new feature, making something new possible A-Rendering Drawing game state to the screen labels Aug 10, 2023

JMS55 added this to the 0.12 milestone Aug 10, 2023

Misc

676c207

WIP: Translate filter.wgsl

2572f1c

Finish filter.wgsl

71c15b1

JMS55 added 2 commits August 12, 2023 21:38

Filter shader fixes

d07c6ee

Finish filter dispatches

71fcf74

Fix specular map size (must be 128x128)

848e92e

JMS55 mentioned this pull request Nov 28, 2023

Implement minimal reflection probes. #10057

Merged

JMS55 added 2 commits November 28, 2023 11:19

Merge commit 'a902ea6f85079777e61ea8627d86277c0d9abaf1' into envfilter

0b607e9

Use new BGL API

2dc5475

Remove rust filter coefficients, fix typo

2ffeec6

JMS55 requested a review from pcwalton December 3, 2023 18:27

pcwalton reviewed Dec 5, 2023

View reviewed changes

JMS55 added 4 commits December 14, 2023 10:20

Merge commit '11065974d487ca4b9680ae683a9d356f2f1b6c36' into envfilter

9fe7a9d

Fix rebase

4676948

Address PR feedback

82fe775

Address more CR feedback

eaf17d3

JMS55 requested review from IceSentry and pcwalton December 15, 2023 20:12

pcwalton approved these changes Dec 18, 2023

View reviewed changes

Merge commit '8067e46049f222d37ac394745805bad98979980f' into envfilter

719402d

atlv24 approved these changes Dec 30, 2023

View reviewed changes

atlv24 reviewed Dec 30, 2023

View reviewed changes

JMS55 marked this pull request as draft December 31, 2023 06:46

JMS55 removed this from the 0.13 milestone Jan 20, 2024

JMS55 mentioned this pull request Mar 1, 2024

Add a Bevy-native tool to generate (and bake) global illumination for light maps #12233

Open

JMS55 closed this Mar 29, 2024

pcwalton mentioned this pull request Jun 14, 2024

Implement omnidirectional cameras for real-time reflection probes. #13840

Open

mate-h mentioned this pull request May 5, 2025

Environment Map Filtering GPU pipeline #19076

Merged

		/// * The first frame this component is added to the skybox entity, an [`EnvironmentMapLight`]
		/// component will be generated and added to the skybox entity.

	let cos_theta = sqrt(1.0 - f32(r.y));
	// map uniformly distributed [0..1)^2 into hemisphere with cosine importance (Lambertian distribution)
	let cos_theta = sqrt(1.0 - f32(r.y));

	let u = (f32(id.x) * 2.0 + 1.0) / f32(res) - 1.0;
	// remap integers [0..res-1]^2 to the centers of every 2x2 texel patch we are mipping down to one texel, in (-1..1)^2
	let u = (f32(id.x) * 2.0 + 1.0) / f32(res) - 1.0;

Uh oh!

GenerateEnvironmentMapLight #9414

GenerateEnvironmentMapLight #9414

Uh oh!

Conversation

JMS55 commented Aug 10, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Objective

Solution

Changelog

Uh oh!

github-actions bot commented Aug 10, 2023

Uh oh!

github-actions bot commented Aug 12, 2023

Uh oh!

github-actions bot commented Aug 12, 2023

Uh oh!

github-actions bot commented Aug 13, 2023

Uh oh!

github-actions bot commented Aug 13, 2023

Uh oh!

github-actions bot commented Aug 13, 2023

Uh oh!

superdump commented Nov 28, 2023

Uh oh!

JMS55 commented Nov 28, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pcwalton left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JMS55 Dec 15, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

pcwalton left a comment

Choose a reason for hiding this comment

Uh oh!

JMS55 commented Dec 29, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JMS55 commented Dec 30, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JMS55 commented Nov 8, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

JMS55 commented Aug 10, 2023 •

edited

Loading

JMS55 commented Nov 28, 2023 •

edited

Loading

JMS55 Dec 15, 2023 •

edited

Loading