Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement minimal reflection probes. #10057

Merged
merged 52 commits into from
Jan 8, 2024

Conversation

pcwalton
Copy link
Contributor

@pcwalton pcwalton commented Oct 9, 2023

Objective

This pull request implements reflection probes, which generalize environment maps to allow for multiple environment maps in the same scene, each of which has an axis-aligned bounding box. This is a standard feature of physically-based renderers and was inspired by the corresponding feature in Blender's Eevee renderer.

Solution

This is a minimal implementation of reflection probes that allows artists to define cuboid bounding regions associated with environment maps. For every view, on every frame, a system builds up a list of the nearest 4 reflection probes that are within the view's frustum and supplies that list to the shader. The PBR fragment shader searches through the list, finds the first containing reflection probe, and uses it for indirect lighting, falling back to the view's environment map if none is found. Both forward and deferred renderers are fully supported.

A reflection probe is an entity with a pair of components, LightProbe and EnvironmentMapLight (as well as the standard SpatialBundle, to position it in the world). The LightProbe component (along with the Transform) defines the bounding region, while the EnvironmentMapLight component specifies the associated diffuse and specular cubemaps.

A frequent question is "why two components instead of just one?" The advantages of this setup are:

  1. It's readily extensible to other types of light probes, in particular irradiance volumes (also known as ambient cubes or voxel global illumination), which use the same approach of bounding cuboids. With a single component that applies to both reflection probes and irradiance volumes, we can share the logic that implements falloff and blending between multiple light probes between both of those features.

  2. It reduces duplication between the existing EnvironmentMapLight and these new reflection probes. Systems can treat environment maps attached to cameras the same way they treat environment maps applied to reflection probes if they wish.

Internally, we gather up all environment maps in the scene and place them in a cubemap array. At present, this means that all environment maps must have the same size, mipmap count, and texture format. A warning is emitted if this restriction is violated. We could potentially relax this in the future as part of the automatic mipmap generation work, which could easily do texture format conversion as part of its preprocessing.

An easy way to generate reflection probe cubemaps is to bake them in Blender and use the export-blender-gi tool that's part of the bevy-baked-gi project. This tool takes a .blend file containing baked cubemaps as input and exports cubemap images, pre-filtered with an embedded fork of the glTF IBL Sampler, alongside a corresponding .scn.ron file that the scene spawner can use to recreate the reflection probes.

Note that this is intentionally a minimal implementation, to aid reviewability. Known issues are:

  • Reflection probes are basically unsupported on WebGL 2, because WebGL 2 has no cubemap arrays. (Strictly speaking, you can have precisely one reflection probe in the scene if you have no other cubemaps anywhere, but this isn't very useful.)

  • Reflection probes have no falloff, so reflections will abruptly change when objects move from one bounding region to another.

  • As mentioned before, all cubemaps in the world of a given type (diffuse or specular) must have the same size, format, and mipmap count.

Future work includes:

  • Blending between multiple reflection probes.

  • A falloff/fade-out region so that reflected objects disappear gradually instead of vanishing all at once.

  • Irradiance volumes for voxel-based global illumination. This should reuse much of the reflection probe logic, as they're both GI techniques based on cuboid bounding regions.

  • Support for WebGL 2, by breaking batches when reflection probes are used.

These issues notwithstanding, I think it's best to land this with roughly the current set of functionality, because this patch is useful as is and adding everything above would make the pull request significantly larger and harder to review.


Changelog

Added

  • A new LightProbe component is available that specifies a bounding region that an EnvironmentMapLight applies to. The combination of a LightProbe and an EnvironmentMapLight offers reflection probe functionality similar to that available in other engines.

Current problems:

* There's no logic yet to determine which reflection probe to apply to
  each mesh; it always takes the first one.

* Reflection probes are regenerated on every frame.
@pcwalton pcwalton marked this pull request as draft October 9, 2023 05:29
@alice-i-cecile alice-i-cecile added C-Feature A new feature, making something new possible A-Rendering Drawing game state to the screen labels Oct 9, 2023
@pcwalton
Copy link
Contributor Author

Both issues have been fixed. I want to improve the example before submitting this for review. This requires messing with the glTF IBL Sampler some more.

@github-actions
Copy link
Contributor

The generated examples/README.md is out of sync with the example metadata in Cargo.toml or the example readme template. Please run cargo run -p build-templated-pages -- update examples to update it, and commit the file change.

2 similar comments
@github-actions
Copy link
Contributor

The generated examples/README.md is out of sync with the example metadata in Cargo.toml or the example readme template. Please run cargo run -p build-templated-pages -- update examples to update it, and commit the file change.

@github-actions
Copy link
Contributor

The generated examples/README.md is out of sync with the example metadata in Cargo.toml or the example readme template. Please run cargo run -p build-templated-pages -- update examples to update it, and commit the file change.

@github-actions
Copy link
Contributor

The generated examples/README.md is out of sync with the example metadata in Cargo.toml or the example readme template. Please run cargo run -p build-templated-pages -- update examples to update it, and commit the file change.

@github-actions
Copy link
Contributor

The generated examples/README.md is out of sync with the example metadata in Cargo.toml or the example readme template. Please run cargo run -p build-templated-pages -- update examples to update it, and commit the file change.

@pcwalton pcwalton changed the title Implement reflection probes. (WIP) Implement reflection probes. Oct 15, 2023
@pcwalton pcwalton changed the title Implement reflection probes. Implement minimal reflection probes. Oct 15, 2023
@github-actions
Copy link
Contributor

The generated examples/README.md is out of sync with the example metadata in Cargo.toml or the example readme template. Please run cargo run -p build-templated-pages -- update examples to update it, and commit the file change.

@pcwalton pcwalton marked this pull request as ready for review October 15, 2023 05:11
@pcwalton
Copy link
Contributor Author

This is ready for review.

@pcwalton
Copy link
Contributor Author

pcwalton commented Jan 2, 2024

For both the view cubemap and any light probes, if the mesh has a lightmap, we should probably ignore the diffuse lighting from cubemaps. Otherwise, we'd double count it.

Yep, now that lightmaps have landed I made that change.

@JMS55
Copy link
Contributor

JMS55 commented Jan 3, 2024

Nvm, I missed that you were using textureNumLevels for the light probes.

@JMS55 JMS55 self-requested a review January 3, 2024 01:58
@alice-i-cecile alice-i-cecile added the S-Ready-For-Final-Review This PR has been approved by the community. It's ready for a maintainer to consider merging it label Jan 4, 2024
@alice-i-cecile alice-i-cecile added this pull request to the merge queue Jan 8, 2024
Merged via the queue into bevyengine:main with commit 54a943d Jan 8, 2024
27 checks passed
@mockersf
Copy link
Member

mockersf commented Jan 9, 2024

This broke rendering on macOS. Example load_gltf panics with

thread 'Compute Task Pool (2)' panicked at .cargo/registry/src/index.crates.io-6f17d22bba15001f/naga-0.14.2/src/back/msl/writer.rs:229:21:
internal error: entered unreachable code: metal requires all arrays be constant sized

so https://github.com/gfx-rs/wgpu/blob/4400a5847080d1164bdca93a90622414963ed9f3/naga/src/back/msl/writer.rs#L228-L228

@mockersf
Copy link
Member

mockersf commented Jan 9, 2024

it also broke iOS:

wgpu error: Validation Error

Caused by:
    In Device::create_bind_group_layout
      note: label = `mesh_view_layout`
    Binding 14 entry is invalid
    Features Features(TEXTURE_BINDING_ARRAY) are required but not enabled on the device

@mockersf
Copy link
Member

mockersf commented Jan 9, 2024

and Android:

wgpu error: Validation Error
Caused by:
    In Device::create_bind_group_layout
      note: label = `mesh_view_layout`
    Too many bindings of type SampledTextures in Stage ShaderStages(FRAGMENT), limit is 16, count was 21

mockersf added a commit to mockersf/bevy that referenced this pull request Jan 11, 2024
github-merge-queue bot pushed a commit that referenced this pull request Jan 12, 2024
# Objective

- Fix working on macOS, iOS, Android on main 
- Fixes #11281 
- Fixes #11282 
- Fixes #11283 
- Fixes #11299

## Solution

- Revert #10057
Ixentus pushed a commit to Ixentus/bevy that referenced this pull request Jan 12, 2024
…yengine#11307)

# Objective

- Fix working on macOS, iOS, Android on main 
- Fixes bevyengine#11281 
- Fixes bevyengine#11282 
- Fixes bevyengine#11283 
- Fixes bevyengine#11299

## Solution

- Revert bevyengine#10057
github-merge-queue bot pushed a commit that referenced this pull request Jan 19, 2024
…11366)

This pull request re-submits #10057, which was backed out for breaking
macOS, iOS, and Android. I've tested this version on macOS and Android
and on the iOS simulator.

# Objective

This pull request implements *reflection probes*, which generalize
environment maps to allow for multiple environment maps in the same
scene, each of which has an axis-aligned bounding box. This is a
standard feature of physically-based renderers and was inspired by [the
corresponding feature in Blender's Eevee renderer].

## Solution

This is a minimal implementation of reflection probes that allows
artists to define cuboid bounding regions associated with environment
maps. For every view, on every frame, a system builds up a list of the
nearest 4 reflection probes that are within the view's frustum and
supplies that list to the shader. The PBR fragment shader searches
through the list, finds the first containing reflection probe, and uses
it for indirect lighting, falling back to the view's environment map if
none is found. Both forward and deferred renderers are fully supported.

A reflection probe is an entity with a pair of components, *LightProbe*
and *EnvironmentMapLight* (as well as the standard *SpatialBundle*, to
position it in the world). The *LightProbe* component (along with the
*Transform*) defines the bounding region, while the
*EnvironmentMapLight* component specifies the associated diffuse and
specular cubemaps.

A frequent question is "why two components instead of just one?" The
advantages of this setup are:

1. It's readily extensible to other types of light probes, in particular
*irradiance volumes* (also known as ambient cubes or voxel global
illumination), which use the same approach of bounding cuboids. With a
single component that applies to both reflection probes and irradiance
volumes, we can share the logic that implements falloff and blending
between multiple light probes between both of those features.

2. It reduces duplication between the existing *EnvironmentMapLight* and
these new reflection probes. Systems can treat environment maps attached
to cameras the same way they treat environment maps applied to
reflection probes if they wish.

Internally, we gather up all environment maps in the scene and place
them in a cubemap array. At present, this means that all environment
maps must have the same size, mipmap count, and texture format. A
warning is emitted if this restriction is violated. We could potentially
relax this in the future as part of the automatic mipmap generation
work, which could easily do texture format conversion as part of its
preprocessing.

An easy way to generate reflection probe cubemaps is to bake them in
Blender and use the `export-blender-gi` tool that's part of the
[`bevy-baked-gi`] project. This tool takes a `.blend` file containing
baked cubemaps as input and exports cubemap images, pre-filtered with an
embedded fork of the [glTF IBL Sampler], alongside a corresponding
`.scn.ron` file that the scene spawner can use to recreate the
reflection probes.

Note that this is intentionally a minimal implementation, to aid
reviewability. Known issues are:

* Reflection probes are basically unsupported on WebGL 2, because WebGL
2 has no cubemap arrays. (Strictly speaking, you can have precisely one
reflection probe in the scene if you have no other cubemaps anywhere,
but this isn't very useful.)

* Reflection probes have no falloff, so reflections will abruptly change
when objects move from one bounding region to another.

* As mentioned before, all cubemaps in the world of a given type
(diffuse or specular) must have the same size, format, and mipmap count.

Future work includes:

* Blending between multiple reflection probes.

* A falloff/fade-out region so that reflected objects disappear
gradually instead of vanishing all at once.

* Irradiance volumes for voxel-based global illumination. This should
reuse much of the reflection probe logic, as they're both GI techniques
based on cuboid bounding regions.

* Support for WebGL 2, by breaking batches when reflection probes are
used.

These issues notwithstanding, I think it's best to land this with
roughly the current set of functionality, because this patch is useful
as is and adding everything above would make the pull request
significantly larger and harder to review.

---

## Changelog

### Added

* A new *LightProbe* component is available that specifies a bounding
region that an *EnvironmentMapLight* applies to. The combination of a
*LightProbe* and an *EnvironmentMapLight* offers *reflection probe*
functionality similar to that available in other engines.

[the corresponding feature in Blender's Eevee renderer]:
https://docs.blender.org/manual/en/latest/render/eevee/light_probes/reflection_cubemaps.html

[`bevy-baked-gi`]: https://github.com/pcwalton/bevy-baked-gi

[glTF IBL Sampler]: https://github.com/KhronosGroup/glTF-IBL-Sampler
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-Rendering Drawing game state to the screen C-Feature A new feature, making something new possible S-Ready-For-Final-Review This PR has been approved by the community. It's ready for a maintainer to consider merging it
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

7 participants