Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add depth unprojection support to batch renderer #2129

Merged
merged 34 commits into from
Jun 25, 2023

Conversation

0mdc
Copy link
Contributor

@0mdc 0mdc commented Jun 13, 2023

Motivation and Context

This changeset adds support for unprojecting depth within the batch renderer.

In this context, unprojected depth means that each pixel of the depth image represents the distance in meters from the camera, rather than the raw depth buffer.

To accomplish this, DepthUnprojection is now shared by the classic and the batch renderer. To preserve batch renderer isolation, it was moved to the batch renderer along with the shaders.

Summary:

  • Batch renderer supports CPU depth unprojection.
  • Shaders are now split between gfx and gfx_batch.
  • gfx shaders can reference gfx_batch shaders, but not the other way around.
  • DepthUnprojection is moved to gfx_batch

Screenshot_20230613_111905

How Has This Been Tested

Tested locally and on CI.
The following tests are added:

  • Unprojection test in GfxBatchRendererTest
  • Image-based depth sensor test in BatchReplayRendererTest

Types of changes

  • Docs change / refactoring / dependency upgrade
  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)

Checklist

  • My code follows the code style of this project.
  • My change requires a change to the documentation.
  • I have updated the documentation accordingly.
  • I have read the CONTRIBUTING document.
  • I have completed my CLA (see CONTRIBUTING)
  • I have added tests to cover my changes.
  • All new and existing tests passed.

@facebook-github-bot facebook-github-bot added the CLA Signed Do not delete this pull request or issue due to inactivity. label Jun 13, 2023
src/tests/CMakeLists.txt Outdated Show resolved Hide resolved
@0mdc 0mdc mentioned this pull request Jun 13, 2023
11 tasks
@0mdc 0mdc marked this pull request as ready for review June 14, 2023 01:45
Copy link
Collaborator

@mosra mosra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

22 comments, sorry, I'll stop now 😅

src/esp/gfx/CMakeLists.txt Outdated Show resolved Hide resolved
src/esp/gfx/RenderTarget.h Show resolved Hide resolved
src/esp/gfx_batch/CMakeLists.txt Outdated Show resolved Hide resolved
/* Combined view and projection matrices. Updated from updateCamera() */
Cr::Containers::Array<ProjectionPadded> cameraMatrices;
/* Unprojection for cameras. Updated from updateCamera() */
Cr::Containers::Array<Mn::Vector2> cameraUnprojections;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Couldn't this be put into the Scene struct among all other per-scene data? Feels unnecessary to allocate a dedicated array for these.

Copy link
Contributor Author

@0mdc 0mdc Jun 19, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm inclined to leave this as-is until we have multi-sensor support because it'll be more straightforward to refactor per-sensor data.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it'll be more straightforward to refactor

I don't know :) There most probably will be another array, yes, but that's about it, its size and contents will be significantly different. In the context of this PR it's unnecessary noise.

src/tests/CMakeLists.txt Outdated Show resolved Hide resolved
src/tests/BatchReplayRendererTest.cpp Outdated Show resolved Hide resolved
src/esp/gfx_batch/Renderer.h Outdated Show resolved Hide resolved
src/esp/gfx_batch/Renderer.h Outdated Show resolved Hide resolved
src/esp/gfx_batch/Renderer.cpp Outdated Show resolved Hide resolved
CONTRIBUTING.md Outdated Show resolved Hide resolved
src/esp/gfx/CMakeLists.txt Outdated Show resolved Hide resolved
src/esp/gfx_batch/Renderer.h Outdated Show resolved Hide resolved
src/shaders/gfx/Shaders.conf Outdated Show resolved Hide resolved
src/tests/GfxBatchRendererTest.cpp Outdated Show resolved Hide resolved
Copy link
Collaborator

@mosra mosra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Final set of code golf suggestions because this was bothering me :P

src/tests/GfxBatchRendererTest.cpp Outdated Show resolved Hide resolved
src/tests/GfxBatchRendererTest.cpp Outdated Show resolved Hide resolved
src/tests/GfxBatchRendererTest.cpp Outdated Show resolved Hide resolved
src/tests/GfxBatchRendererTest.cpp Outdated Show resolved Hide resolved
src/tests/GfxBatchRendererTest.cpp Outdated Show resolved Hide resolved
@mosra mosra force-pushed the batch-renderer-depth-unprojection branch from 1ae1473 to b100aad Compare June 25, 2023 10:51
@mosra
Copy link
Collaborator

mosra commented Jun 25, 2023

Did the two remaining requested changes myself, see the commits below for details.

@mosra mosra force-pushed the batch-renderer-depth-unprojection branch 2 times, most recently from 58f56ed to ec15aac Compare June 25, 2023 11:32
My suggestion was meant to *replace* the original and adapt the code as
necessary, not add a new overload without actually testing any of it.

Due to the copypaste the new overload also had no FMV attribute applied,
meaning it suddenly got significantly slower for no reason.
@mosra mosra force-pushed the batch-renderer-depth-unprojection branch from ec15aac to 46b8286 Compare June 25, 2023 11:44
@0mdc 0mdc merged commit 181aea8 into main Jun 25, 2023
@0mdc 0mdc deleted the batch-renderer-depth-unprojection branch June 25, 2023 16:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CLA Signed Do not delete this pull request or issue due to inactivity.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants