Fixing WideAngleCamera Lensflare issue #3276

sanjuksha · 2022-11-09T21:19:42Z

Signed-off-by: Sanjuksha Nirgude sanjuksha@gmail.com

🦟 Bug fix

Summary

Porting the same change for gazebo classic from gazebosim/gz-rendering#746 and a fix for wide angle camera not showing lensflare at certain angles. Wide Angle camera used to loop through all cameras to find the occlusion scale which resulted into not rendering lensfalre at certain angles. The fix removes the looping through all cameras and uses the camera facing the light source to calculate occlusion.

Checklist

Signed all commits for DCO
Added tests
Updated documentation (as needed)
Updated migration guide (as needed)
Consider updating Python bindings (if the library has them)
codecheck passed (See contributing)
All tests passed (See test coverage)
While waiting for a review on your PR, please help review another open pull request to support the maintainers

Results

gazebo/rendering/WideAngleCamera.cc

sanjuksha · 2022-11-11T20:56:31Z

Added one more change for Z check based on this comment #3273 (comment)

scpeters · 2022-11-16T23:08:34Z

cc @kjeppesen1 @mogumbo

gazebo/rendering/LensFlare.cc

kjeppesen1 · 2022-11-17T22:05:38Z

This fix works for me 👍 I moved around a pair of wideangle cameras at various angles and positions in a world with a DEM, and I no longer see the lens flare disappearing when it isn't supposed to.

gazebo/rendering/LensFlare.cc

scpeters · 2022-11-18T22:29:16Z

gazebo/rendering/LensFlare.cc

      double occlusionScale = 1.0;
      if (lightPos.z >= 0.0)
      {
-        // loop through all env cameras
-        for (auto cam : ogreEnvCameras)
-        {
          // project light world point to camera clip space.
          auto viewProj = cam->getProjectionMatrix() * cam->getViewMatrix();


replace cam with _wideAngleCam two times on this line

Hmm, _wideAngleCam doesn't have getProjectionMatrix() method here.

scpeters · 2022-11-18T22:29:39Z

gazebo/rendering/LensFlare.cc

-                this->dataPtr->lightWorldPos);
-            break;
+            ignition::math::Pose3d(
+            Conversions::ConvertIgn(cam->getDerivedPosition()),


replace cam with _wideAngleCam on this and the following line

Same, _wideAngleCam doesn't have getDerivedPosition() here

scpeters · 2022-11-18T22:30:14Z

gazebo/rendering/LensFlare.cc

+      std::vector<Ogre::Camera *> ogreEnvCameras = _wideAngleCam->OgreEnvCameras();
+
+      // set dummy camera properties based on env cam
+      Ogre::Camera *cam = ogreEnvCameras[0];


I think the cam and ogreEnvCameras variables can be removed if all instances of cam are replaced by _wideAngleCam

actually just checking if cam should be replaced by _wideAngleCam in the calls below. I remember I had to use cam (one of the env cameras) because it has the right properties for doing ray casting. Can you do a quick check and print out the viewport size, FOV, and aspect ratio for cam vs _wideAngleCam? I think they could be different.

that's a good point

more changes to review

scpeters · 2022-11-30T23:30:53Z

I've restarted the windows build

adityapande-1995

It would be nice to add an example world or a test which has the camera in various poses, showing when the lensflare is supposed to appear and when it isn't.

gazebo/rendering/WideAngleCamera.cc

scpeters · 2023-01-11T23:43:39Z

I cleaned up some of the code changes in 5740ce5 and adf7d1f and then made an adjustment to the new test in c63047c to make it easier to confirm the presence of the bug without the fix in this branch

scpeters · 2023-01-31T23:45:31Z

Thanks to some debugging help from @iche033, I added some additional cameras in d33593d and c92f948 to illustrate some problems with the occlusion checking in the test world added in this pull request

gazebo --verbose worlds/lensflare_wideangle_cam.world

Failure to occlude in `wide_angle_cameras_occluded_high` camera image

There is a box directly in front of the camera, so it should occlude but doesn't

Occluding when it shouldn't in `wide_angle_cameras_occluded_higher` camera image

There is a box to the right of the camera (partially visible), but it should not be occluding, but the lens flare is not there

It seems to be checking for occlusions in the wrong direction?

* Fixing Z check in WideAngleCamera::Project3d Signed-off-by: Sanjuksha Nirgude <sanjuksha@gmail.com> Added a test case for wideangle camera lensflare Signed-off-by: Aditya <aditya050995@gmail.com> Add extra model in test world The pre-existing model in the test world exhibits the lens flare rendering bug without the fix in this branch, so I added an extra model at a different pose that does not exhibit the bug. It makes it easier to confirm that the bug is fixed. * Add extra cameras with occluded views * Add another camera with box to its side that is incorrectly occluded Signed-off-by: Steve Peters <scpeters@openrobotics.org>

* Reverse extraneous z-up to y-up transform that was corrupting WideAngleCamera LensFlare results. Fixed small related bugs. * Use fisheye cameras in lensflare_wideangle_cam.world to improve testing of lens flare.

* Fixing Z check in WideAngleCamera::Project3d Signed-off-by: Sanjuksha Nirgude <sanjuksha@gmail.com> Added a test case for wideangle camera lensflare Signed-off-by: Aditya <aditya050995@gmail.com> Add extra model in test world The pre-existing model in the test world exhibits the lens flare rendering bug without the fix in this branch, so I added an extra model at a different pose that does not exhibit the bug. It makes it easier to confirm that the bug is fixed. * Add extra cameras with occluded views * Add another camera with box to its side that is incorrectly occluded Signed-off-by: Steve Peters <scpeters@openrobotics.org>

iche033 previously approved these changes Nov 10, 2022

View reviewed changes

ahcorde requested changes Nov 10, 2022

View reviewed changes

gazebo/rendering/WideAngleCamera.cc Outdated Show resolved Hide resolved

sanjuksha force-pushed the ZFix branch from e5d22e2 to 9ef7cfe Compare November 11, 2022 20:53

sanjuksha changed the title ~~Fixing Z check in WideAngleCamera::Project3d~~ Fixing WideAngleCamera Lensflare issue Nov 16, 2022

sanjuksha force-pushed the ZFix branch from 1c6657c to a545ff7 Compare November 16, 2022 20:08

iche033 reviewed Nov 16, 2022

View reviewed changes

gazebo/rendering/LensFlare.cc Outdated Show resolved Hide resolved

scpeters reviewed Nov 18, 2022

View reviewed changes

scpeters mentioned this pull request Nov 29, 2022

Wide angle lens flare occlusion bug #3273

Closed

adityapande-1995 requested changes Dec 5, 2022

View reviewed changes

gazebo/rendering/WideAngleCamera.cc Outdated Show resolved Hide resolved

adityapande-1995 force-pushed the ZFix branch from 719cc88 to 3db8327 Compare December 28, 2022 21:45

adityapande-1995 requested a review from scpeters December 28, 2022 21:46

scpeters force-pushed the ZFix branch from 7be87db to c63047c Compare January 11, 2023 23:41

scpeters force-pushed the ZFix branch from 0231859 to d33593d Compare January 26, 2023 00:37

sanjuksha and others added 2 commits February 9, 2023 10:39

Improvements to WideAngleCamera LensFlare bug fix (#3296)

adfb4ff

* Reverse extraneous z-up to y-up transform that was corrupting WideAngleCamera LensFlare results. Fixed small related bugs. * Use fisheye cameras in lensflare_wideangle_cam.world to improve testing of lens flare.

scpeters force-pushed the ZFix branch from f1ae214 to adfb4ff Compare February 9, 2023 23:22

scpeters merged commit e1f7f1f into gazebo11 Feb 9, 2023

scpeters deleted the ZFix branch February 9, 2023 23:22

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fixing WideAngleCamera Lensflare issue #3276

Fixing WideAngleCamera Lensflare issue #3276

sanjuksha commented Nov 9, 2022 •

edited

Loading

sanjuksha commented Nov 11, 2022

scpeters commented Nov 16, 2022

kjeppesen1 commented Nov 17, 2022

scpeters Nov 18, 2022

adityapande-1995 Dec 27, 2022 •

edited

Loading

scpeters Nov 18, 2022

adityapande-1995 Dec 27, 2022

scpeters Nov 18, 2022

iche033 Nov 22, 2022

scpeters Nov 28, 2022

scpeters commented Nov 30, 2022

adityapande-1995 left a comment

scpeters commented Jan 11, 2023

scpeters commented Jan 31, 2023 •

edited

Loading

Fixing WideAngleCamera Lensflare issue #3276

Fixing WideAngleCamera Lensflare issue #3276

Conversation

sanjuksha commented Nov 9, 2022 • edited Loading

🦟 Bug fix

Summary

Checklist

Results

sanjuksha commented Nov 11, 2022

scpeters commented Nov 16, 2022

kjeppesen1 commented Nov 17, 2022

scpeters Nov 18, 2022

Choose a reason for hiding this comment

adityapande-1995 Dec 27, 2022 • edited Loading

Choose a reason for hiding this comment

scpeters Nov 18, 2022

Choose a reason for hiding this comment

adityapande-1995 Dec 27, 2022

Choose a reason for hiding this comment

scpeters Nov 18, 2022

Choose a reason for hiding this comment

iche033 Nov 22, 2022

Choose a reason for hiding this comment

scpeters Nov 28, 2022

Choose a reason for hiding this comment

scpeters commented Nov 30, 2022

adityapande-1995 left a comment

Choose a reason for hiding this comment

scpeters commented Jan 11, 2023

scpeters commented Jan 31, 2023 • edited Loading

Failure to occlude in wide_angle_cameras_occluded_high camera image

Occluding when it shouldn't in wide_angle_cameras_occluded_higher camera image

sanjuksha commented Nov 9, 2022 •

edited

Loading

adityapande-1995 Dec 27, 2022 •

edited

Loading

scpeters commented Jan 31, 2023 •

edited

Loading

Failure to occlude in `wide_angle_cameras_occluded_high` camera image

Occluding when it shouldn't in `wide_angle_cameras_occluded_higher` camera image