Wide angle lens flare occlusion bug

[iche033]

From external collaborator:

We have been noticing some issues with wideangle​ cameras in gazebo not always rendering the lens flares even when the sun is in clear sight. I added some print statements in LensFlare.cc​ and I can see that the issue is that FirstContact​ (called here) will often return true for intersection even when nothing is actually in between the camera and light source, which results in the scale​ shader uniform getting cleared.

I combined elements of gazebo's lensflare_plugin.world​ and heightmap.world​ to make an easy testing environment that demonstrates this issue. Please see the attached screenshots. The first shows a wideangle camera and a normal camera side-by-side, but the wideangle camera does not have a visible flare. The second screenshot shows the two cameras arbitrarily rotated to find an orientation that correctly produces the lens flare in both. I have also attached the test world - try it out yourself and rotate around the cameras to see how things change. You can also try removing the heightmap, which seems to help a bit, but the problem doesn't fully go away.

Any ideas why the lens flare occlusion checking isn't working very well with wideangle cameras?

lensflare_plugin.world

<?xml version="1.0" ?>
<sdf version="1.6">
  <world name="default">

  <!-- Light Source -->
  <light type="directional" name="sun">
    <cast_shadows>true</cast_shadows>
    <pose>0 0 0 0.0 0.0 0.0</pose>
    <diffuse>1.0 1.0 1.0 1.0</diffuse>
    <specular>0.2 0.2 0.2 1</specular>
    <direction>-1.0 0.0 -0.2</direction>
  </light>

    <!-- A ground plane -->
    <include>
      <uri>model://ground_plane</uri>
    </include>

    <model name="heightmap">
      <static>true</static>
      <link name="link">
        <collision name="collision">
          <geometry>
            <heightmap>
              <uri>file://media/materials/textures/heightmap_bowl.png</uri>
              <size>129 129 10</size>
              <pos>0 0 0</pos>
            </heightmap>
          </geometry>
        </collision>
        <visual name="visual_abcedf">
          <geometry>
            <heightmap>
              <use_terrain_paging>false</use_terrain_paging>
              <texture>
                <diffuse>file://media/materials/textures/dirt_diffusespecular.png</diffuse>
                <normal>file://media/materials/textures/flat_normal.png</normal>
                <size>1</size>
              </texture>
              <texture>
                <diffuse>file://media/materials/textures/grass_diffusespecular.png</diffuse>
                <normal>file://media/materials/textures/flat_normal.png</normal>
                <size>1</size>
              </texture>
              <texture>
                <diffuse>file://media/materials/textures/fungus_diffusespecular.png</diffuse>
                <normal>file://media/materials/textures/flat_normal.png</normal>
                <size>1</size>
              </texture>
              <blend>
                <min_height>2</min_height>
                <fade_dist>5</fade_dist>
              </blend>
              <blend>
                <min_height>4</min_height>
                <fade_dist>5</fade_dist>
              </blend>
              <uri>file://media/materials/textures/heightmap_bowl.png</uri>
              <size>129 129 10</size>
              <pos>0 0 0</pos>
            </heightmap>
          </geometry>
        </visual>
      </link>
    </model>

    <!-- camera model with lens flare plugin -->
    <model name="camera_lensflare">
      <static>true</static>
      <pose>0 1.5 5.2 0 0 0</pose>
      <link name="link">
        <visual name="visual">
          <geometry>
            <box>
              <size>0.1 0.1 0.1</size>
            </box>
          </geometry>
        </visual>
        <sensor name="camera_sensor_lensflare" type="camera">
          <camera>
            <horizontal_fov>0.78539816339744828</horizontal_fov>
            <image>
              <width>320</width>
              <height>240</height>
            </image>
            <clip>
              <near>0.1</near>
              <far>100</far>
            </clip>
          </camera>
          <always_on>1</always_on>
          <update_rate>25</update_rate>
          <plugin name="lensflare" filename="libLensFlareSensorPlugin.so"/>
        </sensor>
      </link>
    </model>

    <!-- wide-angle camera model with lens flare plugin -->
    <model name="wide_angle_camera_lensflare">
      <static>true</static>
      <pose>0 2 5.2 0 0 0</pose>
      <link name="link">
        <visual name="visual">
          <geometry>
            <box>
              <size>0.1 0.1 0.1</size>
            </box>
          </geometry>
        </visual>
        <sensor name="camera" type="wideanglecamera">
          <camera>
            <horizontal_fov>1.047</horizontal_fov>
            <image>
              <width>320</width>
              <height>240</height>
            </image>
            <clip>
              <near>0.1</near>
              <far>100</far>
            </clip>
            <lens>
              <type>gnomonical</type>
              <scale_to_hfov>true</scale_to_hfov>
              <cutoff_angle>1.5707</cutoff_angle>
              <env_texture_size>512</env_texture_size>
            </lens>
          </camera>
          <always_on>1</always_on>
          <update_rate>25</update_rate>
          <visualize>true</visualize>
          <plugin name="lensflare" filename="libLensFlareSensorPlugin.so"/>
        </sensor>
      </link>
    </model>

  </world>
</sdf>

To reproduce:

gazebo --verbose lensflare_plugin.world

Press Ctrl + t to bring up the topic window, scroll down to the gazebo.msgs.ImageStamped section, and double click on the 2 topic names to bring up the image display window. You'll see that lens flare is visible in the regular camera view but not the wide angle camera view

[darksylinc]

While working on the new gazebo version, I noticed a potential bug in the classic version and wide angle cameras:

In LensFlareCompositorListener::WideAngleCameraPosScale, regardless of which face camera is being rendered, will loop through all cameras, find the first one facing towards the sun; and do calculations based on that.

That will work fine for the face camera that matches the calculation. But for the rest of the 5 cameras, it should be using the camera currently being rendered, not the camera facing towards the sun.

The new-gazebo version I ported uses the camera currently being rendered.

I would also not rule out that the wrong camera is being selected due to using pos.z > 0 instead of pos.z > abs(pos.w)

[darksylinc]

Note: when I talk about pos.z > 0 instead of pos.z > abs(pos.w) I mean the code here, since that test is done in various places and you need to hunt them all.

[scpeters]

I just noticed an additional issue while testing #3276. When attempting to occlude the wide-angle camera with other objects, such as the double_pendulum_with_base and brick_box_3x1x3 using the example world from 3d3a1cd I noticed that the wide-angle camera displays the lens flare in two cases when it should be occluded.

[iche033]

hmm are the bugs there before the changes in #3276?

For the second case with the brick wall, one thing to check is if the camera used for occlusion checking has the correct near / far clip planes.

[scpeters]

hmm are the bugs there before the changes in #3276?

For the second case with the brick wall, one thing to check is if the camera used for occlusion checking has the correct near / far clip planes.

yes, I took those screenshots using gazebo 11.12.0, without the changes from #3276. We should set up a demo world with both the heightmap and the occluding box to test it all at once.

[darksylinc]

I run into this problem when implementing this hack:

In gazebo-classic, the lens flare effect is done on the final stitched image with the caveat that the lens flares are not distorted by the wide angle view:
If we have to pick one limitation to go with, we should probably go with the original behavior with no edges. The edges may cause issues with image processing algorithms. Sorry didn't catch this earlier for ogre2 implementation. Can you make this change?

The problems I had in gz-rendering is that the cameras have potentially different FOVs and Aspect Ratios (cubemap camera vs "stitching" camera). This causes slight offsets in the lens flare location in 2D space from where it really should be; therefore when we perform the ray cast, it returns there are no occluders.

Another problem is that because the LensFlare is done after stitching, it is not lens-distorted. So if a cube in front of the camera is "enlarged" by the distortion, it will look like it should occlude the lens flare.

This may be a limitation of the hack (assuming there are no other bugs present)

[scpeters]

fixed by #3276 and #3296