Removing flaky by running the test only on RGB

[vincentpierre]

Testing depth gets 1.0 values too often which makes the test fail regularly. The same functionality is achieved by running only RGB in this test.

Motivation and Context

How Has This Been Tested

Types of changes

• Docs change / refactoring / dependency upgrade
• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)

Checklist

• My code follows the code style of this project.
• My change requires a change to the documentation.
• I have updated the documentation accordingly.
• I have read the CONTRIBUTING document.
• I have completed my CLA (see CONTRIBUTING)
• I have added tests to cover my changes.
• All new and existing tests passed.

[aclegg3]

Looks like this is asserting that no pixel obs(x,y) of the reset state image is the same as pixel new_obs(x,y) of an image rendered after moving the agent. Does that sound correct?

Your hypothesis is that depth (as a single channel) is more likely than RGB (with 3 channels) to trigger the test failure stochastically?

CI still seems to fail here and this assertion seems doomed to fail stochastically anyway. Maybe better to check global image similarity instead of per-pixel similarity, right? See this sim test.

[vincentpierre]

Sorry, sorry. I got confused with my if statement. I was actually putting depth only in the test rather than rgb only. It should be fixed in this commit.
My guess is that depth has a high chance of returning all 1.0 because the depth maximum distance is not that high. This means that I believe that there are multiple POV from which the depth image will be all 1.0 (everything is far from the camera)

I agree that there is still some stochasticity with RGB only BUT much much less. I have the test running in a while True loop and it has not failed yet.

[mathfac]

Thank you for fixing the test. That makes sense to run the test only for one type of modalities.
Can you elaborate more on "Testing RGB gets 1.0 values too often" ?
Which values do you mean and how RGB contributes to failures?

[vincentpierre]

Can you elaborate more on "Testing RGB gets 1.0 values too often" ?

Sorry, I meant depth. It seems that the max depth of 1 is happening very often.