NaN in frame.IIII.position.hdf5

[Matstah]

Hi there

I am using ai_035_001/images/scene_cam_00_geometry_hdf5/frame.0001.position.hdf5 to get the world positions of all pixels seen on the image.

When looking at pixel (257,837), and printing the 3D point by print('xyz_world: ', xyz_world[257,837,:]), I get a NaN vector back xyz_world: [nan nan nan].

Is there any meaning behind the nan value used for a xyz world point?

Thanks

[mikeroberts3000]

Hi! Great question. NaN values can and do occur in the Hypersim images when there is no scene geometry at that pixel. In the case of {ai_035_001, cam_00, frame.0001}, we see that there are indeed some "empty" pixels, i.e., pixels that don't observe any scene geometry. Here are some preview images from this frame.

frame.0001.tonemap.jpg

frame.0001.depth_meters.png

White values in the frame.0001.depth_meters.png preview image indicate empty pixels, and we expect to have NaN values in these pixels in our HDF5 files. The exact pixel coordinate you're referring to appears to correspond to the top-right pixel of the right-most window.

In summary, you're seeing a NaN value because there is no geometry at that pixel, and therefore the pixel's world-space position is undefined.

[rikba]

We also came across this issue when we tried to use the Evermotion data set in a multi view geometry application.
For example in volume 44, scene 2 some pixel blobs in the depth image are not defined:

Quite many scenes have small portions that have undefined geometry information. When trying to backproject all pixels p of one frame into its own camera plane using the pixel world position r_WP, camera intrinsics K, and camera pose T_CW, i.e.,

p_projected = K * T_CW * r_WP(p_original)

we'd expect p_projected == p_original. The histogram below shows the maximum reprojection error for every frame. We suspect that any image that has a maximum reprojection error greater than 100 pixel contains missing geometry information.

We did not cross-compare whether huge reprojection errors are always caused by undefined geometry information.
However, we made the frame-to-itself reprojection results available.
Thus users can possibly remove frames or mask frame patches where the geometry information is problematic.

[mikeroberts3000]

@rikba this analysis is great!

I could imagine that 30k images (roughly half of the images in Hypersim) contain at least a few pixels with undefined geometry. As we see in the original post above, all it takes is for there to be a visible open window in an otherwise well-behaved image, and you'll get some undefined values. So 30k seems plausible to me. As you already mentioned, it would be interesting to see how many of those >100px max errors are due to undefined geometry. Is that information stored in your error statistics CSV file? Or are the NaN values being converted to finite floating-point values somehow by the time they are saved to your CSV. I hope it is the former.

The other (finite) reprojection errors tell us exactly what scenes are affected by #24. This analysis is very valuable! Thank you 😄

[rikba]

Is that information stored in your error statistics CSV file? Or are the NaN values being converted to finite floating-point values somehow by the time they are saved to your CSV. I hope it is the former.

Unfortunately, the latter. We did not check the values in frame.FFFF.position.hdf5 before doing the reprojection. Thus I think the computation assumes some random memory for the world position of the pixel and comes up with a random reprojection error, e.g. 1.3043818e+19 that is then stored as maximum reprojection error in the CSV. But since this evaluation seems valueable, we might rerun it and flag frames that contain undefined pixel positions.