problem about the resolution of equirect_depth image

[fangchuan]

Hi replica360 team, thank you for your awesome work.
First , there is a minor bug during writing equirectangular depth image:

matryodshka-replica360/ReplicaSDK/src/render_dataset.cpp

Lines 423 to 449 in 33d9cff

	char equirectFilename[1000];
	snprintf(equirectFilename, 1000, "%s/%s_%04zu_pos%02zu.jpeg", outputDir.c_str(), scene.c_str(), j, k);
	pangolin::SaveImage(
	image.UnsafeReinterpret<uint8_t>(),
	pangolin::PixelFormatFromString("RGB24"),
	std::string(equirectFilename), 100.0);
	if( renderDepth && (k==2 || k==5 || k== 8 || k==11)){
	//render depth image for the equirect image
	depthFrameBuffer.Bind();
	glClearColor(1.0f, 1.0f, 1.0f, 1.0f);
	glPushAttrib(GL_VIEWPORT_BIT);
	glViewport(0, 0, width, height);
	glClear(GL_DEPTH_BUFFER_BIT | GL_COLOR_BUFFER_BIT);
	glEnable(GL_CULL_FACE);
	ptexMesh.RenderDepth(s_cam, 1.f/ 16.f, Eigen::Vector4f(0.0f, 0.0f, 0.0f, 0.0f), eye);
	glDisable(GL_CULL_FACE);
	glPopAttrib(); //GL_VIEWPORT_BIT
	depthFrameBuffer.Unbind();
	depthTexture.Download(depthImage.ptr, GL_RGB, GL_UNSIGNED_BYTE);
	char filename[1000];
	snprintf(filename, 1000, "%s/%s_%04zu_pos%02zu.jpeg", outputDir.c_str(), scene.c_str(), j, 11 + (k+1)/3 ); //11+(k+1)/3 maps 2-12; 5-13; 8-14; 11-15
	pangolin::SaveImage(

we should skip the save of rgb panorama image when k == 12 .... Addingf the following lines before line 423:

          if (12 == k || 13 == k || 14 == k || 15 == k){
            continue;
          }

After I use your code to render equirectangular depth images, I find the depth image can be naturally unprojected to a unit sphere, the we can get the hole room pointcloud at once.

./ReplicaSDK/ReplicaRendererDataset /media/ziqianbai/BACKPACK_DATA1/Replica_all/replica_v1/room_0/mesh.ply /media/ziqianbai/BACKPACK_DATA1/Replica_all/replica_v1/room_0/textures/ /media/ziqianbai/BACKPACK_DATA1/Replica_all/replica_v1/room_0/glass.sur ../glob/train/room_0_6dof.txt y /media/ziqianbai/BACKPACK_DATA1/Replica_all/replica_for_panonerf/room_0/ 1024 512 ../glob/pro2pos.txt

But the pointcloud seems imprecise, the depth is likely been truncated cause of inappropriate depth scale or depth image format:

The code I used to unproject the equirect_depth_image :

def test_spherical_depth():
    def get_unit_spherical_map():
        h = 512
        w = 1024
        Theta = np.arange(h).reshape(h, 1) * np.pi / h + np.pi / h / 2
        Theta = np.repeat(Theta, w, axis=1)
        Phi = np.arange(w).reshape(1, w) * 2 * np.pi / w + np.pi / w - np.pi
        Phi = -np.repeat(Phi, h, axis=0)

        X = np.expand_dims(np.sin(Theta) * np.sin(Phi),2)
        Y =  np.expand_dims(np.cos(Theta),2)
        Z = np.expand_dims(np.sin(Theta) * np.cos(Phi),2)
        unit_map = np.concatenate([X,Z,Y],axis=2)

        return unit_map

    depth_img_filepath = '/media/ziqianbai/BACKPACK_DATA1/Replica_all/replica_for_panonerf/room_0/room_0_0000_pos12.png'
    raw_depth_img = Image.open(depth_img_filepath)
    depth_img = ImageOps.grayscale(raw_depth_img)
    depth_img = np.asarray(depth_img)
    # depth_img = np.asarray(Image.open(depth_img_filepath))
    depth_img=np.expand_dims((depth_img*16.0),axis=2)
    pointcloud = depth_img * get_unit_spherical_map()

    o3d_pointcloud = o3d.geometry.PointCloud()
    o3d_pointcloud.points = o3d.utility.Vector3dVector(pointcloud.reshape(-1,3))
    o3d.io.write_point_cloud('/media/ziqianbai/BACKPACK_DATA1/Replica_all/replica_for_panonerf/room_0/room_0_0000_pcl_2.ply', o3d_pointcloud)

@jamestompkin @breuckelen @lynl7130 Could u help me figure out the reason the resulting phenomenon? Thanks in advance!