On dummy camera placement : Open3D vs GS renders

[leblond14u]

Hi,

I followed the #506 issue to get new camera placed in my scene.
However I'm encountering an issue with the placement of those cameras.
I have a pose in 3D that is following the C2W coordinate system and a rotation (to flip the axes to colmap like cameras).
So what I've done is placing the new camera pose at this C2W position and rotation like so :

pose[:3, 3] = torch.from_numpy(path_center).float()
pose[1, 3] += 500
        
# Rotation for BEV: 90-degree rotation around X-axis to look down along Y-axis (COLMAP coordinate system).
R_BEV = torch.tensor([[1,  0,  0],   # X-axis remains the same
                           [0,  0,  -1],   # Z becomes Y
                           [0, 1,  0]])  # Y becomes -Z

pose[:3, :3] = R_BEV  # Update the rotation part with BEV rotation

pose = np.linalg.inv(pose.numpy())  # Convert to NumPy array
self.R = pose[:3, :3]  # Rotation matrix for the camera
self.T = pose[:3, 3]   # Translation vector for the camera

When applying those operations with open3D I can see my coordinate system is indeed well positioned in space with the right axes switch to make it to the colmap format.
However with the GS process I noticed that my renders were taken with a shift in camera position.
I suspect it could come from the fact the operations are done the OpenGL way but I didn't succeed yet to find the right transform.

Thanks in advance,
Best,

Hugo

[guaMass]

As shown in the figure, the camera's default z-axis is forward from the center of the camera, the x-axis is to the right and the y-axis is pointing down according to the right hand system. Normally we would set the z-axis up in world coordinates. In this way, if we set the camera rotation matrix to be a unitary matrix during rendering, the camera will be oriented towards the sky. All we need to do at the beginning is to rotate the camera's rotation matrix by $-90$ degrees around the camera's $x$-axis in order to place the camera horizontally. Rotating the camera by $-90$ degrees around the $y$-axis of the camera will align the $z$-axis of the camera with the $x$-axis of the world coordinates. You can try this on your own PLY file by using this tool: https://github.com/guaMass/3DGS_PoseRender

This image is cited from: https://spectacularai.github.io/docs/sdk/tracking.html

[leblond14u]

Hi thanks for your answer,

Yes I understand the coordinate system transform pretty well, I just need to flip my axes to get to a more conventional coordinate frame with my data (similar to what you explained). My question was more about how to get the transforms to propagate well in the GS code. As I said my transforms are correct with Open3D but it does not appear to work exactly like this with the addition of the projection matrices etc.

So if I understand what you've done in you tool I could replace in my GS process the self.world_view_transform by your more straightforward transformation_matrix right ?
Like :

self.world_view_transform = torch.tensor(getWorld2View2(self.R, self.T, np.array([0,0,0]), 1.0)).transpose(0, 1).cuda()

to 

self.world_view_transform = get_transformation_matrix(position, rotation)

with get_transformation_matrix() as : 

def get_transformation_matrix(position, rotation):
    T = np.eye(4, 4)
    T[:3, :3] = rotation
    T[:3, 3] = position
    return np.linalg.inv(T.T)

Best,

Hugo