What if a partial set of poses is known?

[dscho15]

Hey Naver,

First of all great work, it is very interesting to play around with!

I'm curious, if one knows a partial set of poses and focal lengths aforehand, how should one initialize the pose-graph?

Best regards

[hturki]

looking at the code, there are preset_pose and preset_focal functions that might do what you want?

[dscho15]

I did overwrite the estimated focal lengths, but since I'm only given a partial set of poses I would like to sort of include it in the optimization problem as a landmark. But thanks for the headsup.

[jerome-revaud]

Internally, we have a slower version of the global alignment where you can force a partial initialisation of camera intrinsics and extrinsics (e.g. focals and poses).
If you're interested, we could release this code. @yocabon

[dscho15]

That would be a very nice feature 😁

[mizeller]

I'd also be interested in this code; specifically to be able to provide known camera intrinsics to the pipeline. I assume this additional prior should simplify the optimization procedure as well?

Thanks for the great work! Very interesting :-)

[yocabon]

I'll take a look.

[yocabon]

I added it in 4a414b6
Example usage:

    # here data in list of (intrinsics, pose)
    scene = global_aligner(output, device=device, mode=GlobalAlignerMode.ModularPointCloudOptimizer, fx_and_fy=True)

    scene.preset_pose([data[i][1].cpu().numpy() for i in range(1, 3)], [False, True, True])
    scene.preset_intrinsics([data[i][0].cpu().numpy() for i in range(1, 3)], [False, True, True])
    loss = scene.compute_global_alignment(init="mst", niter=niter, schedule=schedule, lr=lr)

Note, it won't work if you have only one known pose.

[drdsgvo]

I added it in 4a414b6 Example usage:

    # here data in list of (intrinsics, pose)
    scene = global_aligner(output, device=device, mode=GlobalAlignerMode.ModularPointCloudOptimizer, fx_and_fy=True)

    scene.preset_pose([data[i][1].cpu().numpy() for i in range(1, 3)], [False, True, True])
    scene.preset_intrinsics([data[i][0].cpu().numpy() for i in range(1, 3)], [False, True, True])
    loss = scene.compute_global_alignment(init="mst", niter=niter, schedule=schedule, lr=lr)

Note, it won't work if you have only one known pose.

After trying this I got the following error:

loss = scene.compute_global_alignment(init="mst", niter=niter, schedule=schedule, lr=lr)
File "/home/km/.local/lib/python3.10/site-packages/torch/amp/autocast_mode.py", line 16, in decorate_autocast
return func(*args, **kwargs)
File "/home/km/python/check/dust3r/dust3r/cloud_opt/base_opt.py", line 304, in compute_global_alignment
init_fun.init_minimum_spanning_tree(self, niter_PnP=niter_PnP)
File "/home/km/.local/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/home/km/python/check/dust3r/dust3r/cloud_opt/init_im_poses.py", line 77, in init_minimum_spanning_tree
return init_from_pts3d(self, pts3d, im_focals, im_poses)
File "/home/km/python/check/dust3r/dust3r/cloud_opt/init_im_poses.py", line 84, in init_from_pts3d
raise NotImplementedError("Would be simpler to just align everything afterwards on the single known pose")
NotImplementedError: Would be simpler to just align everything afterwards on the single known pose

I used your code, put it into the demo code in the appropriate place and used the following data for preset_pose+intrinsics for 2 images:
pose: torch.Tensor([ [[1,0.0,0,0],[0.0,1,0,0],[0,0,1,0],[0,0,0,0]], [[0.85,0.25,-0.45,0],[-0.1,1,0.4,0],[0.5,-0.3,1,0],[0,0,0,0]] ])
intrinsics: torch.Tensor([ [[685,0,256],[0,685,192],[0,0,1]], [[685,0,256],[0,685,192],[0,0,1]] ])
Am I doing something wrong?

I have no clue how to set the pose+intrinsic data for known camera positions (sorry for me being stupid, I need some advice as I am not an expert in 3D). What are the meanings of the values in the pose+intrinsics tensors?

[PHOENIXFURY007]

I added it in 4a414b6 Example usage:

    # here data in list of (intrinsics, pose)
    scene = global_aligner(output, device=device, mode=GlobalAlignerMode.ModularPointCloudOptimizer, fx_and_fy=True)

    scene.preset_pose([data[i][1].cpu().numpy() for i in range(1, 3)], [False, True, True])
    scene.preset_intrinsics([data[i][0].cpu().numpy() for i in range(1, 3)], [False, True, True])
    loss = scene.compute_global_alignment(init="mst", niter=niter, schedule=schedule, lr=lr)

Note, it won't work if you have only one known pose.

Can this work if I have all the poses known?

[PHOENIXFURY007]

I added it in 4a414b6 Example usage:

    # here data in list of (intrinsics, pose)
    scene = global_aligner(output, device=device, mode=GlobalAlignerMode.ModularPointCloudOptimizer, fx_and_fy=True)

    scene.preset_pose([data[i][1].cpu().numpy() for i in range(1, 3)], [False, True, True])
    scene.preset_intrinsics([data[i][0].cpu().numpy() for i in range(1, 3)], [False, True, True])
    loss = scene.compute_global_alignment(init="mst", niter=niter, schedule=schedule, lr=lr)

Note, it won't work if you have only one known pose.

After trying this I got the following error:

loss = scene.compute_global_alignment(init="mst", niter=niter, schedule=schedule, lr=lr)

File "/home/km/.local/lib/python3.10/site-packages/torch/amp/autocast_mode.py", line 16, in decorate_autocast
return func(*args, **kwargs)
File "/home/km/python/check/dust3r/dust3r/cloud_opt/base_opt.py", line 304, in compute_global_alignment
init_fun.init_minimum_spanning_tree(self, niter_PnP=niter_PnP)
File "/home/km/.local/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/home/km/python/check/dust3r/dust3r/cloud_opt/init_im_poses.py", line 77, in init_minimum_spanning_tree
return init_from_pts3d(self, pts3d, im_focals, im_poses)
File "/home/km/python/check/dust3r/dust3r/cloud_opt/init_im_poses.py", line 84, in init_from_pts3d
raise NotImplementedError("Would be simpler to just align everything afterwards on the single known pose")
NotImplementedError: Would be simpler to just align everything afterwards on the single known pose
I used your code, put it into the demo code in the appropriate place and used the following data for preset_pose+intrinsics for 2 images: pose: torch.Tensor([ [[1,0.0,0,0],[0.0,1,0,0],[0,0,1,0],[0,0,0,0]], [[0.85,0.25,-0.45,0],[-0.1,1,0.4,0],[0.5,-0.3,1,0],[0,0,0,0]] ]) intrinsics: torch.Tensor([ [[685,0,256],[0,685,192],[0,0,1]], [[685,0,256],[0,685,192],[0,0,1]] ]) Am I doing something wrong?

I have no clue how to set the pose+intrinsic data for known camera positions (sorry for me being stupid, I need some advice as I am not an expert in 3D). What are the meanings of the values in the pose+intrinsics tensors?

Can you share the whole code which you used to arrive at this error ?

[gogojjh]

Issue: the results become worse if intrinsics are set. Is it something wrong with the code?
here is my code and data:
data: https://drive.google.com/drive/folders/1X_H0hsZ7GlXh5Qaejop8gdPUxVpwI6r6?usp=sharing
code:

images = load_images(['frame_00000.jpg', 'frame_00005.jpg', 'frame_00003.jpg'], size=512)
pairs = make_pairs(images, scene_graph='complete', prefilter=None, symmetrize=True)
output = inference(pairs, model, device, batch_size=1, verbose=False)
scene = global_aligner(output, device=device, mode=GlobalAlignerMode.ModularPointCloudOptimizer)
# Intrinsics: camera intrinsics
K1 = np.array([[205.47, 0, 320.0], [0, 205.47, 180.0], [0, 0, 1]])
K2 = np.array([[205.47, 0, 320.0], [0, 205.47, 180.0], [0, 0, 1]])
K3 = np.array([[205.47, 0, 320.0], [0, 205.47, 180.0], [0, 0, 1]])   
scene.preset_intrinsics([K1, K2, K3], [True, True, True])
# Pose: from world to camera
pose1 = np.eye(4)
pose2 = np.array([[ 0.97291657,  0.,          0.23115654,  0.3718034 ],
                                 [ 0.        ,  1.,          0.        ,  0.        ],
                                 [-0.23115654,  0.,          0.97291657,  1.04774425],
                                 [ 0.        ,  0.,          0.        ,  1.        ]])
pose3 = np.eye(4)
scene.preset_pose(known_poses=[pose1, pose2, pose3], pose_msk=[True, True, False])
loss = scene.compute_global_alignment(init="mst", niter=niter, schedule=schedule, lr=lr)

Pointmap without giving intrinsics: results seem to be correct

Pointmap with giving intrinsics: results become wrose

[2cu-1001]

@gogojjh I also have this issue. Did you make progress?

[gogojjh]

@2cu-1001 a possible issue is that the camera intrinsics should be rescaled w.r.t. the image. But I have not tried yet.

[gogojjh]

@2cu-1001 you should rescale the camera intrinsics like this

# Given:
# - ori_K: Original 3x3 camera intrinsics matrix
# - ori_im_size: Original image dimensions (width, height)
# - new_im_size: New image dimensions (width, height)

# Calculate scaling factors
scale_x = new_im_size[0] / ori_im_size[0]
scale_y = new_im_size[1] / ori_im_size[1]

# Create new intrinsics matrix
new_K = ori_K.clone() if isinstance(ori_K, torch.Tensor) else ori_K.copy()
new_K[0, 0] *= scale_x  # Focal length X
new_K[1, 1] *= scale_y  # Focal length Y
new_K[0, 2] = new_K[0, 2] * scale_x  # Principal point X
new_K[1, 2] = new_K[1, 2] * scale_y  # Principal point Y

[2cu-1001]

@gogojjh thanks a lot!