On the Calculation of the Final Pose of Registered Images

[adrianJW421]

Hello!

Based on my understanding, the Tpre matrix generated by the match results is the transformation matrix that converts the extrinsics of the input point cloud (pcd) projection to the camera pose where the image is registered within the point cloud.

Could you provide a example tool code that allows the user to directly obtain the final pose of the registered image? If my understanding is incorrect, could you please explain how to accurately determine the final pose of the registered image?

Thank you!

[HpWang-whu]

Hi @adrianJW421,
Many thanks for your suggestion.
I have open-sourced a easy-to-use version of FreeReg Here. There are some brief introductions about pose and more easy-to-understand. Welcome to use it!

Yours,

[adrianJW421]

@HpWang-whu
Hello, thanks for releasing FFreeReg, I tried it on the demo data, the registeration looks fine.

After I tried it on some custom image and ply scene, the value of translation vector in the estimated Extrinsic often went to over 100, for example
[Result] Estimated intrinsic of image:
[[2.99842139e+03 0.00000000e+00 2.04750000e+03]
[0.00000000e+00 2.99842139e+03 1.53550000e+03]
[0.00000000e+00 0.00000000e+00 1.00000000e+00]]
[Result] Estimated extrinsic of image:
[[ 9.52254601e-01 -7.50058867e-02 2.95948124e-01 -3.68457958e+02]
[ 8.94472803e-02 9.95357155e-01 -3.55431825e-02 1.34429317e+02]
[-2.91908135e-01 6.03179138e-02 9.54542503e-01 -4.00320020e+02]
[ 0.00000000e+00 0.00000000e+00 0.00000000e+00 1.00000000e+00]]

after visualization, the estimated camera pose and the ply looks like this:

Was it purely the performance issue or is there some requirements for preprocessing the input image and ply that I have overlooked? Or is there a particular condition that the ply point cloud should follow (like a certain coordinate system)?

[HpWang-whu]

Hi @adrianJW421 ,
Thanks for your interest! Can you provide an example raw data to me so I can do some check.
The result might be a little late (~ after two weeks) for I am currently going through a very busy month. Really sorry for that!

Yours,

[adrianJW421]

Hi @adrianJW421 , Thanks for your interest! Can you provide an example raw data to me so I can do some check. The result might be a little late (~ after two weeks) for I am currently going through a very busy month. Really sorry for that!

Yours,

@HpWang-whu Thanks a lot. Can you try download the badcase data from this drive link?

Looking forward to hearing from you

[LoveMYChen]

Excuse me, are you using your own dataset to run the demo

[HpWang-whu]

Hi @adrianJW421 ,
Thank you for sharing the data. I have fixed a few bugs.

1. The above issue mainly because I forgot to set the "distance thresold for inlier judgement of RANSAC" according to the scale of point clouds. I have fixed it.
2. I add a scale alignment between the mono-reconstructed point cloud and the target input point cloud before alignment.
3. I have exposed several adjustable parameters that impact the transformation estimation. These parameters are used for inlier determination within RANSAC. In our RANSAC transformation scoring, we simultaneously consider both the PnP method and the Kabsch method. Each relies on an inlier threshold: ir_2d (in pixels) for the PnP method and ir_3d (in meters) for the Kabsch method. You can also choose not to set them, as FFreeReg will automatically configure them.

You might re-pull the repo to have a try. The hyperparameters are introduced in the demo.py.

Since YOHO was trained on indoor data quite different from yours, its performance is not stable. With ir_2d set to 30, I obtained the following results.

Yours,