pose_refinement error & failed to estimate pose in other dataset

[szhang963]

Hi, I am very interested in your great work.

1. I want to try the pose refinement in waymo data (you provided scene), but I met an error:

F0202 06:06:36.188254 3254 problem_impl.cc:482] Parameter block not found: 0x5645a00b9e60. You must add the parameter block to the problem before it can be set constant.
*** Check failure stack trace: ***
@ 0x7f9ba4388b03 google::LogMessage::Fail()
@ 0x7f9ba43909d1 google::LogMessage::SendToLog()
@ 0x7f9ba43887c2 google::LogMessage::Flush()
@ 0x7f9ba438a78f google::LogMessageFatal::~LogMessageFatal()
@ 0x7f9ba3ffb72f ceres::internal::ProblemImpl::SetParameterBlockConstant()
@ 0x56459d8594f1 colmap::BundleAdjuster::ParameterizeCameras()
@ 0x56459d8622b5 colmap::RigBundleAdjuster::SetUp()
@ 0x56459d8627a2 colmap::RigBundleAdjuster::Solve()
@ 0x56459d6994df colmap::RunRigBundleAdjuster()
@ 0x56459d6449b3 main
@ 0x7f9ba20eed90 (unknown)
@ 0x7f9ba20eee40 __libc_start_main
@ 0x56459d64e455 _start
./pose_refinement/stpr/scripts/mvs/run_multi_cam_pose_opt.sh: line 9: 3254 Aborted (core dumped) ${COLMAP_EXE} rig_bundle_adjuster --input_path $POSE_PATH/sparse/0 --output_path $POSE_PATH/sparse/0 --rig_config_path "$POSE_PATH/cam_rigid_config.json" --estimate_rig_relative_poses 0 --RigBundleAdjustment.refine_relative_poses 1 --BundleAdjustment.max_num_iterations 50 --BundleAdjustment.refine_focal_length 0 --BundleAdjustment.refine_principal_point 0 --BundleAdjustment.refine_extra_params 0

What causes it? However, I can still get the sparse point and camera pose in the images.bin file, and is the pose in the images.bin refined?

2. I can't get the sparse point and camera pose in the images.bin file for my data, and also reported the above error. The result is bad as below:

Could you provide me with some help?

Thanks in advance!

[kcheng1021]

Hi, thanks for your attention in our work! It seems that the poses are not loaded correctly, so the parameter block is not found during BA. The pose refinement codes has been checked before upload, but it includes some absolute path in the current state.
Sorry for that. I have been busy in these days, but I will update a more flexible version as soon as possible. Or you can check whether the poses have been loaded by yourself first if you are urgent.

[szhang963]

Hi, thank you for your reply. The issue was solved by fixing the camera poses. However, I'm confused about the camera coordinate system. The pose estimation was not right when I converted my camera coord to the colmap coord. The result was right when I used my data coord (x forward，y left，z up) ?
Besides, I found the reconstruction quality from stpr is not better than the original pose for waymo and myself dataset?
The results below are from nerfstudio using poses from stpr (fig.1) and original poses (fig.2). You can find the rendered image (right image) is blurred more than the original image (left image).

Could you offer some suggestions to solve it? It seems the poses are not refined precisely for stpr? Or the colmap fall into local optimality?

Thanks in advance!
Best wishes.

[szhang963]

Hi, could you provide the sparse reconstruction model for Waymo-100613 from stpr? I want to compare the estimated poses with mine.
Thanks a lot.

[kcheng1021]

Hi, thank you for your reply. The issue was solved by fixing the camera poses. However, I'm confused about the camera coordinate system. The pose estimation was not right when I converted my camera coord to the colmap coord. The result was right when I used my data coord (x forward，y left，z up) ? Besides, I found the reconstruction quality from stpr is not better than the original pose for waymo and myself dataset? The results below are from nerfstudio using poses from stpr (fig.1) and original poses (fig.2). You can find the rendered image (right image) is blurred more than the original image (left image).

Could you offer some suggestions to solve it? It seems the poses are not refined precisely for stpr? Or the colmap fall into local optimality?

Thanks in advance! Best wishes.

Hi, how did you load your poses? For our stpr, we use the standard opencv coordinate. For the demo scene Waymo-100613, the c2w loaded from scenario.pt is also in opencv coordinate (you can also check in https://github.com/PJLab-ADG/neuralsim/blob/main/docs/data/autonomous_driving.md).

Referring to your posted images, it seems that your Fig. 1 and Fig. 2 do not compare rendered images of stpr and original poses at the same timestamp. More importantly, our stpr could alleviate the blurriness in the overlapping region of different cameras, such as the following image.

But not the global blurriness you mentioned. In my opinion, your rendered images appear blurrier than the original image because the used rendering algorithm lacks proper anti-aliasing design or has limited abilities to model large scenes. I guess that you use instant-ngp or mipnerf360 from nerfstudio, which are known to produce blurry results when rendering self-driving scenes (also shown in our paper).

Finally, I can not access to the laboratory server data from the outside network during winter break. I will share my poses later.

[szhang963]

Hi, thank you for your reply.

1. I converted my pose using the following codes:
   
   

2. Yes, Fig.1 and Fig.2 are not the same timestamp. I find they are all blurred, and the stpr does not improve the blurriness (so, I want to compare the poses with your poses). Your provided results were trained from zip-nerf using the original pose and stpr pose?

[kcheng1021]

1. Is your parameter ``pose'' in opencv coordinate?

2. Yes. They are trained from zip-nerf using the original poses and stpr poses. The blurriness you show is not caused from inaccurate poses, but more likely from the neural rendering algorithm itself. A simple way to validate it is that you could train your NeRF with images from one camera.

[szhang963]

Thanks for your reply.

1. The "pose" is the pose of ego car in the world coord and is not in opencv coord. It is the same as the coord of my cam. I should convert the pose to opencv? However, I still got an error result using the code pose = np.matmul(np.linalg.inv(opencv2phigent), pose).
2. OK, I will try it. Thanks!

[kcheng1021]

Thanks for your reply.

1. The "pose" is the pose of ego car in the world coord and is not in opencv coord. It is the same as the coord of my cam. I should convert the pose to opencv? However, I still got an error result using the code pose = np.matmul(np.linalg.inv(opencv2phigent), pose).
2. OK, I will try it. Thanks!

If you switch to the OpenCV coordinate system, all the poses involved in your calculations should be transformed to the OpenCV coordinate system as well. It seems like you might be mixing OpenCV coordinate with other coordinates in your usage. If your data contains depth, one simple way to check whether your converted poses are correct is to accumulate the point cloud obtained through backprojection frame by frame and load it into COLMAP. By doing so, you can check the positions of the points and their relationship with the coordinate system.

[grandoba]

@szhang963 Hi I am getting the same error.

F0702 11:02:12.864603 3932059 problem_impl.cc:635] Parameter block not found: 0x55e292b36dc0. You must add the parameter block to the problem before it can be set constant.
*** Check failure stack trace: ***
    @     0x7f8422af51c3  google::LogMessage::Fail()
    @     0x7f8422afa25b  google::LogMessage::SendToLog()
    @     0x7f8422af4ebf  google::LogMessage::Flush()
    @     0x7f8422af56ef  google::LogMessageFatal::~LogMessageFatal()
    @     0x7f8422c63582  ceres::internal::ProblemImpl::SetParameterBlockConstant()
    @     0x55e29183e429  colmap::BundleAdjuster::ParameterizeCameras()
    @     0x55e2918447a5  colmap::RigBundleAdjuster::SetUp()
    @     0x55e291844ba2  colmap::RigBundleAdjuster::Solve()
    @     0x55e29169821d  colmap::RunRigBundleAdjuster()
    @     0x55e29164d642  main
    @     0x7f84210e2083  __libc_start_main
    @     0x55e29165187e  _start
./pose_refinement/stpr/scripts/mvs/run_multi_cam_pose_opt.sh: line 18: 3932059 Aborted                 (core dumped) ${COLMAP_EXE} rig_bundle_adjuster --input_path $POSE_PATH/sparse/0 --output_path $POSE_PATH/sparse/0 --rig_config_path "$POSE_PATH/cam_rigid_config.json" --estimate_rig_relative_poses 0 --RigBundleAdjustment.refine_relative_poses 1 --BundleAdjustment.max_num_iterations 50 --BundleAdjustment.refine_focal_length 0 --BundleAdjustment.refine_principal_point 0 --BundleAdjustment.refine_extra_params 0

Could you provide how you solved this problem by "fixing the camera poses?"