Regarding IGazeControl::triangulate3DPoint() and yarp-device-realsense2

[paliasgh]

Hello,

I am using IGazeControl::triangulate3DPoint() method for my application. I noticed if the robot's head moves (e.g., rotates to left/right), then the returned 3d point of a fixed object changes significantly. The documentation says the output is in the root reference frame, which does not move in this case, as it is attached to the robot's waist. Can the changes I see be related to distortion that still exists in the camera outputs? In the ideal case, will triangulate3DPoint() automatically take care of the head position?

Also, is there any way to calibrate the returned 3d points other than playing with the eye vergence and adjusting camera calibration parameters? Thank you very much.

[pattacini]

Hi @paliasgh

What is the context of your application?

The documentation¹ reports in multiple places the following caveat:

Note

The triangulation makes use of the mid-point method and thus is deeply affected by uncertainties in the cameras extrinsic parameters and cameras alignment.

Therefore, this method can be applied, essentially, only in simulation² as a tiny error affecting the stereo calibration yields a substantial modification of the triangulated 3D point.

Can the changes I see be related to distortion that still exists in the camera outputs?

Not only the distortion but primarily the extrinsic parameters of the stereo rig plus how the rig is (dis-)aligned with the robot's kinematics. This is full of unknowns, eventually, which can be barely estimated/compensated for.

In the ideal case, will triangulate3DPoint() automatically take care of the head position?

Yes.

Also, is there any way to calibrate the returned 3d points other than playing with the eye vergence and adjusting camera calibration parameters? Thank you very much.

Abandon all hope... 😄
That's the reason why with the iCub many of us ended up using a RealSense mounted on top of the head.
See https://icub-tech-iit.github.io/documentation/upgrade_kits/realsense_holder/support/.

Footnotes

1. See:

   • https://yarp.it/latest/classyarp_1_1dev_1_1IGazeControl.html#a30cb9848e3d8d92c33bd5a807e1038d9
   • https://robotology.github.io/robotology-documentation/doc/html/group__iKinGazeCtrl.html
   ↩

2. Examples:

   • https://github.com/vvv-school/assignment_make-it-roll
   • https://github.com/vvv-school/assignment_grasp-it
   ↩

[paliasgh]

Hello @xEnVrE ,

Thanks for your prompt response! I now see the problem. The visp I built from their master branch does not have the required tutorials. In fact, there is not folder called calibration there. The v3.6.0 branch seems to have that folder. I will switch now and hopefully the problem will be solved. Thanks again!

[paliasgh]

Hello again @xEnVrE ,

Yes, that worked! Thanks again.

There is something to mention: visp/tutorial/calibration/tutorial-hand-eye-calibration does not seem to have --ndata parameter anymore, as it is supposed to run using it. I removed --ndata $N_POSES and could run realsense-holder-calibration-process with no issue.

During the process, realsense-holder-calibration-process rejected my data a couple times because collected pictures were not fully covering the half sphere over the chessboard. I changed the location of the cheeseboard so that it finally generated a calibration matrix. With the resulted matrix, however, realsense-holder-publisher does not give estimates as good as with the given matrix. How exactly should I hold the cheeseboard to obtain better results? I have been using calibration poses as in config_iCubGenova01_holder_tilt.ini and already updated camera intrinsics there. Thanks! Pictures I worked with look like this:

[xEnVrE]

With the resulted matrix, however, realsense-holder-publisher does not give estimates as good as with the given matrix.

Can you elaborate a bit more on this? When do you define the estimates to be "good"? Are you using any additional benchmark to validate the estimates?

Otherwise, if you are just comparing the output of the publisher, it is normal that you get different estimates than those obtained using the provided matrix as it depends, e.g., on how you mounted the holder, how you collected the pictures, if you are using additional adapters between the holder and the camera, etc.

[paliasgh]

Hello @xEnVrE ,

I am using realsense-holder-publisher output to convert 3D estimation of a point in real sense frame to robot frame (by a rotation and a translation in the coordinate). In this case, a good estimate is an accurate one (by measuring distances) which is also unaffected by changes in the head position (seeing if head rotations affect results).

Today, I tried putting the chessboard on a table in about 1 meter fart from the robot instead of holding it. I obtained a matrix using which the above process gives almost perfect estimates of 3D location in robot frame! Thanks.

0.9984773479  0.03933398265  -0.03867587759  -0.009536235387
-0.008752365042  0.8051971884  0.5929425637  -0.03185936805
0.05446450041  -0.5917012131  0.8043154186  0.0703791129
0  0  0  1

[xEnVrE]

Glad to know you have managed to achieve the desired performance :)