Problem calibrating Camera-IMU system

[ParthaEth]

Hi,

I am trying to calibrate a rolling shutter camera - IMU system. My update rate is 30 FPS for camera and 150 HZ for IMU. The re-projection error reported is ~0.3 px and gyro error is ~0.08 while Accelerometer is about 0.51. Since I do not have much experience I am not really sure if these are good or bad values. It would be great if you could share a summary of a good result. Ideally the PDF generated by your program.

Further more the relative transformation between IMU camera is the one that worries me the most. Although rotation looks correct (imu's z is opposite to that of camera. physically verified). The translation is wrong! I am certain of this because it is [0.08, 0.08, 0.10] => it thinks the imu sits ~11 cm away from the camera which is physically impossible as the phone I am using is physically smaller. I also asked in my lab around and others have faced similar problems with kalibr before without any definitive solution.

Finally any practical advise would be much appreciated. It would also be great if you could share a calibration video demo so I can see which kind of motions to perform in order to get a good calibration. Also just to have a sanity check kalibr and Rovio expects acceleration and gyro reading in m/s^2 and rad/s right?

[hitimo]

Hi @ParthaEth,

• Have you seen these sample datasets?
• What camera are you using?
• Can you post your report for camera intrinsics and cam-imu calibration?

My lessons learned, maybe helpful:

Camera intrinsics calibration

• camera static; move checkerboard
• "Capture the images of a checkerboard pattern at different orientations relative to the camera. Capture enough different images of the pattern so that you have covered as much of the image frame as possible. Lens distortion increases radially from the center of the image and sometimes is not uniform across the image frame. To capture this lens distortion, the pattern must appear close to the edges." (Taken from [1], same applies here.)
• target out of image --> not fatal, image will simply be skipped

Camera-IMU transformation

• make sure that IMU noise/bias in parameter file are correct
• make sure that camera and IMU are correctly time-synchronized (common cause of errors)
• target static; excite all axes of camera-IMU rig
• make sure to set low shutter width + halogen/LED floodlight to avoid motion blur
• no shocks on camera-IMU rig
• target outside image / flips --> erroneous constraint

[flankechen]

@ParthaEth
Can you kindly show the result? I might have the same problem. physical relation between camera and imu is not the calibrated ones. and don't know how to tell a good result.

[SubMishMar]

@hitimo how to ensure that the camera and imu readings are synchronized? Can you please clarify this?

I am using a pointgrey chameleon3 camera and a pixhawk with px4 firmware running in it. I don't think I got the correct results after calibrating. I say so, because when I use this result of calibration with VINS Mono, the output is unstable.

[ZacharyTaylor]

The easiest way to see if your sensors are both being timestamped without an offset between them is just to run kalibr with the --time-calibration flag and see how large the estimated offset is.

When I have worked with the PX4 I have always found its timesync to be very good and never needed to modify it. The chameleon 3 however is another story, we have our own fork of the driver where we have done some of our own timestamping (see the pull request here https://github.com/ethz-asl/pointgrey_camera_driver/pull/5/files). However, this will still leave a static time offset (from experience this is usually around ~20 ms for these cameras) which is hopefully small enough for most applications.

Ideally you would use the px4 to trigger the chameleon so you know exactly when the images happened.

Also from memory VINS Mono only needs a rough guess to the extrinsics and can estimate them online, have you tried doing this, as if this is still unstable it would probably indicate a timestamping issue. Though keep in mind the PX4 doesn't have the best IMU and this will limit the quality of the results of any visual-inertial estimation strategy.

[SubMishMar]

Thanks for the response, I will try what you mentioned here and get back to you. Is it necessary to use your version of point grey driver??

Besides this, since you have worked with PX4, I wanted to know how you obtained the following parameters for PX4:

accelerometer_noise_density:
accelerometer_random_walk:
gyroscope_noise_density:
gyroscope_random_walk:

As far as I know, PX4 uses more than one IMUs so I had to look into the datasheets of all of them, yet I could not find these params, may be they are called something else in the datasheet. So I used a matlab package available in fileexchange to find these parameters and that package threw at me some results, although I am not sure about the veracity of those.

To conclude, I want to use Kalibr to get a near accurate estimation of camera-imu extrinsic calibration so that I do not have to depend too much on the VINS Mono calibrator because it is kind of tedious to give my drone the "SLAM Wiggle" everytime before I start flying to collect data.

I have also attached a pdf file of the latest calibration I did, for your kind perusal.
Thanks.
[
report-imucam-mediasubodhElementsPhDkalibr_calibration2016-02-11-10-43-37_clipped.pdf
](url)

[ZacharyTaylor]

It depends how accurate you need to get, the standard ros driver just timestamps using ros::time::now() (https://github.com/ros-drivers/pointgrey_camera_driver/blob/master/pointgrey_camera_driver/src/nodelet.cpp#L494)

@HannesSommer actually did an experiment with the cameleon 3 and got this result

The red is what you will get with the current driver, green is what translating the hardware timestamps gives you (our driver) and blue is the level of error you can get if you go the extra mile and setup a triggerline from your microcontroller.

[SubMishMar]

Okay cool! Let me run kalibr with --time-calibration flag and see if the delay is acceptable.

[SubMishMar]

I want to confirm something else,

In the yaml file for april tags, there are parameters like :

tagSize: 0.088 #size of apriltag, edge to edge [m]
tagSpacing: 0.3 #ratio of space between tags to tagSize

Does tagSize mean the length of a side of an apriltag or is it the distance between respective edges of adjacent tags?

[ZacharyTaylor]

I am actually not 100% sure, I will measure one of our ones tomorrow and let you know.

[SubMishMar]

Please do! Thanks 👍

Btw, the optimization fails when I ran the code with --time-calibration flag, so I increased the timeOffsetPadding from 0.02 to 0.09, still fails..

[hitimo]

Does tagSize mean the length of a side of an apriltag or is it the distance between respective edges of adjacent tags?

It's the length of the black square containing the unique april tag
https://drive.google.com/file/d/0B0T1sizOvRsUdjFJem9mQXdiMTQ/edit

[SubMishMar]

Thanks @hitimo !

[SubMishMar]

Is it necessary to maintain a constant frequency for camera and IMU topics? My frequency slowly varies around 30 Hz and 245 Hz for the camera and IMU respectively.

[SubMishMar]

@ZacharyTaylor using the --time-calibration flag doesn't help me to find the temporal calibration parameters, the program breaks down midway! Any comments about it?

[ZacharyTaylor]

If you read through the error I am guessing it is about the timeoffset-padding, by default kalibr only lets the two systems be out of sync by up to 10ms. Try increasing this to something like 100ms via --timeoffset-padding 0.1

[SubMishMar]

Yes that was the cause of error if I remember well. Okay I will try with 0.1 sec. Thanks again!

[bsbretly]

Hi @SubMishMar,

Could you please post your results?

In particular, these parameters -

accelerometer_noise_density:
accelerometer_random_walk:
gyroscope_noise_density:
gyroscope_random_walk:

Also, which PixHawk version are you using? I'm currently using the PixHawk 1.

[SubMishMar]

@bsbretly , I am sorry, can't do it for the next two months. I was using Pixhawk 2.