Camera calibration data gets occasionally lost. How to fix this or how to reload previously saved calibration data in C?

[for-no-one]

Required Info
Camera Model	D435 / D435i
Firmware Version	05.11.15.00
Operating System & Version	Win 10 & Linux (Ubuntu 18.04)
Kernel Version (Linux Only)	4.15.0-47-lowlatency
Platform	PC
SDK Version	2.29.0-0
Language	C
Segment	Robot

Issue Description

I've had problems with the depth accuracy at longer distances, but I'm able to get the desired accuracy with the dynamic calibration. However, after working well for a few days, calibration seems to revert back to the factory calibration. I have not yet been able to find steps to reproduce this issue. There might be something wrong with my code because the amount of lines becomes quite big if you're using multiple streams and post-processing filters and there are not many examples in C. Is it possible that issues in my code could somehow corrupt the calibration info? Some more background info: I'm using 3 camera setup and I occasionally experience problems where one of the cameras is not recognized by the RealSense Viewer and I have to unplug and plug the camera back into the PC for it to start to work.

Because I can't give steps to reproduce the issue, I understand that this is almost impossible to solve, so I'm asking if there is a way to upload the XML file that I can get out of Intel.Realsense.CustomRW tool in C? If yes, then I could upload the calibration data every time I connect to the cameras until I find the actual root cause. Can it be done just with the base library without using librscalibrationapi which I'm not familiar with?

Many thanks in advance.

[dorodnic]

Hi @for-no-one

Please take a look at rs_device.hpp#L287-L528, in particular get/set and write calibration table functions. These are functionally equivalent to librscalibrationapi and using the XML.

Please let us know if you find out the root cause of the issue. Calibration can get worse due to mechanical movement of the lenses, but the camera should not implicitly and permanently revert to factory calibration during any normal flow.

[for-no-one]

Hi @dorodnic

Thank you! I was looking at old include files that I had downloaded in October so that's why I couldn't find a way to do this. My mistake! I'll try to implement this and let you know how it goes. To verify that this issue is truly caused by the camera losing its calibration I'm going to try to reproduce the problem and then reload the previously saved XML to see if it fixes the issue. This way I could rule out the mechanical movement of the lenses. I've seen some weird issues previously when I was using some methods in an incorrect way, so I will probably look into that if I'm able to verify that reverting is actually happening. I will let you know if I find anything!

[for-no-one]

Hi @dorodnic

I tried using get/set calibration table functions. I updated the SDK to 2.31 and camera firmware to 05.12.01.00 and then tried reading calibration table and sending it back to the camera with rs2_set_calibration_table, but I'm getting this error:

rs_error was raised when calling rs2_set_calibration_table(calibration:000002D8D7B3AFE0, device:000002D8E4931180):
hwmon command 0x51 failed. Error type: Invalid Command (-1).

Here's my C code:

rs2_error* e = 0;
rs2_raw_data_buffer* buff = rs2_get_calibration_table(device, &e);
const unsigned char* arr = rs2_get_raw_data(buff, &e);
int len = rs2_get_raw_data_size(buff, &e);
rs2_set_calibration_table(device, arr, len, &e);

Am I doing something wrong?

[dorodnic]

I suspect you need to be calling set calibration during streaming

[for-no-one]

@dorodnic I was now able to verify that the problem is indeed with calibration being reset to the factory values. When the issue occurs, I'm able to solve it by reuploading the XML file that I saved after calibrating the camera. When I compare XML files, RotationLeftRight, RotationLeftRGB, TranslationLeftRight and TranslationLeftRGB are the parameters that get changed when the issue occurs. I never have to recalibrate the camera with the pattern again, so it's not the mechanical change of the distances between the lenses. I've been able to reproduce it by running SDK's multicam example with a bash script that starts and then almost immediately kills the multicam executable in a loop. It also happens when I don't violently kill the multicam executable, but instead close it normally after a few seconds of streaming, but it's much more difficult to reproduce with a script. With my own more complicated code the issue happens much more often in normal use, but I'm still working on narrowing down the problematic part of my code.

This is my bash script which usually reproduces the issue:

#!/bin/bash
for i in {1..100}
do
    ./rs-multicam &
    last_pid=$!
    sleep 1
    kill -KILL $last_pid
done

My test setup for this was Ubuntu 18.04 PC, SDK version 2.29.0 and 2 cameras: D435i and D435, both running 05.11.15.00 FW. D435i is the one losing calibration, I have not calibrated the other camera, so I don't know if the issue happens with it as well.

[for-no-one]

@dorodnic I've verified that set calibration is working during streaming, so that's good! Thank you. Are you able to reproduce the original issue with my example? Or do you think you'll be working on it? I'm occasionally seeing some other weird behavior (sometimes I lose the ability to connect to the camera) and I'm wondering if it's caused by this same weird behavior that causes the calibration to reset.

[dorodnic]

Hi @for-no-one
Thank you for the reproduction scenario. I tried to reproduce but with no luck (1100 iterations and the calibration did not change). I hope someone from @RealSenseCustomerSupport could also try to reproduce.

[for-no-one]

Hi @dorodnic
Thank you for trying to reproduce. I just tried again with the latest SDK and firmware (05.12.02.100). All 3 cameras are now D435i and I'm on Ubuntu 18.04. I saved the original calibration data, calibrated all cameras, saved new calibration data and ran my bash script. One of the three cameras went back to the original calibration. The data from this camera is shown at the bottom in rs-multicam example, so it's probably the last camera of the three that is connected.

When I'm comparing calibration data, I'm mainly checking TranslationLeftRight field. I'm also seeing the impact of losing calibration in my application and I'm able to fix it by reloading the previously saved calibration data, so I'm pretty sure that the calibration loss actually happens. My PC is Advantech ARK-2250.

[RealSenseCustomerSupport]

---

Hi @for-no-one,

When you have reproduced the issue few times and one of the 3 cameras went back to the original calibration, is that the same camera each time?

Thank you!

[for-no-one]

Hi @RealSenseCustomerSupport ,

In this set of cameras, yes it was the same camera each time. But I've seen this with at least three different cameras. I have several robots with three cameras on each and this happens on every robot. So I don't think that this is a case of a single faulty unit.

[RealSenseCustomerSupport]

---

@for-no-one,

Do you still need help with this question?

[for-no-one]

@RealSenseCustomerSupport

Yes, of course, it hasn't been solved.

[RealSenseCustomerSupport]

---

@for-no-one,

Do you experience the same issue using 2 camera setup?

[for-no-one]

@RealSenseCustomerSupport

Yes. Just ran a test with two cameras: Saved xml calibration files of both cameras, ran my script once, checked the calibration files, one of them was changed.

[for-no-one]

I'm now suspecting that this has something to do with issue #4158
I had the same issue with cameras randomly crashing upon startup and I think that those random crashes were also corrupting calibration files. The workaround suggested by @kroq-gar78 fixed the crashing issue and I will try to see if it also solves the calibration issue. It seems that pipelines are currently broken and should not be used with multi camera setups.

[RealSenseCustomerSupport]

---

@for-no-one,

Based on your last comment, it seems like all that time we had only partial description of the issue scenario. Do you experience the above mentioned calibration issue only after the camera crashed upon startup or you see the camera loosing the calibration even when it never crashed after being calibrated?

[for-no-one]

It was just a guess that this might be related, I don't know when and what exactly is happening. It always happens within a couple days of using cameras and there have been crashes during that time because cameras are unstable when using pipelines, but I don't know when exactly the corruption occurs. In my original post I mentioned that occasionally I have issues with cameras not being recognized, but I should have dived deeper into those issues.

I now added comparison to the original calibration file to rs-multicam.cpp to see when exactly it gets corrupted and ran rs-multicam a few times. This was the result:

1. successful start, calibration unchanged
2. successful start, calibration unchanged
3. successful start, calibration corrupted
4. crash on start: hwmon command 0x10 command failed
5. successful start, calibration corrupted

So in this case it didn't happen after the crash but maybe something happens that corrupts the calibration and causes the crash later. Someone in #4158 mentioned that they are experiencing the crashing issue on Intel NUC. You probably have access to those, so it might be worthwhile to test with 3 or 4 cameras, Intel NUC and multicam example if you want to solve the crashing issue and possibly this issue as well. With the workaround of not using pipelines and forcing calibration on the start I'm probably fine for now, but I wanted to point out that this issue still exists.

[for-no-one]

Okay, ran more tests now and confirmed that corruption of calibration files doesn't happen when I'm using sensor objects instead of pipelines. The root cause is better described in #4158 and this is just a side effect.

[RealSenseCustomerSupport]

---

@for-no-one,

Just to make sure we understood you correctly: when you use rs2::stream instead of rs2::pipeline​ the corruption of calibration files doesn't occur and your setup can normally work. Please let us know if that's correct.

Thank you!

[for-no-one]

@RealSenseCustomerSupport I use rs2::sensor to open and start the streams as implemented in this comment: #4158 (comment)

When I do that instead of using pipelines the corruption of calibration doesn't occur.

[RealSenseCustomerSupport]

---

@for-no-one So I understood that the issue resolved after use rs2::stream instead of rs2::pipeline.
Any other questions? Looking forward to your update.Thanks!

[for-no-one]

Hmm, a question... Are you going to fix the issue with pipelines? If not, I'd recommend modifying the multicam example to use streams because at least I, for example, almost deemed this product unusable in production because of the startup and calibration loss problems and I would imagine that this will happen to other people as well. You're free to close this ticket, I'm happy with my current workaround, but I didn't want to close this myself to make sure that you're aware that the problem still exists.

[RealSenseCustomerSupport]

---

@for-no-one Thank you for your confirmation! Will close this accordingly.