sync timestamp

[Miranda0920]

Hi,
I am new to gopro. I wonder how to sync the timestamp of camera and IMU in HERO7.

[dnewman-gpsw]

The data is effectively frame rate in HERO7, no timestamps are needed for most applications. We added µs timestamps in HERO8 for some sub-frame precision. You can get a little more precision using the sample code call GetGPMFSampleRate(). This will return the sample rate for any stream and a high precision in and out sample time, the is relative to video.

[Miranda0920]

Thanks for your reply! I want to use kalibr for joint calibration of IMU and camera. Thus I try to get the timestamp of each frame and each IMU data. Does gopro HERO7 record the collection time of each IMU data? Can I only calculate the timestamp based on the return value of the function GetGPMFSampleRate()?

[dnewman-gpsw]

None of our cameras record the time of every IMU sample, as that would double the payload size. Also there is the question of the validity of timestamps and their precision. Instead you can compute, more accurately in many designs, the relative timing of all samples after capture. GetGPMFSampleRate() is a demo of how this can be done. In HERO8 we did get hardware timestamps that are precise enough, so we include one timestamp per payload. GetGPMFSampleRate() will use timestamps if available, or compute them if not.

[rlamarche]

Hi, I have developed a ros node to read GOPRO5 mp4 files and publish imu and images with timestamp headers in topics.
My formula was, for each payload (duration = 1040ms) : imu_ts = index * duration / nb_samples.
I tried to calibrate with kalibr (followed all the steps, camera calibration, imu calibration, camera / imu calibration) but with no good results (openvins or vins-mono slam were not working correctly).

Just today after reading this issues comments (thanks @dnewman-gpsw ), I've improved my code to use GetGPMFSampleRate start/end/rate informations.

Now I do the following :

• calling GetGPMFSampleRate with precise flag to get average sample rate, start and end timestamp
• interpolating the imu data with global index, using the calculated rate and adding the start timestamp (imu_ts = global_imu_index * rate + start_ts)
• publish video and imu to distinct topics with correct header in the right order
  (I use the video timestamp found in the mp4 data).

Because on gopro 5 the gyro is 400hz and accl 200hz, I use 400hz and publish 2 times the same accl.

Kalibr is currently still running to calibrate the camera/imu. It is very long and I'm not sure that my dataset has enough quality.
When finished (it no error, the last time I had to increase the time shift tolerance), I will try again using a visual/inertial slam algorithm, but I'm not very confident.

I used this project to calibrate the noise model of imu with 1 hour video : https://github.com/gaowenliang/imu_utils

[dnewman-gpsw]

You are doing the correct timestamp extraction, however this IMU data wasn't intended for this type of post processing. The latest cameras run the IMU at much higher frequencies to compute the Hypersmooth™ stabilization in camera, and only uses the low frequency IMU data within the mobile editing software to automatically find the good bits. However, Reelsteady Go (https://www.reelsteady.com/) uses this data with excellent results, so good GoPro bought the company. So it is possible.

[rlamarche]

My calibration ran fine (after updating the noise model because my first was too optimistic), but I don't get good results with the visual / inertial slam. Sometimes it works for a few seconds, but if I move a little faster, in a location with not so much keypoints (I stay indoor), it diverges.
The best would be able to do it in realtime, so I could better understand the way it works. But the gopro is not made for this.
By the way I learned a lot of stuff doing theses experiments, and that's was the main goal of these.
Thank you for your support.

[yayafu666]

@rlamarche You really did a good job. I am also new to gopro. When I tried to calibrate my gopro HERO8 with kalibr, I found it really difficult to create an IMU rosbag through this gpmf-parser.
@rlamarche Would you please teach me how to do it or provide your code on your github?
Thank you very much!

[rlamarche]

hi @treenewbee213 , thank you. What do you plan to do with the gopro & kalibr ? Post stabilization ?

Yes I can post the code on github, I will do it in the coming days. It's a ROS node so if not you should get familiar with ROS before.

Regards,
Romain.

[yayafu666]

I plan to run a VIO project with my handheld gopro, so it's crucial to calibrate it first.
@rlamarche I'm looking forward to your excellent work!

[joshi-bharat]

@dnewman-gpsw I am trying to use STMP from gpmf data to synchronize video. Is the microsecond timestamp with respect track start?

[dnewman-gpsw]

This is a microsecond timestamp, but for the GPMF streams relative to each other, not directly capture start. The first timestamp in CORI (HERO8 & 9) or SHUT (HERO7) is the timestamp for the first video frame. So the timestamp on any stream, can be converted to capture start timestamps by substracting the first TS in the CORI or SHUT streams.

[joshi-bharat]

@dnewman-gpsw I will be doing something like this.
stmp_accl = get_stamp(STR2FOURCC("ACCL")) //for all payload
stmp_cori = get_stamp(STR2FOURCC("CORI")) // for the first payload only

stamp_accl_sync = unix_start_time + stmp_accl - stmp_cori // unix time for absolute reference

Does this looks working?

Thanks.

[dnewman-gpsw]

No idea why you are adding unix time, but it might be what you need.

[joshi-bharat]

@dnewman-gpsw it's because I am using ROS and it requires epoch time. If the relative time is correct, it should not matter yeah. or I can just use video creation time.

[dnewman-gpsw]

I don't if you can used video create time, it is not intended to be precise (not at video precision.) Your needs are application specific, but you should have all the info on the timestamps necessary.

[joshi-bharat]

@dnewman-gpsw thanks to you I was able to run visual-inertial odometry using GoPro IMU data.
https://youtu.be/iNbRslYIQnA

[rlamarche]

@joshi-bharat nice! I did try the same thing as you with vins mono, but did not succeed. I was using a gopro 5 and the imu seemed to not be synchronized enough with the images. Well that was a good excuse to stop.
But you did it with a gopro 9, congrats! It is very interesting to know that it is possible.
In what the specs are different from a gopro 5, 6,7 or 8.
Or maybe I did something wrong.
The next step would be to do it in real time, I don't know if the gopro is able so send the imu data with the video stream at the same time, but that would be awesome.
A technology like webrtc would allow this, using datachannels for transmitting the imu data.
Keep the good work, and that reminds me that I have to publish my source code experiments before it is erased from my disk !

[rlamarche]

@treenewbee213 finally, here my source code : https://github.com/rlamarche/gopro_ros
Sorry for the delay, but now I have plenty of time in France, due to the lockdown ...

[joshi-bharat]

@rlamarche I was not able to do with GoPro5 as well. There is a start stamp for each payload in the new GoPros (GoPro 8 and 9). I used that to synchronize IMU data with images. I will also soon publish the code later. It needs some refactoring.

Please check the previous conversation on how to use that information.

[rlamarche]

Thanks for the information, indeed I might have missed the relevant informations in this topic. So I have to find a GoPro 8 at least if I want to continue/restart my experiments.
I was reviewing my code, it might need some refactoring as well.

[yayafu666]

@rlamarche Thank you very much! Actually I tried to solve the problem by ffmpeg and an offline rosbag creator, it works but still influenced by temporal misalignment. Your code really inspired me in some way, a nice job!

[nuyhead-hwang]

Hello @joshi-bharat

How to set the IMU noise parameters? (Bias random walk, accelerometer and gyroscope noise density)
Could you please explain the calibration process? (Did you use Kalibr? If so, did you use the rolling shutter camera calibration method?)

[joshi-bharat]

@vislero I calculated the noise parameters using https://github.com/rpng/kalibr_allan.
You need to save a bag file with no motion for around 4 hours and then calculate the allan plots.
I just the default camera-imu calibration from Kalibr.

[urbste]

Hi, I just saw the thread.
Do not know if this is still of interest for you guys (@treenewbee213, @vislero, @rlamarche) but I got VI ORB-SLAM3 running with a GoPro9 quite smoothly. Even with 25fps which should lead to relatively high rolling shutter distortions (which ORB-SLAM3 can not compensate for):
https://github.com/urbste/ORB_SLAM3/
https://www.youtube.com/watch?v=0wIqkUEjhiw

Even using the MaxLens mod in full frame fisheye and rapid movements: https://www.youtube.com/watch?v=Phw_OVP6sxI