D415 RGB image distortion at 6fps, Out of sync with IR at 30 fps

[t0mlane]

Required Info
Camera Model	{ D415 }
Firmware Version	(2.17.0)
Operating System & Version	Ubuntu 18.04
Kernel Version (Linux Only)	(e.g. 4.14.13)
Platform	Rock64/ Ubuntu
SDK Version	{2.0}
Language	{python }

Hello,
I have RGB image distortion at low framerates (6 fps) at 640x480 resolution due to motion, not lens distortion (see below, vs. IR which appears non-distorted) and have run out of ideas.

Waving hand side-to-side:

Oddly, this seems to rectify itself when using higher framerates and saving every nth image to simulate 6fps (every 5th at 30fps, for example). This is good, although the RGB and IR images are out of sync with the RGB appearing to "drop" frames (e.g. images appear to have been taken at identical moment but don't have corresponding filenames).

Above shows filenames which correspond to the same moment, yet are different. Strangely, the "correct" image seems to have been captured, although does not have the correct filename, yet this is only correct for a few frames, not all of them.

I have also tried adding a depth stream and using the align() function to align to the colour stream. I thought this would make the depth and IR hardware synced and the colour then aligned to the depth, this, however does not work.

My code is below, i have tried with different resolutions and framerates but none work. My project demands synced images. Unless anyone can see anything obviously wrong with the code below, I think this is a software problem.

Regards,

Tom.

`#!/usr/bin/env python3
import pyrealsense2 as rs
import numpy as np
import cv2
import time

#constants
count=0
x=640
y=480
framerate=30
ir_exposure=10000			# IR and RGB shutter speeds.
colour_exposure= 100			# Note 10,000=1 second in RGB but 
iteration=0				# IR is in us (e.g. 1,000,000
preset=1				# =1 second)
value=0
value_colour=0
value_ir=0
how_many=300				# 300 frames with every 10th being 						# saved

config=rs.config()
pipeline=rs.pipeline()
config.enable_stream(rs.stream.color, x, y, rs.format.bgr8, framerate)
config.enable_stream(rs.stream.infrared,1, x, y, rs.format.y8, framerate)
config.enable_stream(rs.stream.depth, x, y, rs.format.z16, framerate)
profile= pipeline.start(config)
dev=profile.get_device()

colour_sensor=dev.query_sensors()[1]	# Setting IR and RGB shutter speeds
ir_sensor=dev.query_sensors()[0]				
ir_sensor.set_option(rs.option.exposure,ir_exposure)		
ir_sensor.set_option(rs.option.laser_power, 0)			
ir_sensor.set_option(rs.option.gain, 1)
colour_sensor.set_option(rs.option.exposure, colour_exposure)
colour_sensor.set_option(rs.option.gain, 1)

align_to = rs.stream.color		# Depth stream and alignment
align = rs.align(align_to)		# added to try and force 		
print('We are off!')			# syncronicity, no luck.
start=time.time()

while count<how_many:
    frames=pipeline.wait_for_frames()		# Should wait for a coherent 
    aligned_frames = align.process(frames)	# set of frames, thus 
    color_frame = frames.get_color_frame()	# be in sync?
    infrared_frame = frames.get_infrared_frame(1)
    depth_frame=frames.get_depth_frame()
    if not color_frame or not infrared_frame: # or not depth_frame: 
        continue

    test=count/10 
    testicle=test-int(test)       # Save every 10th image in this example.

    if testicle==0.0:

        iteration = iteration + 1

        if iteration > 6:		# Gain values for IR and RGB 
            iteration=0			# increased by a 10th 				 
            if preset>10:		# every 6 saved images
                preset=0

            preset = preset + 1
            range_colour = colour_sensor.get_option_range(rs.option.gain)
            range_ir = ir_sensor.get_option_range(rs.option.gain)
            value_colour= (range_colour.max/10)*preset
            value_ir = (range_ir.max/10)*preset
            colour_sensor.set_option(rs.option.gain, value_colour)
            ir_sensor.set_option(rs.option.gain, value_ir)
            print("\n colour gain= %g ir gain= %g \n" % (value_colour, value_ir)) 


        rgb=np.asanyarray(color_frame.get_data())
        ir=np.asanyarray(infrared_frame.get_data())
        print('Barp!')
       # np.save("rgb%g gain= %g.npy" % ((count+100), value_colour), rgb)
       # np.save("rgb%g gain= %g.npy" % ((count+100), value_colour), rgb)
        cv2.imwrite("rgb%g gain= %g.jpg" % ((count+100), value_colour), rgb)
        cv2.imwrite("ir%g gain= %g.jpg" % ((count+100),value_ir), ir)
    # Apply colormap on depth image (image must be converted to 8-bit per pixel first)
    else:
        print('chugga chugga')
    count=count+1
    cv2.waitKey(1)

end=time.time()
difference=end-start				# Time taken for operation.
print(difference)
pipeline.stop()

`

[dorodnic]

Hi @t0mlane
Could you please check the value of color_frame.get_timestamp_domain()?

[t0mlane]

Hi @dorodnic

Attached are the colour and IR timestamps and difference for several settings:
480p no skips 6fps
480p no skips 30fps
480p 30fps saving every 5th (simulating 6fps, no distortion)
720p no skips 6fps
720p no skips 30fps
720p 30fps saving every 5th (simulating 6fps, no distortion)

The most interesting results are at 6fps with no skips for both resolution where there is a uniform difference between IR and RGB of ~14 ns indicating they are in sync for both 480p and 720p. The rest of the examples using 30 fps have very variable differences between IR and RGB timestamps.
Timestamp_data.zip

480p, 6fps, no skips:
RGB IR Difference
1.54878e+12 1.54878e+12 -180.228
1.54878e+12 1.54878e+12 -14.2993
1.54878e+12 1.54878e+12 -14.0435
1.54878e+12 1.54878e+12 -14.0522
1.54878e+12 1.54878e+12 -14.0645
1.54878e+12 1.54878e+12 -14.1438
1.54878e+12 1.54878e+12 -14.0632
1.54878e+12 1.54878e+12 -14.3521
1.54878e+12 1.54878e+12 -14.0649
1.54878e+12 1.54878e+12 -14.0549
1.54878e+12 1.54878e+12 -14.0571
1.54878e+12 1.54878e+12 -14.0862
1.54878e+12 1.54878e+12 -14.0144

720p 6fps no skips
1.54878e+12 1.54878e+12 -22.2007
1.54878e+12 1.54878e+12 -22.1941
1.54878e+12 1.54878e+12 -22.2107
1.54878e+12 1.54878e+12 -22.2397
1.54878e+12 1.54878e+12 -22.2363
1.54878e+12 1.54878e+12 -22.2791
1.54878e+12 1.54878e+12 -22.2371
1.54878e+12 1.54878e+12 -22.2007
1.54878e+12 1.54878e+12 -22.2612
1.54878e+12 1.54878e+12 -22.2034
1.54878e+12 1.54878e+12 144.554
1.54878e+12 1.54878e+12 978.426
1.54878e+12 1.54878e+12 1159.24
1.54878e+12 1.54878e+12 1311.99
1.54878e+12 1.54878e+12 1478.8
1.54878e+12 1.54878e+12 -22.2712
1.54878e+12 1.54878e+12 -22.2427
1.54878e+12 1.54878e+12 -22.2507
1.54878e+12 1.54878e+12 -22.239
1.54878e+12 1.54878e+12 -22.2483
1.54878e+12 1.54878e+12 -22.2317
1.54878e+12 1.54878e+12 -22.2249
1.54878e+12 1.54878e+12 -22.2734

480p/720p at any 30fps config:
1.54878e+12 1.54878e+12 -103.927
1.54878e+12 1.54878e+12 -0.559326
1.54878e+12 1.54878e+12 -34.3721
1.54878e+12 1.54878e+12 -101.208
1.54878e+12 1.54878e+12 16.9355
1.54878e+12 1.54878e+12 -117.39
1.54878e+12 1.54878e+12 10.3049
1.54878e+12 1.54878e+12 -23.2385
1.54878e+12 1.54878e+12 -90.3479
1.54878e+12 1.54878e+12 -123.98
1.54878e+12 1.54878e+12 -5.75635
1.54878e+12 1.54878e+12 -39.8926
1.54878e+12 1.54878e+12 -106.424
1.54878e+12 1.54878e+12 -140.107
1.54878e+12 1.54878e+12 -22.1572
1.54878e+12 1.54878e+12 -122.688
1.54878e+12 1.54878e+12 -12.4358

[dorodnic]

Hi @t0mlane
I'm still missing the timestamp domain. There is an additional function on the frame object that returns either "System Time" or "Hardware Clock". When hardware clock is available (requires Kernel 4.16+ or Kernel patches), we will use it, but if it is not we will fall back to system clock. When this happens, the chances of proper optical sync get reduced significantly.
Since the platform you specified is Rock64, I suspect Kernel patches were not applied, and hence hardware timestamps are not available. If it is the case, a relatively easy work-around is to add -DFORCE_LIBUVC=true to your CMake command and rebuild the library.
I can't say for sure if this is the root-cause, but that's a start.

[RealSenseCustomerSupport]

---

Hi t0mlane,

Not sure if there are any updates regarding to what dorodnic had mentioned.
Or anything else you still need help?

Thanks!

[RealSenseCustomerSupport]

---

Hi Hi t0mlane,

As suggested by dorodnic, you may want to provide timestamp domain info (something like: rs.frame.get_frame_timestamp_domain(frame) ).
Or let us know if you still need help on this one.

Thanks!

[RealSenseCustomerSupport]

---

Hi t0mlane,

Wonder if there is any updates? If nothing else is needed, this one will be closed.

Thanks!

[t0mlane]

Hello,

Sorry for my silence. I have since tested with an UP board as well as the rock64. The rock returns an error when calling timestamp_domain. As the kernel is 4.4... there are no kernel patches and the hardware clock is not used. Rebuilding with libuvc=true did not make any difference.

I tested the UP board (with its in-built RTC) and it appears to return frames in sync with difference in timestamps less than 1 us and can be confirmed visually between IR and RGB images. However, "smearing" of RGB (NOT IR) frames still occurs at 6 fps and 15 fps but is rectified at 30fps· This would suggest the on-board RTC is being used, especially as the kernel is 4.15.

This is OK although the system is unstable, particularly at boot. This is despite rectifying issues with identifying as USB3 (I used a micro b to USB A female adaptor) and writing a C++ script to reset the USB hub (removes get_xu(id=7) failed errors) and trying my best to ensure adequate power and cooling. This means the camera will consistently start and images will be saved but will often switch off for no reason or "frames not recieved in 5000".

For this reason, despite the UP giving undistorted RGB, IR and depth images at 30fps (saving every 5 to simulate 6 fps) I have had to move back to the rock64 and accept only colour images as it works every time, indefinitely.

I am unsure if this is an Realsense or UP issue but find it ironic I have found a community-supported, ARM system more stable than the more "recommended", established, powerful, intel-based option.