Reducing fps for depth frames and timestamp questions

[czy10383]

Required Info
Camera Model	L515
Firmware Version	01.05.08.01
Operating System & Version	Win10
Kernel Version (Linux Only)	N.A.
Platform	PC
SDK Version	2.48.0.3381
Language	python
Segment	others

Issue Description

The framerate of the depth sensor in L515 is fixed at 30

However, I wish to process the depth frames at a lower fps e.g. 3 fps, it's to reduce computational load on the cpu when I transfer the program onto a raspberry pi later.

I tried the following first:
config.enable_stream(rs.stream.depth, 640, 480, rs.format.z16, 3)
but this results in an error

My current approach is to use the timestamp for every depth frame

May I ask the following:

1. is this way acceptable?
2. or are there other ways we can use to reduce the fps?
3. what is timestamp relative to?

Thank you

import cv2
import numpy as np
import pyrealsense2 as rs

if __name__ == "__main__":
    colorizer = rs.colorizer(color_scheme=0)
    pipeline = rs.pipeline()
    config = rs.config()
    config.enable_stream(rs.stream.depth, 640, 480, rs.format.z16, 30)
    profile = pipeline.start(config)
    depth_sensor = profile.get_device().first_depth_sensor()
    MAX_DIST = 5000
    MIN_DIST = 0
    colorizer.set_option(rs.option.max_distance, MAX_DIST / 1000.)
    colorizer.set_option(rs.option.min_distance, MIN_DIST / 1000.)
    prev_timestamp = 0
    period = np.floor(1000/3.)
    cv2.namedWindow('example', cv2.WINDOW_NORMAL)
    while True:
        frames = pipeline.wait_for_frames()
        depth = frames.get_depth_frame()
        if not depth: continue
        curr_timestamp = depth.timestamp
        elapsed = curr_timestamp - prev_timestamp
        if (elapsed > 0 and elapsed < period):
            continue
        else:
            print(f'{curr_timestamp}-{prev_timestamp}={elapsed}')
            prev_timestamp = curr_timestamp
        print(depth.frame_number)
        depth_colormap = np.array(colorizer.colorize(depth).get_data())
        # other processing omitted here #
        cv2.imshow('example', depth_colormap)
        key = cv2.waitKey(1)
        # Press esc or 'q' to close the image window
        if key & 0xFF == ord('q') or key == 27:
            cv2.destroyAllWindows()
            break
    pipeline.stop()

[MartyG-RealSense]

Hi @czy10383 An easier way to set a custom frame rate that is not normally supported is to only use every 'nth' frame (e.g every 5th frame out of 30 when using 30 FPS to simulate 6 FPS). So in the case of a target of 3 FPS, every 10th frame could be looked at. A Python script that demonstrates this technique is in the link below.

#3169

In regard to what the timestamp is relative to: this is a complex subject, but the information in the link below provides useful introductory information.

#2188

[czy10383]

Thank you @MartyG-RealSense !

After reading #3169 , I have changed my program to use frame_number instead, where I will process every 10th frame

This is a side question:

I observed the following when I tested this on a Raspberry Pi:

• the frame_number resets itself to 0 after a certain time, and it seems to happen once the LLD temperature goes up to 70 degrees celsius.
• the frame_number then increments in steps of 10 to about 300, and then it resets itself again to 0, and the cycle repeats itself.

I suspect the frame_number resets to 0 because the sensor temperature became too hot (> 70 degrees celsius), but I'm not sure if my hunch is correct? May I ask what other reasons could lead to the frame_number being reset to 0? thank you

import cv2
import numpy as np
import pyrealsense2 as rs

if __name__ == "__main__":
    colorizer = rs.colorizer(color_scheme=0)
    pipeline = rs.pipeline()
    config = rs.config()
    config.enable_stream(rs.stream.depth, 640, 480, rs.format.z16, 30)
    profile = pipeline.start(config)
    depth_sensor = profile.get_device().first_depth_sensor()
    MAX_DIST = 5000
    MIN_DIST = 0
    colorizer.set_option(rs.option.max_distance, MAX_DIST / 1000.)
    colorizer.set_option(rs.option.min_distance, MIN_DIST / 1000.)
    cv2.namedWindow('example', cv2.WINDOW_NORMAL)
    while True:
        frames = pipeline.wait_for_frames()
        depth = frames.get_depth_frame()
        if not depth: continue
        iframe = depth.frame_number
        
        # monitor sensor temperature every 10s
        if iframe % 300 == 0:
            lld_temp = depth_sensor.get_option(rs.option.lld_temperature)
            print(f'sensor temp = {lld_temp:.2f}')
        
        # check every 10th frame
        if iframe % 10 != 0:
            continue
            
        print(iframe)
        depth_colormap = np.array(colorizer.colorize(depth).get_data())

        # other processing omitted here #

        cv2.imshow('example', depth_colormap)
        key = cv2.waitKey(1)
        # Press esc or 'q' to close the image window
        if key & 0xFF == ord('q') or key == 27:
            cv2.destroyAllWindows()
            break
    pipeline.stop()

[MartyG-RealSense]

Reasons why a reset of the frame counter to zero can be triggered are described in the link below.

#7819 (comment)

The frame counter can also be reset in multi-camera hardware sync configurations if there is an electrostatic discharge (ESD) event in hardware sync cabling that does not have ESD protection components. This would not apply in your single-camera application though.

The recommended temperature guidelines for the L515 camera are shown in the table below, quoted from page 17 of the current edition of the data sheet document for the L515. They recommend that the camera be operated in a maximum ambient environmental air temperature of 30 degrees C and be exposed to a maximum camera housing temperature of 50 degrees C.

https://dev.intelrealsense.com/docs/lidar-camera-l515-datasheet

[czy10383]

thank you @MartyG-RealSense ! I will proceed to close this issue : )

[MartyG-RealSense]

You are very welcome, @czy10383 - thanks very much for the update!