[Question] Getting aligned depth from incomplete framesets

[prybicki]

---

Required Info
Camera Model	D415
Firmware Version	05.12.10.00
Operating System & Version	Linux
Kernel Version (Linux Only)	4.4.40
Platform	PC
SDK Version	2.41.0
Language	C++
Segment	Robot

Issue Description

In my application, I need to use a large number of cameras which puts a high load on the USB bus. I am fine with some frames being dropped. I observed that if I use a single camera, I receive almost always a complete frameset (color, depth, infra1, infra2). However, the more cameras are streaming, the more fragmented the frameset gets and I often receive a frameset containing only a single frame (only depth, only color, etc.). This seems reasonable to me because a shorter packet is easier to transfer over a congested bus.

I care most about the aligned depth to color/infra and I'm using the rs2::align filter to get it. However, based on my understanding of its API frameset process(frameset frames), the filter to work correctly needs to receive a frameset containing (depth) + (the frame/stream it has to align the depth to). This, unfortunately, means that I cannot use the incomplete framesets and they are being completely wasted and, what's worse, the latency to get an aligned depth frame increases. I don't care about metadata and synchronization very much - cameras are looking at a static scene (most of the time).

Concluding, I'd like to know:

1. Is my understanding of the way the rs2::align filter works correct?
2. Is there a way to synthesize a complete frameset from incomplete framesets?
3. Or perhaps there is a way to produce an aligned depth frame without having a complete frameset?

[MartyG-RealSense]

Hi @prybicki Before addressing your questions about incomplete frames, may I begin by checking whether some alternative approaches might suit your project:

1. The USB standard allows up to 5 USB hubs to be attached to the same computer. So if you are not already using hubs, then spreading your large number of cameras over multiple hubs should allow them to be handled by a greater number of USB controllers.

2. You could take an ethernet networking approach, with each camera connected to an inexpensive Raspberry Pi 4 board and all of the boards connected to a central computer. This method is detailed in Intel's open-source ethernet networking white-paper.

https://dev.intelrealsense.com/docs/open-source-ethernet-networking-for-intel-realsense-depth-cameras

3. If the scene being observed is relatively unchanged over time and you do not mind the possibility of dropping frames, then you could program an application to only take a single capture from all of the cameras every 'x' number of minutes.

[prybicki]

Thank you @MartyG-RealSense for the suggestions.

1. I am already using USB hubs.
2. I may try out the network solution in the future, but I am still interested in hearing how to approach the problem using a single machine.
3. I assume that by taking a single capture you mean turning the camera on, receive a frame, turn it off. Unfortunately, this method has a too high latency (around a second). I know that having a static scene and complaining about the latency seems to be contradictory, but I actually know exactly when the scene changes and need to take a picture just right afterwards with a minimal delay to speed up further processing.

And by the way @MartyG-RealSense why do you say that 5 is the max number of hubs? I cannot confirm this information anywhere. It's okay to have up to 7 hubs connected in series, but I think the only limit to the number of parallel connections is the number of ports in the machine.

[MartyG-RealSense]

I checked the specification. Yes, you're right, the limit is 5 hubs daisy chained together (hence the phrase '5 tiers deep' that is ingrained in my mind) but you can have a total of 7 hubs due to the total being the number of non-root hubs in the chain plus 2. Apologies for that!

If latency is an important factor to you then captures per 'x' minute likely would not be suitable for you, as other RealSense users carrying out the same kind of periodic capture have also tried to avoid this method if they needed to capture at a precise moment.

How many cameras are attached to the same computer, please?

[prybicki]

@MartyG-RealSense We're talking here about 10-20 cameras, connected via USB hubs and PCIe USB expanders with a separate controller per USB port.

[MartyG-RealSense]

It is ideal to have a dedicated USB controller for each port like your PCIEe expander setup does, as that is optimal for performance. Connecting the camera to a hub is also fine though as long as the overall streaming demand is within the hub's bandwidth capabilities. Intel's multiple camera white paper provides tables with estimates of multi-camera bandwidth usage for direct connections and for hubs.

https://dev.intelrealsense.com/docs/multiple-depth-cameras-configuration#section-2-multi-camera-considerations

If you are streaming more than one type of sensor on a camera then you can reduce the possibility of dropped frames by altering the frame queue size to introduce more latency. The default frame queue size is '1', with a value of '2' suggested for two streams (e.g depth and color). I have seen higher values than that used (such as '5') but this apparently carries a risk of the RealSense user choosing a value that may accidentally break the streaming.

https://github.com/IntelRealSense/librealsense/wiki/Frame-Buffering-Management-in-RealSense-SDK-2.0#latency-vs.-performance

The specification of the computer hardware that you are using may have a bearing on performance when you have multiple cameras attached that are simultaneously active. For example, an original-model Up Board can handle one 400 Series camera, whilst a more powerful Up Squared board model can handle two. During Intel's 2018 RealSense seminar about multiple cameras, a spec of an Intel Core i7 processor was suggested for a 4-camera setup. Since then, more powerful processors have become available.

In regard to alignment, it is worth noting that infrared is already aligned to depth because they originate from the same sensor.

#5093 (comment)

This is also described in point 2 of the section of Intel's camera tuning paper linked to below:

https://dev.intelrealsense.com/docs/tuning-depth-cameras-for-best-performance#section-use-the-left-color-camera

In regard to greater control over framesets, the discussion in the link below is worth reading.

#5847

[MartyG-RealSense]

Hi @prybicki Do you require further assistance with this case, please? Thanks!

[prybicki]

Hi @MartyG-RealSense, I think #5847 answers my question, thank you!

[MartyG-RealSense]

You are very welcome @prybicki - thanks for the update!