Align color frame and depth frame which are not a composite frame

[klausondrag]

---

Required Info
Camera Model	D435
Firmware Version	(not sure because the camera is currently not in my possession)
Operating System & Version	Win 10 / Linux (Ubuntu 18.04)
Kernel Version (Linux Only)	5.3.0-46-generic
Platform	PC
SDK Version	pyrealsense2==2.34.0.1470
Language	python

Issue Description

I've created a dataset with the Realsense D435. I've stored a single frame with a color and a depth stream in a ros bag file. I've also managed to read from the bag file again and continue processing such as aligning the depth channel and coloring it. This worked on my local PC with Ubuntu.

However, something was wrong with the setup on the actual Windows PC. It still stored a bag file but reading from this ros file with the color and the depth stream enabled never returns from wait_for_frames. Same thing happens with polling instead of waiting for a frame. In the realsense viewer, this problem shows as only the depth frame is loading. The color stream appears after a minute or so, before that it's white. I've managed to work around it in python by only enabling one stream and polling for up to 30k times. If this does not work, I re-create the pipeline from the file and try again. This part is not the main issue for now as at least I can get the images from the bag file. Now, my main focus is aligning them.

This workaround with polling works. This way, I receive the color frame and the depth frame. Usually, I would align a frame like this

import pyrealsense2 as rs
aligner = rs.align(rs.stream.color)
frame = ...
aligned_frame = aligner.process(frame)

However, due to my workaround, I have two separate frames. Note that they are coming from a regular single frame, and that one was possible to align before storing it as a ros bag file. I have not found a way to either pass both the color and the depth frames to the aligner, or to create a new composite frame from the two. Since I've already taken quite a few pictures, I would really want to process these images. Re-taking them is not possible because it was a one-time situation.

So ultimately, how do I align these two streams?

[MartyG-RealSense]

From the end of your question, I interpret that your situation is that you have a collection of image files (not live camera data) and you want to upload the image files and align depth to color. And you need to do so with Python. Is that correct, please?

If so, the discussion in the link below regarding doing depth to color alignment in Python with image files will hopefully be helpful to you.

#1274

[klausondrag]

It is sort of correct. I can have image files but I also have the ros bag file which means I can start streaming it from it with a pipeline. The issue is that these files are apparently broken so I cannot retrieve a single frame with the color and depth stream. Instead, I have to retrieve to separate frames with a single frame each. They are, however, from the same original frame because there is only a single frame stored in the ros bag file.

Yes, I would strongly prefer a solution in Python.

I looked at the linked issue. zivsha proposes three solutions:

1. rs2::software_device but that is not available in Python. I will have to investigate but given that also realsense-viewer has some issues with the file, I'm not overly optimistic.
2. Use rs2_deproject_pixel_to_point, rs2_transform_point_to_point, rs2_project_point_to_pixel to implement the alignment myself. This seems like quite some work and might introduce bugs.
3. Use bag files - This is what I'm already doing.

My hope would be that either I can create a composite_frame myself in Python and filling the data with the values from the other two frames. Is that possible? Or can we pass two frames to align?

[MartyG-RealSense]

The subject of creating framesets manually is discussed in the link below,

#5847

[MartyG-RealSense]

Case closed due to no further comments received.