trouble using pipeline callback feature with pyrealsense2

[sinback]

Required Info
Camera Model	D435
Firmware Version	05.11.01.100
Operating System & Version	Ubuntu 16.04
Kernel Version (Linux Only)	4.15.0-72-generic
Platform	PC
SDK Version	2 (pyrealsense2 pip package reports version 2.30.0.1216)
Language	python 3.5.2
Segment	robot

Hi,

I'm updating an application to support using Realsense cameras. I'd like to register a callback for handling rgb+depth framesets because this callback will help me integrate the Realsense camera with my already-existing application much more easily.

I'm new to the realsense API and what I'm trying to do doesn't seem to be documented officially, but I've looked through a bunch of the python wrapper's documentation as well as the example use of callbacks with the C++ API https://github.com/IntelRealSense/librealsense/tree/master/examples/callback. The example I just linked makes it seem like the callback you can register takes a whole frameset object (which contains rgb data as well as depth data), but when I try to use a callback in python to handle that frameset object, it seems like that frameset object actually only contains data associated with one stream. Furthermore trying to cast that frameset into a depth frame or a video frame does not work (which I would expect it to) -- it breaks somewhere inside the C++ API.

Here's an example of what I'm experiencing:

$ ipython
Python 3.5.2 (default, Oct  8 2019, 13:06:37) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.9.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import pyrealsense2 as rs                                                                                      

In [2]: pipeline = rs.pipeline()                                                                                       

In [3]: def test_callback(fs): 
   ...:     global frameset 
   ...:     frameset = fs 
   ...:     pipeline.stop() 
   ...:                                                                                                                

In [4]: profile = pipeline.start(test_callback)

# Two streams associated with this pipeline
In [5]: profile.get_streams()                                                                                         
Out[5]: 
[<pyrealsense2.video_stream_profile: 1(0) 1280x720 @ 30fps 1>,
 <pyrealsense2.video_stream_profile: 2(0) 640x480 @ 30fps 5>]                                                                      

# Frameset captured by callback successfully and seems like it is probably a valid frameset
In [6]: frameset                                                                                                       
Out[6]: <pyrealsense2.pyrealsense2.frame at 0x7f6d3d212ca8>

In [7]: frameset.is_frameset()                                                                                        
Out[7]: True

In [8]: frameset.get_data()                                                                                           
Out[8]: <pyrealsense2.pyrealsense2.BufData at 0x7f6d05b81b20>

# Frameset only associated with a single profile (I didn't find a get_profiles() method or similar)
In [9]: frameset.get_profile()                                                                                         
Out[9]: <pyrealsense2.video_stream_profile: 1(0) 1280x720 @ 30fps 1>

# Tries to get a depth or video frame out of the frameset fail :(
In [10]: frameset.as_depth_frame()                                                                                      
Out[10]: <pyrealsense2.pyrealsense2.depth_frame at 0x7f6d3d24be68>

In [11]: frameset.as_depth_frame().get_data()                                                                           
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-8-564895166424> in <module>
----> 1 frameset.as_depth_frame().get_data()

RuntimeError: null pointer passed for argument "frame_ref"

In [12]: frameset.as_video_frame()                                                                                     
Out[12]: <pyrealsense2.pyrealsense2.video_frame at 0x7f6d05bcfb90>

In [12]: frameset.as_video_frame().get_data()                                                                          
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-31-1f108a30e99a> in <module>
----> 1 frameset.as_video_frame().get_data()

RuntimeError: null pointer passed for argument "frame_ref"

Does this seem like a buggy python wrapper or poor use on my part?

It would be great if I could use some way of handling incoming frames asynchronously; my application does a lot more stuff than just talk to a camera. So if the python wrapper is in fact buggy, and you have any other suggestions I'd love to hear them.

[sinback]

I resolved this. I needed to call as_frameset() on the frame the callback was processing and then iterate over the frames therein. (eg to deal with a depth frame in the callback you can do something like depth_data = fs.as_frameset().get_depth_frame().get_data(). That was a little non-intuitive but I think I am set now.

[Dontla]

In [6]: frameset

NameError: name 'frameset' is not defined

How to use it correctly?

[07hokage]

In [6]: frameset

NameError: name 'frameset' is not defined

How to use it correctly?

before the callback function, initialize the variable frameset=None

[Dontla]

In [6]: frameset
NameError: name 'frameset' is not defined
How to use it correctly?

before the callback function, initialize the variable frameset=None

I can't understand...

[07hokage]

frame = None
def callbackFunction(frame):
global frame
''' write your script ""

Now here use the frame variable