Mediapipe VS Nvidia's Deep Stream

[ShettyHarapanahalli]

Hi,

Just curious to understand the differences between the Mediapipe and Deepstream (https://developer.nvidia.com/deepstream-sdk), are these both addressing the same problem space. Maybe solutions spaces could be different, but are problem spaces same?

Best Regards,
Suman

[mcclanahoochie]

Hi,
Good question; at first glance, Medaipipe (MP) and DeepStream (DS) may seem similar. At the core of it, they both provide a framework that can run an inference task.
There are a few important differences:

• Data types: DS supports only video/image streams, while MP operates on generic packets (which can include video, audio, image, primitives, or any user-defined object types)
• Graphs: Data flow in both frameworks are described by a graph. DS is built on top of Gstreamer, which has a strictly linear pipeline model with very basic control flow. MP graphs offer more flexibility in design, and allows for very complex graph construction, including dynamic control flow and back edges.
• Hardware/OS: DS is restricted to high-end desktop Nvidia GPUs (and appears to be Linux only). MP supports running on desktop/mobile CPUs and mobile GPUs (via OpenGL), and desktop GPU support is planned. MP runs on Android/iOS/Linux/OSX.
• Tools: MP provides a powerful graph visualizer for interactive building and analyzing graphs, including graph performance tracing. I think DS relies on existing Gstreamer visualization tools like graphviz or dot (which is not interactive).
• Plugins/Calculators: Each node in a DS pipeline is a Gstreamer filter (limited to how you write Gstreamer plugins). Each node in a MP graph is a Calculator, representing a very generic operation[s] to any or multiple packet stream types, also not limited to inference or even machine learning at all.

Hope that can help distinguish the two frameworks.

[barzan-hayati]

Hi, Good question; at first glance, Medaipipe (MP) and DeepStream (DS) may seem similar. At the core of it, they both provide a framework that can run an inference task. There are a few important differences:

• Data types: DS supports only video/image streams, while MP operates on generic packets (which can include video, audio, image, primitives, or any user-defined object types)
• Graphs: Data flow in both frameworks are described by a graph. DS is built on top of Gstreamer, which has a strictly linear pipeline model with very basic control flow. MP graphs offer more flexibility in design, and allows for very complex graph construction, including dynamic control flow and back edges.
• Hardware/OS: DS is restricted to high-end desktop Nvidia GPUs (and appears to be Linux only). MP supports running on desktop/mobile CPUs and mobile GPUs (via OpenGL), and desktop GPU support is planned. MP runs on Android/iOS/Linux/OSX.
• Tools: MP provides a powerful graph visualizer for interactive building and analyzing graphs, including graph performance tracing. I think DS relies on existing Gstreamer visualization tools like graphviz or dot (which is not interactive).
• Plugins/Calculators: Each node in a DS pipeline is a Gstreamer filter (limited to how you write Gstreamer plugins). Each node in a MP graph is a Calculator, representing a very generic operation[s] to any or multiple packet stream types, also not limited to inference or even machine learning at all.

Hope that can help distinguish the two frameworks.

This argument was made in 2019. Has anything changed since then?

While I'm not an NVIDIA enthusiast, it's important to highlight some information about DeepStream. It's true that DeepStream operates on NVIDIA GPUs, leveraging hardware to accelerate processing, particularly in decoding plugins. Additionally, it simplifies the creation of output RTSP streams from processed video, a feature that MediaPipe does not support.

Due to its use of TensorRT engines, models can run more quickly. NVIDIA also offers code to facilitate the easy conversion of various models into TensorRT engines. You can visualize graphs using Graphviz, but for analysis, GstShark is utilized.