Realsense hardware video/depth encoding and streaming over custom MLSP protocol.
This includes streaming:
- color (H.264, HEVC Main)
- infrared/infrared-rgb (H.264, HEVC Main)
- depth (HEVC Main10)
- textured depth (HEVC Main10 + HEVC Main)
See unity-network-hardware-video-decoder as example network decoder & renderer (color, infrared and depth).
See how it works on wiki to understand the code.
See hardware-video-streaming for other related projects.
See videos to understand point cloud streaming features:
| Point Cloud Streaming | Infrared Point Cloud Streaming | Point Cloud Streaming To UMPC |
|---|---|---|
 |
 |
 |
Unix-like operating systems (e.g. Linux), more info.
Tested on Ubuntu 18.04.
Tested on Intel KabyLake with Realsense D435, D455, L515.
Requires Intel VAAPI compatible hardware encoder (QuickSync Video). For depth encoding at least KabyLake.
Other technologies may also work but were not tested.
Infrared textured depth encoding is implemented for D415, D435, D455 and L515. Color textured depth encoding is implemented for D415, D435, D455, L515.
This program depends on:
- librealsense2
- NHVE Network Hardware Video Encoder
- HVE Hardware Video Encoder
- FFmpeg
avcodec,avutil,avfilter(at least 3.4 version)
- FFmpeg
- MLSP Minimal Latency Streaming Protocol
- HVE Hardware Video Encoder
Install RealSense™ SDK 2.0 as described on github
NHVE and its dependencies are included as submodules so you only need to satifisy HVE dependencies.
Works with system FFmpeg on Ubuntu 18.04 and doesn't on 16.04 (outdated FFmpeg and VAAPI ecosystem).
Tested on Ubuntu 18.04.
# update package repositories
sudo apt-get update
# get avcodec and avutil
sudo apt-get install ffmpeg libavcodec-dev libavutil-dev libavfilter-dev
# get compilers and make
sudo apt-get install build-essential
# get cmake - we need to specify libcurl4 for Ubuntu 18.04 dependencies problem
sudo apt-get install libcurl4 cmake
# get git
sudo apt-get install git
# clone the repository with *RECURSIVE* for submodules
git clone --recursive https://github.com/bmegli/realsense-network-hardware-video-encoder.git
# finally build the program
cd realsense-network-hardware-video-encoder
mkdir build
cd build
cmake ..
makeStream H.264 Realsense color/infrared/infrared-rgb video over UDP.
Usage: ./realsense-nhve-h264
<host> <port>
<color/ir/ir-rgb>
<width> <height> <framerate> <seconds>
[device] [bitrate]
examples:
./realsense-nhve-h264 127.0.0.1 9766 color 640 360 30 5
./realsense-nhve-h264 127.0.0.1 9766 ir 640 360 30 5
./realsense-nhve-h264 127.0.0.1 9766 ir-rgb 640 360 30 5
./realsense-nhve-h264 127.0.0.1 9766 color 640 360 30 5 /dev/dri/renderD128
./realsense-nhve-h264 127.0.0.1 9766 ir 640 360 30 5 /dev/dri/renderD128
./realsense-nhve-h264 127.0.0.1 9766 ir-rgb 640 360 30 5 /dev/dri/renderD128
./realsense-nhve-h264 192.168.0.125 9766 color 640 360 30 50 /dev/dri/renderD128 500000```Stream Realsense:
- color/infrared/infrared-rgb with HEVC Main
- depth with HEVC Main10
Usage: ./realsense-nhve-hevc
<host> <port>
<color/ir/ir-rgb/depth>
<width> <height> <framerate> <seconds>
[device] [bitrate] [depth units] [json]
examples:
./realsense-nhve-hevc 127.0.0.1 9766 color 640 360 30 5
./realsense-nhve-hevc 127.0.0.1 9766 ir 640 360 30 5
./realsense-nhve-hevc 127.0.0.1 9766 ir-rgb 640 360 30 5
./realsense-nhve-hevc 127.0.0.1 9766 depth 640 360 30 5
./realsense-nhve-hevc 127.0.0.1 9766 color 640 360 30 5 /dev/dri/renderD128
./realsense-nhve-hevc 127.0.0.1 9766 ir 640 360 30 5 /dev/dri/renderD128
./realsense-nhve-hevc 127.0.0.1 9766 ir-rgb 640 360 30 5 /dev/dri/renderD128
./realsense-nhve-hevc 127.0.0.1 9766 depth 640 360 30 5 /dev/dri/renderD128
./realsense-nhve-hevc 192.168.0.125 9766 color 640 360 30 50 /dev/dri/renderD128 500000
./realsense-nhve-hevc 127.0.0.1 9768 depth 848 480 30 50 /dev/dri/renderD128 2000000
./realsense-nhve-hevc 192.168.0.100 9768 depth 848 480 30 500 /dev/dri/renderD128 2000000 0.0001
./realsense-nhve-hevc 192.168.0.100 9768 depth 848 480 30 500 /dev/dri/renderD128 2000000 0.00005
./realsense-nhve-hevc 192.168.0.100 9768 depth 848 480 30 500 /dev/dri/renderD128 2000000 0.000025
./realsense-nhve-hevc 192.168.0.100 9768 depth 848 480 30 500 /dev/dri/renderD128 2000000 0.0000125
./realsense-nhve-hevc 192.168.0.100 9768 depth 640 480 30 500 /dev/dri/renderD128 8000000 0.0000390625 my_config.jsonStream Realsense D415/D435/D455/L515:
- depth with HEVC Main10, infrared or infrared rgb with HEVC
Usage: ./realsense-nhve-depth-ir
<host> <port>
<ir/ir-rgb>
<width> <height> <framerate> <seconds>
[device] [bitrate_depth] [bitrate_ir] [depth units] [json]
examples:
./realsense-nhve-depth-ir 127.0.0.1 9766 ir 640 360 30 5
./realsense-nhve-depth-ir 127.0.0.1 9766 ir-rgb 640 360 30 5
./realsense-nhve-depth-ir 127.0.0.1 9766 ir 640 360 30 5 /dev/dri/renderD128
./realsense-nhve-depth-ir 192.168.0.125 9766 ir-rgb 640 360 30 50 /dev/dri/renderD128 4000000 1000000
./realsense-nhve-depth-ir 192.168.0.100 9768 ir 848 480 30 500 /dev/dri/renderD128 8000000 1000000 0.0001
./realsense-nhve-depth-ir 192.168.0.100 9768 ir-rgb 848 480 30 500 /dev/dri/renderD128 8000000 1000000 0.00005
./realsense-nhve-depth-ir 192.168.0.100 9768 ir 848 480 30 500 /dev/dri/renderD128 8000000 1000000 0.000025
./realsense-nhve-depth-ir 192.168.0.100 9768 ir-rgb 848 480 30 500 /dev/dri/renderD128 8000000 1000000 0.0000125
./realsense-nhve-depth-ir 192.168.0.100 9768 ir 848 480 30 500 /dev/dri/renderD128 8000000 1000000 0.0000125
./realsense-nhve-depth-ir 192.168.0.100 9768 ir-rgb 848 480 30 500 /dev/dri/renderD128 8000000 1000000 0.00003125
./realsense-nhve-depth-ir 192.168.0.100 9768 ir 640 480 30 500 /dev/dri/renderD128 8000000 1000000 0.0000390625 my_config.json```Stream Realsense D415/D435/D455/L515:
- depth with HEVC Main10, color with HEVC (aligned)
Usage: ./realsense-nhve-depth-color
<host> <port>
<color/depth> # alignment direction
<width_depth> <height_depth> <width_color> <height_color>
<framerate> <seconds>
[device] [bitrate_depth] [bitrate_color] [depth units] [json]
examples:
./realsense-nhve-depth-color 127.0.0.1 9766 color 640 360 640 360 30 5
./realsense-nhve-depth-color 127.0.0.1 9766 color 640 360 640 360 30 5 /dev/dri/renderD128
./realsense-nhve-depth-color 192.168.0.125 9766 color 640 360 640 360 30 50 /dev/dri/renderD128 4000000 1000000
./realsense-nhve-depth-color 192.168.0.100 9768 color 848 480 848 480 30 500 /dev/dri/renderD128 8000000 1000000 0.0001
./realsense-nhve-depth-color 192.168.0.100 9768 color 848 480 848 480 30 500 /dev/dri/renderD128 8000000 1000000 0.00005
./realsense-nhve-depth-color 192.168.0.100 9768 color 848 480 848 480 30 500 /dev/dri/renderD128 8000000 1000000 0.000025
./realsense-nhve-depth-color 192.168.0.100 9768 color 848 480 848 480 30 500 /dev/dri/renderD128 8000000 1000000 0.0000125
./realsense-nhve-depth-color 192.168.0.100 9768 depth 848 480 848 480 30 500 /dev/dri/renderD128 8000000 1000000 0.0000125
./realsense-nhve-depth-color 192.168.0.100 9768 color 848 480 848 480 30 500 /dev/dri/renderD128 8000000 1000000 0.00003125f
./realsense-nhve-depth-color 192.168.0.100 9768 depth 848 480 1280 720 30 500 /dev/dri/renderD128 8000000 1000000 0.00003125f
./realsense-nhve-depth-color 192.168.0.100 9768 depth 640 480 1280 720 30 500 /dev/dri/renderD128 8000000 1000000 0.0000390625 my_config.json
./realsense-nhve-depth-color 192.168.0.100 9768 color 640 480 1280 720 30 500 /dev/dri/renderD128 8000000 1000000 0.0000390625 my_config.jsonIf you don't have receiving end you will just see if hardware encoding worked/didn't work.
You may need to specify VAAPI device if you have more than one (e.g. NVIDIA GPU + Intel CPU).
If you get errors see also HVE troubleshooting.
Code in this repository and my dependencies are licensed under Mozilla Public License, v. 2.0
This is similiar to LGPL but more permissive:
- you can use it as LGPL in prioprietrary software
- unlike LGPL you may compile it statically with your code
Like in LGPL, if you modify the code, you have to make your changes available. Making a github fork with your changes satisfies those requirements perfectly.
Since you are linking to FFmpeg libraries consider also avcodec, avutil and avfilter licensing.
The implementation down to FFmpeg (encoding) and up to MLSP (sending) is zero copy (passing the pointers to data) whenever possible.