-
-
Notifications
You must be signed in to change notification settings - Fork 49
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[META] Codec improvement and additional support including AV1/H.265, Intel/AMD GPU encoding support #34
Comments
Note: https://blogs.igalia.com/vjaquez/2018/03/28/gstreamer-va-api-troubleshooting/ Relevant code: https://github.com/selkies-project/selkies-gstreamer/blob/master/src/selkies_gstreamer/gstwebrtc_app.py Supporting GPU statistics is also needed. GPUtils only supports NVIDIA. |
AV1 encoders will come soon to NVIDIA and Intel. AV1 should be possible to be used as a decoding codec for major web browsers. This should be available to GStreamer soon and confirmed it is in the works. |
More discussions are going on related to the VAAPI plugin in #42. |
Does |
@maxpain Depends on the web browser. I'm trying to make updates. |
@ehfd Chrome supports WebRTC AV1 encoder starting from v90 beta |
We need this. https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/1489 AV1 support tends to be meaningless without support from NVIDIA. |
The https://gstreamer.freedesktop.org/documentation/msdk/msdkh264enc.html https://gstreamer.freedesktop.org/documentation/amfcodec/amfh264enc.html |
|
AMD users on Linux are stuck with Check https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/5811 Edit: AMF has been integrated into the Vulkan API with RADV. https://www.phoronix.com/news/RADV-Vulkan-VIdeo-H265-H264 https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/5739 |
Look for
I will implement these after 6-12 months if there are no contributions. Anything other than these will be done after GStreamer 1.24 is out. |
https://webrtc.googlesource.com/src/+/972f28335a7f989ee1ffccb7c64fda1f29e09747 H.265 coming to Chromium? And an apparent solution to the packetization issue in Apple. |
Note to self regarding x264enc - option added to the pipeline: https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/3034 |
H.264 is implemented and tested via vah264enc with Intel and AMD GPUs. |
|
GStreamer 1.24.0 and Rust plugins are now enabled. |
H.265 is currently not in Chromium. I cannot test Safari without the device. Waiting for VA AV1/VP9 encoders and NVIDIA AV1 encoders. SVT-AV1 has an issue with changing properties such as bitrate on the fly. |
va plugin
|
nvcodec 1.24
nvcodec 1.25 (1.26)
|
SW encoder 1.24
SW colorspace and X11 source
PipeWire/Wayland Source
|
Audio element
|
RTP payloader
|
WebRTC/RTP element
|
However, a lot of these encoders have not been optimized.
Contributions for optimized encoder or payloader settings are welcome (consult https://github.com/LizardByte/Sunshine/blob/nightly/src/video.cpp).
https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/6754 |
https://developer.apple.com/documentation/safari-technology-preview-release-notes/stp-release-179#Media |
It's already merged. How can I use AV1 with NVENC today? |
@maxpain Even if you're compiling from HEAD (compile https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/issues/558 Wait for this to be resolved. |
Outstanding issues with AV1: Outstanding issues with VA: Outstanding issues with H.265: |
The previous AV1 issue https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/issues/558 has been fixed a few hours ago. |
Looking for contributors to help out.
Guide for improving video encoder parameters (also applicable to non-NVIDIA encoders): https://docs.nvidia.com/video-technologies/video-codec-sdk/12.2/nvenc-video-encoder-api-prog-guide/index.html#recommended-nvenc-settings__table-nvenc-settings
Remember that live (as in < 100 ms) streaming is all about keeping the bitrate consistent at all frames to minimize congestion from unexpected bandwidth spikes (which lead to stutters and latency spikes) and keeping the buffer small. Use the configured Constant Bitrate bandwidth as possible at all times but do not go over it (ideal maximum overshoot tolerance = 10%).
Priority: Latency > Quality (minimum sacrifices of <= 5% in latency for quality or bandwidth consistency is acceptable, using more GPU resources or CPU cores also acceptable if without latency impact) >> Overall Bandwidth Efficiency (since encoded frames are only used once unlike .mp4/.mkv)
This is also applicable to other encoders if available.
***: These features are useful for error recovery during transmission across noisy mediums. For usecases where the client requires reduced video memory footprint (this typically means reducing any buffer that exists, preventing frame lag, except the ones unrelated to latency, stated below), following guidelines should be followed.
Avoid using B-frames. B-frames requires additional buffers for reordering, hence avoiding B-frames would result to savings in video memory usage. (Incompatible with WebRTC in current (2023) web browsers)
Reducing the maximum number of reference frames doesn't matter. Reducing number of maximum reference frames results in NVIDIA display driver allocating lesser number of buffers internally thereby reducing video memory footprint. (May impact latency, but long reference should be safe)
Single pass rate control modes doesn't matter. Two pass rate control consume additional video memory in comparison to single pass due to additional allocations for first pass encoding. Two pass rate control mode with first pass with full resolution consumes more than first pass with quarter resolution. (Not influential to latency but adds GPU load, two-pass is recommended nonetheless)
Adaptive Quantization / Weighted Prediction has unknown effects. Features such as Adaptive Quantization / Weighted Prediction allocate additional buffers in video memory. These allocations can be avoided if these features are not used. (Not influential to latency but adds GPU load, so worth checking both features)
Avoid Lookahead. Lookahead allocates additional buffers in video memory for frames that are buffered in the lookahead queue. (Important because it holds generated frames)
Avoid Temporal Filter. Temporal filter requires neighbouring frames and allocates additional buffers in the video memory. (Important because it holds generated frames)
Avoid UHQ Tuning Info. UHQ Tuning Info enables lookahead and temporal Filter, that have higher memory requirements. (Important because it holds generated frames)
It should be possible to obtain the sweet spot of encoder maximum and minimum QP parameters
https://multi.app/blog/making-illegible-slow-webrtc-screenshare-legible-and-fast
https://multi.app/blog/measuring-shared-control-latency
The text was updated successfully, but these errors were encountered: