Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

lavc/videotoolboxenc: add MJPEG support #372

Merged
merged 1 commit into from
Apr 21, 2024

Conversation

gnattu
Copy link
Member

@gnattu gnattu commented Mar 28, 2024

Changes

For example:

./ffmpeg -hwaccel videotoolbox \
  -i INPUT -c:v mjpeg_videotoolbox \
   out.mp4

This encoder does not have many options and can only use -q:v to control quality.

For our use case, this should be used with -allow_sw 1 because not all Intel Macs come with a hardware-backed MJPEG engine.

A bigger problem would be if the use of hardware acceleration for MJPEG is actually worth it. I've tested both on M1 Max and on i9-12900 (with its UHD770), and both platforms showed that a pure software chain outperforms a pure hardware chain in downscaling a UHD video into a 320x180 thumbnail for this use case. I believe the main purpose of hardware chain would be to reduce power consumption rather than enhance performance, and we may want to document that to tell our users.

If the other encoder implementation is trivial, like VT, then we can implement it just in case. However, if it requires too much effort, then it's probably not worth it, as modern CPUs handle MJPEG at this size quite well.

Issues

@nyanmisaka
Copy link
Member

A bigger problem would be if the use of hardware acceleration for MJPEG is actually worth it. I've tested both on M1 Max and on i9-12900 (with its UHD770), and both platforms showed that a pure software chain outperforms a pure hardware chain in downscaling a UHD video into a 320x180 thumbnail for this use case. I believe the main purpose of hardware chain would be to reduce power consumption rather than enhance performance, and we may want to document that to tell our users.

If the other encoder implementation is trivial, like VT, then we can implement it just in case. However, if it requires too much effort, then it's probably not worth it, as modern CPUs handle MJPEG at this size quite well.

Software encoding MJPEG may cause the CPU to continuously turbo on a single core thus increasing power consumption and fan noise. Especially when trickplay is set to non-blocking and low priority, this task may continue in the background for a long time. Therefore, for fixed hardware MJPEG impl (Intel, Apple, Rockchip), performance is not primary, but power consumption is.

The only thing that can surpass software MJPEG is nvJPEG over CUDA cores. It has not yet been implemented in FFmpeg since it's not NVENC, but the example code shows that it is easy to use, and each MJPEG frame is a key frame, there is no need to worry about managing reference frames. So one CUDA frame input, one MJPEG packet output. Also, since it is a GPGPU impl, it does not save energy too much, but it can significantly increase the speed. I may look into it in the future.

BTW I already have a patch for mjpeg_rkmpp on the way. Since the RKMPP patchset is huge and comes before mjpeg_videotoolbox in quilt order, I would merge it first to avoid me having to adjust the configure & Makefile & allcodecs in your patch.

@gnattu
Copy link
Member Author

gnattu commented Mar 28, 2024

One of my concerns with this is that if it takes a long time in the background, it may affect the foreground playback experience, as it adds load to the decoder. Additionally, we have some Nvidia GPUs that have a concurrent session limit, and this could cause issues with the background encoding as well.

@nyanmisaka
Copy link
Member

One of my concerns with this is that if it takes a long time in the background, it may affect the foreground playback experience, as it adds load to the decoder. Additionally, we have some Nvidia GPUs that have a concurrent session limit, and this could cause issues with the background encoding as well.

Maybe we can add a restriction to stop and prevent trickplay when transcoding is happening in the foreground.

nvJPEG is a CUDA impl. Therefore it is not affected by NVENC concurrent sessions limit. Therefore trickplay will only use NVDEC/CUVID+CUDA on Nvidia.

@nyanmisaka
Copy link
Member

Ubuntu noble repo is broken again...

For example:
./ffmpeg -hwaccel videotoolbox \
  -i INPUT -c:v mjpeg_videotoolbox \
   out.mp4

This encoder does not have many options and can only use `-q:v` to control quality.

Signed-off-by: Gnattu OC <gnattuoc@me.com>
@gnattu gnattu force-pushed the jellyfin-add-mjpeg-videotoolbox branch from df2ae4d to 22924ab Compare April 21, 2024 15:19
@gnattu gnattu merged commit 3c9bf86 into jellyfin:jellyfin Apr 21, 2024
28 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants