lavc/videotoolboxenc: add MJPEG support

[gnattu]

Changes

For example:

./ffmpeg -hwaccel videotoolbox \
  -i INPUT -c:v mjpeg_videotoolbox \
   out.mp4

This encoder does not have many options and can only use -q:v to control quality.

For our use case, this should be used with -allow_sw 1 because not all Intel Macs come with a hardware-backed MJPEG engine.

A bigger problem would be if the use of hardware acceleration for MJPEG is actually worth it. I've tested both on M1 Max and on i9-12900 (with its UHD770), and both platforms showed that a pure software chain outperforms a pure hardware chain in downscaling a UHD video into a 320x180 thumbnail for this use case. I believe the main purpose of hardware chain would be to reduce power consumption rather than enhance performance, and we may want to document that to tell our users.

If the other encoder implementation is trivial, like VT, then we can implement it just in case. However, if it requires too much effort, then it's probably not worth it, as modern CPUs handle MJPEG at this size quite well.

Issues

[nyanmisaka]

A bigger problem would be if the use of hardware acceleration for MJPEG is actually worth it. I've tested both on M1 Max and on i9-12900 (with its UHD770), and both platforms showed that a pure software chain outperforms a pure hardware chain in downscaling a UHD video into a 320x180 thumbnail for this use case. I believe the main purpose of hardware chain would be to reduce power consumption rather than enhance performance, and we may want to document that to tell our users.

If the other encoder implementation is trivial, like VT, then we can implement it just in case. However, if it requires too much effort, then it's probably not worth it, as modern CPUs handle MJPEG at this size quite well.

Software encoding MJPEG may cause the CPU to continuously turbo on a single core thus increasing power consumption and fan noise. Especially when trickplay is set to non-blocking and low priority, this task may continue in the background for a long time. Therefore, for fixed hardware MJPEG impl (Intel, Apple, Rockchip), performance is not primary, but power consumption is.

The only thing that can surpass software MJPEG is nvJPEG over CUDA cores. It has not yet been implemented in FFmpeg since it's not NVENC, but the example code shows that it is easy to use, and each MJPEG frame is a key frame, there is no need to worry about managing reference frames. So one CUDA frame input, one MJPEG packet output. Also, since it is a GPGPU impl, it does not save energy too much, but it can significantly increase the speed. I may look into it in the future.

BTW I already have a patch for mjpeg_rkmpp on the way. Since the RKMPP patchset is huge and comes before mjpeg_videotoolbox in quilt order, I would merge it first to avoid me having to adjust the configure & Makefile & allcodecs in your patch.

[gnattu]

One of my concerns with this is that if it takes a long time in the background, it may affect the foreground playback experience, as it adds load to the decoder. Additionally, we have some Nvidia GPUs that have a concurrent session limit, and this could cause issues with the background encoding as well.

[nyanmisaka]

One of my concerns with this is that if it takes a long time in the background, it may affect the foreground playback experience, as it adds load to the decoder. Additionally, we have some Nvidia GPUs that have a concurrent session limit, and this could cause issues with the background encoding as well.

Maybe we can add a restriction to stop and prevent trickplay when transcoding is happening in the foreground.

nvJPEG is a CUDA impl. Therefore it is not affected by NVENC concurrent sessions limit. Therefore trickplay will only use NVDEC/CUVID+CUDA on Nvidia.

[nyanmisaka]

Ubuntu noble repo is broken again...