HLS on arm64 #697

chudsaviet · 2020-11-23T04:07:27Z

HLS does not work on arm64 build.
This is the bug to track these issues.

chudsaviet · 2020-11-23T04:12:45Z

On 0.18.0 RC1 arm64 version, it was found that HLS stream is not working.

I have investigated these issues.

So far we were relying on RaspberryPi hardware h264 encoding capabilities.
Unfortunately, now there is no hardware encoding library /opt/vc/lib/libopenmaxil.so for arm64.
We can track next repositories on arm64 hardware encoding implementation:

Basically, libopenmaxil.so is the part of userland repo. It's built and the binaries are put to the firmware repo. So far, it does not build for arm64.

Also, we will have to use FFmpeg built with --enable-omx-rpi flag, which is on by default for armhf and off for arm64.

chudsaviet · 2020-11-23T04:21:27Z

As a solution, we can use software h264 encoding. Actually, Raspberry Pi 4 is powerful enough to do this without interfering with OctoPrint wannabe-realtime core.

Good news is that even with veryfast preset, libx264 is producing much lower bitrate for given visual quality.

I think I will work on webcamd analog script written in Python3, which will be flexible enough to choose between software/hardware encoder.

chudsaviet · 2020-11-23T04:23:04Z

For now, users can manually set encoder to h264 instead of h264_omx, and set --preset:v veryfast in ffmpeg-hls Systemd unit.

DrDrache · 2020-11-23T14:12:38Z

not to add unuseful information,
but from my understanding OMX (openmax) will "never" come to 64bit userland. it would be a nightmare to map the VPU.
the current (supported?) h264 hardware-assisted method is h264_v4l2m2m via ffmpeg >=4.3

chudsaviet · 2020-11-23T17:36:21Z

Thanks, @DrDrache , it’s actually useful.
Yeah, I was trying to use V4L2M2M, but it had not supported all options I needed.
Let me try to investigate it more.

chudsaviet · 2020-11-26T02:21:02Z

Good news - h264_v4l2m2m works on Rpi4 on arm64, and pretty fast. :-D
I will work on a patch, and maybe we will switch armhf version to it too.

chudsaviet · 2021-01-09T19:15:53Z

Bad news: ‘h264_v4l2m2m’ encoder in ffmpeg is pretty buggy. It does not work correctly when I’m trying to split streams from webcam to two h264 and one jpeg stream.
I will investigate if we can use software encoder without overloading CPU.

fastcompjason · 2021-06-09T16:55:24Z

Any updates on this? I am trying to use an older Logitech Pro 9000 webcam with OctoPi running on a new RPi 4B+ 8GB and when I specifically enable HLS, I get nothing at all. I have even tried to specifically change the config file to specify a resolution and refresh rate. I would very much like to run a H.264 stream out of OctoPi to import into OBS or into other software hardware platforms for monitoring if possible.

guysoft · 2021-06-09T20:25:10Z

None, also found this and there seems to be no news there neither: jellyfin/jellyfin#4023 (comment)

chudsaviet · 2021-06-10T02:08:32Z

Yeah, we can only run the software encoder on arm64 for now.

cryptoAlgorithm · 2022-07-05T04:18:51Z

Bad news: ‘h264_v4l2m2m’ encoder in ffmpeg is pretty buggy. It does not work correctly when I’m trying to split streams from webcam to two h264 and one jpeg stream.

@chudsaviet yeah I observed that too, using h264_v4l2m2m in place of h264 throws a few errors, firstly the -preset option isn't supported, next the stream simply doesn't work with the preset removed. It works perfectly with h264 though, using around 110% CPU on my pi4b overclocked to 1.8GHz.

cryptoAlgorithm · 2022-07-05T04:19:12Z

Is there a way to get h264_v4l2m2m to work with just one stream? That would be great!

chudsaviet · 2022-07-05T05:56:04Z

Yep, just remove JPEG stream parameters from ffmpeg config,

cryptoAlgorithm · 2022-07-05T06:06:17Z

@chudsaviet So I could just get rid of the jpeg stream part, and it would work? I can still have 2 streams?

cryptoAlgorithm · 2022-07-05T07:21:35Z

@chudsaviet No luck, substituting h264 for h264_v4l2m2m produces an unplayable stream (notice the "Plug-in handled load" error, ignore all the AdGuard messages)

Here's the full ffmpeg command that I'm running:

/usr/bin/ffmpeg \
    \
    -framerate 30 -video_size 1280x960 \
    -i /dev/video0 \
    -pix_fmt yuv420p \
    \
    -c:v h264_v4l2m2m \
    -b:v 800k -flags +cgop \
    -g 30 -keyint_min 30 \
    \
    -f hls -hls_time 1 \
    -hls_flags delete_segments+program_date_time+temp_file+independent_segments \
    -hls_allow_cache 0 -hls_segment_type fmp4 \
    -hls_list_size 2 -hls_delete_threshold 3 \
    /run/webcam/hls/720p/stream.m3u8

chudsaviet · 2022-07-05T07:45:54Z

v4l2m2m is buggy :(
A good way to reduce CPU usage of software h264 encoder is to reduce resolution, fps and use -preset veryfast parameter.

cp2004 · 2022-07-05T11:07:38Z

HLS is also now broken on the 32 bit RPi OS images since the OMX encoders were removed - I tried to get it working myself but ran into very similar issues when actually debugging it as well (#770 (comment)).

chudsaviet · 2022-07-05T15:26:50Z

That’s bad news, and definitely a regression. I will go look into if I can debug v4l2m2m.

chudsaviet · 2022-07-06T00:41:46Z

It looks like h264_v4l2m2m works on the latest RaspberryPi OS version with kernel 5.15.32-v8+, but only for fixed-duration video files, not HLS.
There is a problem with 'dts` timestamp, which is required by HLS.
I will look into fixing it - https://github.com/raspberrypi/linux/blob/6b945e6f05d5cb0a299dbbb9bdd285aff330a204/drivers/staging/vc04_services/bcm2835-codec/bcm2835-v4l2-codec.c#L1289

chudsaviet · 2022-07-06T00:47:07Z

Example for my Microsoft LifeCam Studio:

sudo fuser -k /dev/video0 ; \
sudo rm /var/www/html/test.mp4 ; \
sudo v4l2-ctl -d /dev/video0 -c exposure_auto=1 && \
sudo v4l2-ctl -d /dev/video0 -c exposure_absolute=20 && \
ffmpeg \
    -loglevel info \
    \
    -t 00:00:10 \
    -input_format mjpeg \
    -framerate 60 -video_size 1280x720 \
    -i /dev/video0 \
    \
    -filter:v "format=yuv420p" \
    \
    -c:v h264_v4l2m2m \
    -b:v 2048k \
    \
    /var/www/html/test.mp4

cryptoAlgorithm · 2022-07-06T02:55:10Z

Yeah I experienced the same issues when using h264_v4l2m2m, which made me suspect some meta is missing in the m3u8 stream produced by it

chudsaviet · 2022-07-10T00:10:38Z

Ok, good news everyone!
I made it work.
I used:

FFmpeg built from master, commit 59018254c7a178e4650ef63016493c75383a2076
Latest stable RaspberryPi OS on 2022-07-09, kernel 5.15.32-v8+
Rpi 4 8GiB RAM

These FFmpeg parameters (MS LifeCam Studio produces 720p@30fps only in MJPEG):

sudo fuser -k /dev/video0 ; \
sudo rm /var/www/html/stream.m3u8 ; \
sudo v4l2-ctl -d /dev/video0 -c exposure_auto=1 && \
sudo v4l2-ctl -d /dev/video0 -c exposure_absolute=20 && \
./ffmpeg \
     -loglevel info \
     -input_format mjpeg \
     -framerate 30 -video_size 1280x720 \
     -i /dev/video0 \
     \
     -filter:v "format=yuv420p" \
     -c:v h264_v4l2m2m \
     -b:v 3792k \
     -level:v 4.0 \
     \
     -f hls -hls_time 1 \
     -hls_flags delete_segments+program_date_time+temp_file+independent_segments \
     -hls_allow_cache 0 -hls_segment_type fmp4 \
     -hls_list_size 2 -hls_delete_threshold 3 \
     /var/www/html/stream.m3u8

CPU load was around 10% on all cores.

I will now work on putting it into OctoPi.

chudsaviet · 2022-07-10T01:20:39Z

Looks like I will need latest FFmpeg from ‘master’. It can be done, but it will increase image build time.
@guysoft , would you merge such a change?

cryptoAlgorithm · 2022-07-10T02:33:51Z

@chudsaviet is the raspberry pi os 64-bit? octopi seems to be using Ubuntu instead for the 64 bit images though, don't know if this solution will work for it too.

cryptoAlgorithm · 2022-07-10T02:41:29Z

@chudsaviet is the raspberry pi os 64-bit? octopi seems to be using Ubuntu instead for the 64 bit images though, don't know if this solution will work for it too.

chudsaviet · 2022-07-10T03:01:04Z

Ubuntu was chosen for 64bit only because there were no stable 64bit RaspberryPi OS builds. Now there are. Its time to switch back.

V4L2 M2M It shall work on 32bit too, actually. I just haven't tested it yet.

cryptoAlgorithm · 2022-07-10T03:06:09Z

Time to reinstall I guess...

guysoft · 2022-07-10T11:06:35Z

I am not sure if we should switch from Ubuntu or not. Quick vote of hands - is there any preference to one or the other to you?

cryptoAlgorithm · 2022-07-10T12:42:59Z

If we can move over flawlessly and everything remains the same, I'm in vote of moving to raspbian since its easier to use tools everyone is familiar with like raspi-config etc. Ubuntu is lacking in that aspect.

…

Message ID: ***@***.***>

chudsaviet · 2022-07-10T16:24:13Z

As we see, RaspberryPi OS have better hardware support - correctly working V4L2 M2M is an example.
And now its stable in 64bit. Anyway, its a question for another topic.

My question was about building a copy of FFmpeg just for HLS on OS image build time. Will it be OK?

cp2004 · 2022-07-10T18:44:30Z

I am not sure if we should switch from Ubuntu or not. Quick vote of hands - is there any preference to one or the other to you?

It would make user-support much easier if the two distros were the same. No need for different guides or constant questions of which OS you installed before helping people.

chudsaviet · 2022-07-11T06:12:44Z

Created pull request #784

guysoft · 2022-07-11T10:13:25Z

Ok, based on this and a few other similar answers I will move the 64bit build to use the rpi image.
Also got a an answer from rpi I am referencing here:
RPi-Distro/pi-gen#481 (comment)

cryptoAlgorithm · 2022-07-13T07:17:05Z

Ok! Does the 64-bit image work on 32-bit Pis? if it does then you could just use one image, no need for 2.

chudsaviet · 2022-07-13T16:20:50Z

No, 64bit image works on RPi 3+ and Zero 2+.

guysoft · 2022-07-17T21:46:02Z

64bit can only work on devices that can execute arm64bit instruction set. That means it will not run on Rpi1, zero 2, and 3, 1A+ etc

cryptoAlgorithm · 2022-07-20T02:21:46Z

Ok that's a small bummer. IMO tho, you shouldn't be using OctoPrint on a pi 1 or similar, the performance would be basically unusable. Yeah, you could just keep 2 images around then.

cryptoAlgorithm · 2022-07-20T02:23:12Z

@chudsaviet Does compiling ffmpeg from source work on Ubuntu too? I currently still have that installed.

cryptoAlgorithm · 2022-07-20T14:15:25Z

@chudsaviet Could you confirm if your ffmpeg version is above 5.0? I'm going to try compiling from source on my ubuntu image, hopefully I can get hardware encoding working too. If so, it could be possible to just stick with using ubuntu for the 64-bit image.

chudsaviet · 2022-07-20T14:39:21Z

Yes, of course, you can easily build FFmpeg from ‘master’ branch on Ubuntu. Its newer than any released version. Just use https://github.com/FFmpeg/FFmpeg .

chudsaviet · 2022-07-20T18:43:03Z

But on Ubuntu, I'm not sure you have the same latest kernel as on latest RpiOS. Kernel plays a major role in V4L2 M2M too.

chudsaviet mentioned this issue Nov 23, 2020

OctoPi 0.18.0 RC1 Status #692

Closed

guysoft mentioned this issue Dec 6, 2020

Webcam not appearing for 64bit 0.18 nightlies #700

Open

kennethjiang mentioned this issue Dec 8, 2020

[0.18 RC1 - 64-bit build]: OSError: libmmal.so: cannot open shared object file: No such file or directory #702

Open

guysoft mentioned this issue Dec 15, 2020

raspios_lite_arm64 images are not being updated RPi-Distro/pi-gen#481

Closed

guysoft mentioned this issue Jul 11, 2022

Switch arm64 nightly build to raspios_arm64 #785

Closed

HLS on arm64 #697

HLS on arm64 #697

Comments

chudsaviet commented Nov 23, 2020

chudsaviet commented Nov 23, 2020 • edited Loading

chudsaviet commented Nov 23, 2020

chudsaviet commented Nov 23, 2020

DrDrache commented Nov 23, 2020

chudsaviet commented Nov 23, 2020

chudsaviet commented Nov 26, 2020

chudsaviet commented Jan 9, 2021

fastcompjason commented Jun 9, 2021

guysoft commented Jun 9, 2021

chudsaviet commented Jun 10, 2021

cryptoAlgorithm commented Jul 5, 2022

cryptoAlgorithm commented Jul 5, 2022

chudsaviet commented Jul 5, 2022

cryptoAlgorithm commented Jul 5, 2022

cryptoAlgorithm commented Jul 5, 2022

chudsaviet commented Jul 5, 2022

cp2004 commented Jul 5, 2022

chudsaviet commented Jul 5, 2022

chudsaviet commented Jul 6, 2022

chudsaviet commented Jul 6, 2022

cryptoAlgorithm commented Jul 6, 2022 • edited Loading

chudsaviet commented Jul 10, 2022 • edited Loading

chudsaviet commented Jul 10, 2022

cryptoAlgorithm commented Jul 10, 2022

cryptoAlgorithm commented Jul 10, 2022

chudsaviet commented Jul 10, 2022

cryptoAlgorithm commented Jul 10, 2022

guysoft commented Jul 10, 2022

cryptoAlgorithm commented Jul 10, 2022 via email

chudsaviet commented Jul 10, 2022

cp2004 commented Jul 10, 2022

chudsaviet commented Jul 11, 2022 • edited Loading

guysoft commented Jul 11, 2022

cryptoAlgorithm commented Jul 13, 2022

chudsaviet commented Jul 13, 2022

guysoft commented Jul 17, 2022

cryptoAlgorithm commented Jul 20, 2022

cryptoAlgorithm commented Jul 20, 2022

cryptoAlgorithm commented Jul 20, 2022

chudsaviet commented Jul 20, 2022 • edited Loading

chudsaviet commented Jul 20, 2022 • edited Loading

chudsaviet commented Nov 23, 2020 •

edited

Loading

cryptoAlgorithm commented Jul 6, 2022 •

edited

Loading

chudsaviet commented Jul 10, 2022 •

edited

Loading

chudsaviet commented Jul 11, 2022 •

edited

Loading

chudsaviet commented Jul 20, 2022 •

edited

Loading

chudsaviet commented Jul 20, 2022 •

edited

Loading