Using RPI2 h264 decode frame delay

[atic-atac]

Hi all,

Getting this exact problem on an RPI2.

#232

Wondering if a firmware bump was needed for the RPI2 as well ?

[popcornmix]

The fix in #232 will apply to Pi1 and Pi2.

[atic-atac]

Mmm, running the latest but I see that I only get the port change event after the second frame.

[popcornmix]

Are you passing in the correct flags (e.g. OMX_BUFFERFLAG_ENDOFFRAME)?

[atic-atac]

Yes.

[popcornmix]

I think you'll have to provide a small program that demonstrates the issue.

[atic-atac]

O.k. let me come up with something.

[atic-atac]

Actually, it looks like it might be specific to the bit stream. If I use the video_decode sample with buck bunny a similar thing happens where you only get the port change after 9 frames.

[atic-atac]

I can see the buck bunny video has out of order frames, but the first I frame should push the decoder to output it immediately. Is there some reason why the decoder can't activate the port change after the first I frame ?

[6by9]

What is your timing on presenting input frames?
IIRC There is a decent sized FIFO at the front of the codec, so you can probably present quite a few buffers before the codec itself has managed to consume enough to get the information for the port change event.
As popcornmix says, an example app would be useful.

[atic-atac]

I'm using the default video_decode buffers which is 20 @ 80K.

Also using the example hello_video app with buck bunny video seems to give similar behaviour where after a number of frames you get the port change event. If there's a reason why the hello_video app can't present after the first I frame ? it might help me understand my issue.

[popcornmix]

hello_video plays a raw bitstream without framing.
If packets are framed (using OMX_BUFFERFLAG_ENDOFFRAME) then decoder can run straight after receiving a packet.
If packets are unframed (without OMX_BUFFERFLAG_ENDOFFRAME) then you need to submit another frame for decoder to detect the next NAL marker to work out if the current frame is complete or if next packet is a continuation of the previous NAL unit.

[atic-atac]

So I've added OMX_BUFFERFLAG_ENDOFFRAME to the hello_video app and used a sample of my h264 video, but I still see port changed after one I frame and two P frames. Which is the same symptoms I'm seeing in my standalone app.

I'm setting CodedPicture too in the BrcmDataUnit.

[deborah-c]

The decoder only starts to display frames once the DPB is full (or once reorder limits are hit); it needs to do this because there's nothing to stop an encoder from subsequently coding a frame that's earlier in display order. It does look ahead, so it doesn't need to have decoded subsequent frames all the way to pixels, but it at least needs to have seen the metadata at the start of the first slice of each subsequent picture that affects the DPB.

In the case of Big Buck Bunny, there are lots of versions out there, but at least some use a large DPB; for such streams, you're not going to get a frame emitted (and thus the port change generated) until several frames in, once the DPB fills.

[atic-atac]

Sorry for the delay on this. I've got a standalone app with sample h264 file.

Do you need the source code as well ?

[6by9]

Source would be the easiest way, as then we can tweak if we see something broken.

[atic-atac]

omxplay.zip

Attached are binaries just to see the problem for now. After the SPS/PPS/SEI packets we get an IDR frame then two P frames, and it's only on the second which forces the video to show.

I'm wondering if the SPS/PPS data has something that spawns this issue, but I'd need some guidance on what that is.

Oh, it sleeps for 1 second after each frame.

[atic-atac]

Anyone had chance to run the binary and see what happens ?

[atic-atac]

Ping ?

[atic-atac]

Source code at the ready. Just ping when you are ready.

[6by9]

Sorry, been busy with other stuff. I'll take a look when I get a chance.

[atic-atac]

O.k. I'll keep an eye on this thread when you reply.

[atic-atac]

Any news, is there anything else I can do to move this along ?

[atic-atac]

Don't worry, got it.

[6by9]

For the sake of the curious, could you elaborate on the solution?

[realjiangms]

@6by9 @popcornmix
Do we have more idea to eliminate the decoder delay? I'm recently trying to implement immediate h264 decoder on Pi3. After long time investigation and google searching, I found it is extremely hard. None of the result I search from network gives real solution.

I tried to set num_reorder_frames=0 and max_dec_frame_buffering=1 in sps, append OMX_BUFFERFLAG_ENDOFFRAME and OMX_BUFFERFLAG_ENDOFNAL in buffer flag, add NAL delimator (9) or even add some meanless SEI after pps. None of them really work. The best result is, currently it always hold 1 single frame. That is, push IDR, push P, then get the first frame, and later always get the previous frame instead of the current frame which has just been pushed.

This happens for several encoders: openh264, vaapi, etc. Those streams should be valid for no delay output (certainly, option for the encoders are carefully set for no delay output, e.g., B frame is forbidden) as many software decoders and arm SoC hardware decoders could generate immediate no delay result frame.

Until, I got one mystery stream (maybe generated by x264 which I didn't test) which dramatically be able to be decoded without any delay. I modified the stream and removed some SEI and type 9 NAL, now it only contains 1 SPS/PPS and one leading IDR NAL. It could still be decoded without delay. I tried to carefully tune my test encoder to generate a stream which have similar SPS/PPS as it, but still have delay.

Now I have 2 streams, both have 1 SPS/PPS and one single IDR NAL, both 1920x1080, both have num_reorder_frames=0 and max_dec_frame_buffering=1 in sps. But one can be decoded without delay, another can not. I think they are helpful to narrow this issue down.

Could you help to point out what's the key difference and how can we forcely get frames decoded immediately?

I upload the 2 test stream with only one IDR frame, and my test program in a zip, with the source code. The impact of different flag or IL configuration could be ruled out as 2 streams are tested with same program.
Files in zip:

delay.h264 -- stream with delay
nodelay.h264 -- stream without delay
video.c -- Source code of test program. It is just simple change on sample hello_pi/hello_video. To compile it, just replace the original video.c in sample.
hello_video.bin -- test program.

With nodelay stream, it will print message that port config changed after stream pushed into decoder:
./hello_video.bin nodelay.h264
Pushing buffer 10112
Port changed!!!!!
With delay stream, it stuck forever on waiting for OMX_EventPortSettingsChanged event

decoder_delay.zip

I'm running on Raspbian GNU/Linux 8 \n \l with kernel Linux raspberrypi 4.9.34-v7+
The note in #232 seems to say that the issue had already been fixed in 2013. But issue still exists after I updated with rpi-update and recompiled the sample and test binary.

[longo92-vrmedia]

I am experiencing the same issue on Raspeberry pi Zero-2 with bookworm O.S. (Linux bookworm-dev 6.1.0-rpi7-rpi-v7 #1 SMP Raspbian 1:6.1.63-1+rpt1 (2023-11-24) armv7l GNU/Linux).
Is there an update?

Thanks,
Alessandro

[popcornmix]

Commenting on an issue that was closed 8 years ago is not going to get much response.

This issue is related to the openmax api which has been deprecated for many years.
If you have a problem with bookworm, using supported APIs (V4L2) then create a new issue
explaining exactly what the problem is.