h264 stream playback

[yhrgsp]

hi,
I'm using windows 10 and vs 2019.
I need to play h264 video stream.
The code stops when execute "FFmpegMediaSource.CreateFromUriAsync("udp://127.0.0.1:1230",configuration)" and never continue.

this is my code:

            MediaSourceConfig configuration = new MediaSourceConfig()
            {
                MaxVideoThreads = 8,
                SkipErrors = uint.MaxValue,
                DefaultBufferTime = TimeSpan.Zero,
                FastSeek = true,
                VideoDecoderMode = VideoDecoderMode.ForceFFmpegSoftwareDecoder,
            };
            configuration.FFmpegOptions.Add("tune", "zerolatency");
            configuration.FFmpegOptions.Add("flags", "low_delay");
            FFmpegMediaSource decoder = await FFmpegMediaSource.CreateFromUriAsync("udp://127.0.0.1:1230",configuration);
            var mediastreamsource = decoder.GetMediaStreamSource();
            var mediaStreamSource = decoder.GetMediaStreamSource();
            mediaStreamSource.BufferTime = TimeSpan.FromSeconds(0);
            var FrameServer = new Windows.Media.Playback.MediaPlayer() { IsVideoFrameServerEnabled = true };
            FrameServer.Source = MediaSource.CreateFromMediaStreamSource(mediaStreamSource);
            FrameServer.RealTimePlayback = true;
            FrameServer.VideoFrameAvailable += MediaPlayer_VideoFrameAvailable;
            FrameServer.Play();

My ffmpeg stream is:
ffmpeg -stream_loop -1 -re -i video.mp4 -c:v copy -f mpegts udp://127.0.0.1:1230

[brabebhin]

Hi.

Does it work if you use VLC?

[lukasf]

This reminds me of #212. FFmpeg hangs in avformat_open_input and never returns.
Looks like it cannot detect the format correctly and tries forever.

[yhrgsp]

Hi.

Does it work if you use VLC?

Yes, it is.

[lukasf]

Okay looks like this is a UWP problem. In UWP, for some stupid reason you cannot use UDP for local loopback. TCP loopback is possible at least when the debugger is attached. But UDP loopback is not possible. No error is given but no data is received, that is why ffmpeg hangs forever trying to detect the format.

During debugging, I noticed that our FFmpegOptions are not applied in C++/WinRT. Will push a fix soon. When setting "timeout" property in FFmpegOptions, the Create... call will at least return with an error after timeout.

[brabebhin]

It should work of you use the router/DHCP assigned IP instead of 127.0.0.1

[yhrgsp]

Okay looks like this is a UWP problem. In UWP, for some stupid reason you cannot use UDP for local loopback. TCP loopback is possible at least when the debugger is attached. But UDP loopback is not possible. No error is given but no data is received, that is why ffmpeg hangs forever trying to detect the format.

During debugging, I noticed that our FFmpegOptions are not applied in C++/WinRT. Will push a fix soon. When setting "timeout" property in FFmpegOptions, the Create... call will at least return with an error after timeout.

I tried use transmit udp from another computer and its still don't work.
rtsp protocol works fine.

[yhrgsp]

It should work of you use the router/DHCP assigned IP instead of 127.0.0.1

I tried and it didn't works

[lukasf]

Local IP (as per DHCP) are disabled as well for UDP in UWP apps..

I tried to connect to the stream hosted on a different PC. But I could not connect, not with FFmpegInteropX but also not with VLC. Not sure why it is not working. Firewall was disabled on both PCs.

Do you need to use UDP or is it an option to switch to a different format that works?

[yhrgsp]

Local IP (as per DHCP) are disabled as well for UDP in UWP apps..

I tried to connect to the stream hosted on a different PC. But I could not connect, not with FFmpegInteropX but also not with VLC. Not sure why it is not working. Firewall was disabled on both PCs.

Do you need to use UDP or is it an option to switch to a different format that works?

I need to use udp protocol.
When I stream from another PC I can watch from vlc or ffplay but not with FFmpegInteropX.

Also, with hardware encoder with rtsp protocol I can watch the stream with FFmpegInteropX but not if I stream rtsp With ffmpeg from another pc.

[lukasf]

I tried again. Streaming UDP from PC A to B or back does not work. Neither in FFmpegInteropX, nor in VLC or ffplay. ffplay shows 0 bytes received all the time. Firewalls are disabled. No clue why there is no data transfer at all.

The only thing that works is streaming on local PC (loopback) with ffplay as receiver. It takes quite a long time (up to 20 seconds) for it to pickup the stream. Probably low keyframe interval. But even VLC cannot connect to local stream (loopback).

[brabebhin]

Local IP (as per DHCP) are disabled as well for UDP in UWP apps..

I tried to connect to the stream hosted on a different PC. But I could not connect, not with FFmpegInteropX but also not with VLC. Not sure why it is not working. Firewall was disabled on both PCs.

Do you need to use UDP or is it an option to switch to a different format that works?

This is interesting. I'm pretty sure multicast and unicast work with local address "192.168.."

[lukasf]

I just read that ffmpeg cannot act as a rtsp server. It can only transmit data to a rtsp server (which will then be broadcasted by the server). And on rtp, it can only transmit a single stream, so either video or audio, but not both. That's unfortunate, because both use UDP as transport layer, and receiving rtp and rtsp does work with FFmpegInteropX.

[lukasf]

Oh lol I actually just managed to get udp transmission working, from different PC to FFmpegInteropX! I used wrong IP addresses, which is why it did not work before. Of course, on the server I had to use the IP of the target receiver (and on the receiver the same IP and port). You also need to set the app capabilities for local network.

The experience was not perfect: Sometimes it would not even start because it could not find the format. Might need to set higher probesize parameter. I guess it can only get the format on a keyframe. Sometimes it would error out quickly (incorrectly getting EOF because reading too fast / no data). When it worked, it had a bit of trouble and lag in the beginning, but once it got clear, it was playing nicely. Once or twice I saw a broken packet wich would destroy the image until next keyframe. But that cannot be avoided if using UDP over lossy wifi connection.

The stream-buffer branch might help with the experience. It allows to start buffering before starting playback. That should avoid the incorrect EOF error situation and allow more stable playback.

I will push a PR with the app capabilities set in the samples plus the fix for FFmpegOptions.

[brabebhin]

You should be able to do it on a single PC as well, as long as you use the 192 address to connect (which means a round trip to the router). Only 127 address is blocked in uwp.

[lukasf]

You should be able to do it on a single PC as well, as long as you use the 192 address to connect (which means a round trip to the router). Only 127 address is blocked in uwp.

I tried that on both PCs and it is not working. I also read that this does not help. That would be way too easy as a workaround, then they could just allow loopback then. Packets also do not take round trip if sent to own PC IP Address. Pinging your own IP never has latency since the IP protocol finds the shortest way.

[brabebhin]

Strange. Tcp works this way.
It isn't quite the same as loopback. First, you need to get the permission for local network.

Then 127.0.0.1 and 192.168 are different interfaces as well. You normally don't bind 192 addresses to do IPC because there's actual packages going to the router and back, whereas that's not the case with loopback. This is a clear security breach.

I used to implement udp network discovery in my own app (which is a multicast + unicast back from whoever heard the multicast) and it would always discover itself, which means it was able to get the multicast and also unicast on itself. This is a pretty standard behavior. Unless this was a bug to begin with and was patched recently.

[brabebhin]

Ah ok maybe that might work cause it is the same app and technically not an IPC.

[lukasf]

I can only say that from my testing, local loopback did not work, no matter if I used 127.x or my router assigned IP. I also tested this once again, after I successfully streamed from second machine. It's too bad, because that would make testing so much easier.

The change in EOF handling did not help with sudden stream stopping. The stream really returns EOF and subsequent reads would fail as well. Not sure why this is happening.

I have merged the changes to the stream-buffer branch as well, so you can test this with the new buffer. I'd recommend to call StartBuffering() after creating the FFmpegMediaSource, and then wait for a few seconds before assigning the FFmpegMediaSource to MediaPlayer. That way, you should get more stable playback due to local buffering.

I will be on vacation the next two weeks, so I won't be able to go further with this until in 2 or 3 weeks.

Oh and on stream-buffer branch, you can use Shift+Return shortcut to open last stream in the sample apps!

[brabebhin]

Enjoy your vacation!

[lukasf]

Thank you!