Stuttering Audio - Questions #291
Comments
|
Hi @tomvdb, first you need to verify if you actually get the audio 'slower' than the video. Because, if that happens then I'm not sure how you could avoid the audio shutter, the only way it would be to slow down the Fps. You can enable from the Engine's configuration the LogLevel (not ffmpeg's just flyleaf's) to Trace so you can see how the demuxer receives the packets and the timestamps. You can send it to me to analyze it as well. I think it could be stable if you would try something like:- playerConfig.Decoder.MaxAudioFrames = 40;
playerConfig.Demuxer.BufferDuration = TimeSpan.FromSeconds(30).Ticks;
playerConfig.Player.MinBufferDuration = TimeSpan.FromSeconds(5).Ticks;Just to see if it will be more stable, this is not good for low latency but you need to find the proper values to make it stable enough. Ideally, you need to ensure also that the other side sends the data properly (eg. splits the audio/video packets in away that you can play them in sync) |
|
ill play a bit with those settings, but its highly unlikely that I would get the audio slower than the video. Feeding the same data at the same rate to vlc plays fine. Im assuming that im dealing with the stream class correctly then? |
|
@tomvdb No, I didn't check the class, but at the first glance looks wrong (I will review it tomorrow and if you actually need to write a class like it I will try to make a sample). I though you used that just for testing and you had the shuttering when you were using a direct url? |
|
No, there is no url here, im getting the ts data straight from the hardware
via usb.
…On Sun, 30 Apr 2023, 23:10 John Stamatakis, ***@***.***> wrote:
@tomvdb <https://github.com/tomvdb> No, I didn't check the class, but at
the first glance looks wrong. I though you used that just for testing and
you had the shuttering when you were using a direct url?
—
Reply to this email directly, view it on GitHub
<#291 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAK2YO2G2EW7OHXX32DV5FTXD3INRANCNFSM6AAAAAAXRDTNXI>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
Did you check if you could avoid the customization and use a protocol/format that ffmpeg supports directly? Eg: with ffmpeg devices maybe (dshow)? I trace log will help to see if the ffmpeg reads the data properly (maybe add also ffmpeg to trace). I will have a better look tomorrow. |
|
Its a supported ffmpeg protocol (I can play the ts file with ffplay just
fine), but I need to feed the data directly, cant have ffmpeg connect to
device via usb. Im doing a similar thing on vlc side, basically use a
stream to send the data.
…On Sun, 30 Apr 2023, 23:15 John Stamatakis, ***@***.***> wrote:
Did you check if you could avoid the customization and use a
protocol/format that ffmpeg supports directly? Eg: with ffmpeg devices
maybe (dshow)?
—
Reply to this email directly, view it on GitHub
<#291 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAK2YO6LTFMRWXAH7YBOIPLXD3I7HANCNFSM6AAAAAAXRDTNXI>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
@tomvdb Sorry for yday, it was my turn to have a sleeping brain :) So, your Stream actually looks fine. I was thinking as you mention that this is happening only with some specific types of input, might the issue is with the buffer size. It's possible that is big enough (that's why you wait to fill it) and that delay causes the aDistanceMs messages (meaning that the audio distance come too late by X ms, minus means late plus means long gap without anything to play). I will expose the buffer size (as I was planning to do and forgot it somewhere in the middle of coding) so you can mess with this as well. Log log log, check VLC's buffer size as well. I'm just testing your MediaStream and passing some ts data and so far can't reproduce it but I will keep trying. Update 1: Found a bug from my side that does not respect the MinBufferDuration in case the stream's start time is not 0 (did some changes on that lately) |
|
Thanks John,
No worries, thanks for all the help.
Im going to keep playing with it today, i will collect some more logs and
get back to you shortly.
T
…On Mon, 01 May 2023, 07:21 John Stamatakis, ***@***.***> wrote:
@tomvdb <https://github.com/tomvdb> Sorry for yday, it was my turn to
have a sleeping brain :) So, your Stream actually looks fine. I was
thinking as you mention that this is happening only with some specific
types of input, might the issue is with the buffer size. It's possible that
is big enough (that's why you wait to fill it) and that delay causes the
aDistanceMs messages (meaning that the audio distance come too late by X
ms, minus means late plus means long gap without anything to play).
I will expose the buffer size (as I was planning to do and forgot it
somewhere in the middle of coding) so you can mess with this as well. Log
log log, check VLC's buffer size as well. I'm just testing your MediaStream
and passing some ts data and so far can't reproduce it but I will keep
trying.
—
Reply to this email directly, view it on GitHub
<#291 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAK2YO673VCR77LFSAMFW6DXD5B7DANCNFSM6AAAAAAXRDTNXI>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
@tomvdb I think the MinBufferDuration was the issue as I was able to reproduce it and what I see in .ts formats is that the audio packets are coming very slow comparing to video packets. So if you don't buffer enough packets it will not be able to play stable. I will make a new package for you to tested and confirm it. |
|
Thanks John, not sure if this confirms it, but seems like the it happens after OnBufferingCompleted events 08.25.47.230 | Debug | [#1] [Player ] OnBufferingStarted |
- Demuxer: Introduces Config.Demuxer.IOStreamBufferSize to specify the AVIO Context's buffer size - Player: Fixes an issue that it wouldn't respect MinBufferDuration (when demuxer's StartTime wasn't 0) (possible fix for #291) - Renderer: Adds HDR10+ support - Renderer: Adds HDR HLG transfer function support - Renderer: Adds Rotation support based on VideoStream DisplayMatrix data
|
I would prefer a trace but yes more or less it does confirm this. I wanted to see the demuxer's start time as well. playerConfig.Decoder.MaxAudioFrames = 40;
playerConfig.Demuxer.BufferDuration = TimeSpan.FromSeconds(30).Ticks;
playerConfig.Player.MinBufferDuration = TimeSpan.FromSeconds(5).Ticks;Try the above config with the new package (just to confirm that it works and then you can remove playerConfig.Decoder.MaxAudioFrames and use default which is 10, and probably less MinBufferDuration as well about 2 seconds) |
|
Hi John, It definitely seems better, although you can see the difference between VLC and FFMPEG in terms of VLC starting to play earlier. Also there aren't a lot of signals on the satellite now, but I managed to receive on where i've had some trouble before. Sounds a lot smoother, although there was a gap here and there. I've attached the trace log, essentially starting the app, tuning to the signal and starting receive |
|
@tomvdb I don't see any issues with the log. No audio/video has been dropped. It didn't have to re-buffer. The demuxer buffer duration stays stable at 4+ seconds. You can make it run faster, that was for testing purposes. Reduce MinBufferDuration to 1-2 seconds. You might also disable the Config.Demuxer.AllowFindStreamInfo to start faster. Reduce probesize analyze duration. More configs can make it start faster. Why do you still have audio gaps when the log shows the opposite? |
|
Not sure. I fired up my own tx and transmitted some video via the satellite. Attached is log, it still pops up with aDistance messages (although a lot less). I'll try out the config changes, btw - if I output the same data via udp and play via ffplay udp://127.0.0.1:9080 then there are now sound issues |
|
OK I will check it, here is a config that should meet your requirements :- playerConfig.Player.MinBufferDuration = TimeSpan.FromSeconds(1.5).Ticks;
playerConfig.Demuxer.BufferDuration = TimeSpan.FromSeconds(10).Ticks;
playerConfig.Demuxer.AllowFindStreamInfo = false;
playerConfig.Demuxer.FormatOpt["probesize"] = (5 * (long)1024 * 1024).ToString();
playerConfig.Demuxer.FormatOpt["analyzeduration"] = (2 * (long)1000 * 1000).ToString();
playerConfig.Decoder.MaxAudioFrames = 7;
playerConfig.Decoder.MaxVideoFrames = 3; |
|
@tomvdb You should try a release version to see if you still have the gaps. I think it's because of debug and logging which affects the performance. You can also possible try to increase the the threadpool threads(?). |
This seemed to have killed audio completely |
Good point, let me try the release side. |
|
It's the stream suggester plugin that fails to suggest the audio stream I will fix it (you can choose it yourself though). Otherwise until then:- playerConfig.Demuxer.AllowFindStreamInfo = true; |
|
Something went wrong somewhere, getting this in my vs designer
No packages, or anything like that has been changed, apart from your new package this morning. Not sure if this is related, but only noticed now. Still compiles, runs and plays video though. btw - does the ffmpeg.autogen version link to ffmpeg version ? (I'm using ffmpeg 6 files) currently still getting audio gaps (although a lot less) and a few received transmission videos looked funny, but could be unrelated. anyways, I have to get to my work. thanks for all the help, I'll keep on tinkering and seeing what I can find tonight. Thanks again, |
|
Probably you changed something in the constructor of your Form that triggers something that the designer does not like. Just use :- bool isDesigner = LicenseManager.UsageMode == LicenseUsageMode.Designtimeand prevent starting for example the Engine or things that designer will not like (I will do the same for WinForms FlyleafHost also in OnPaintBackground but I don't think that's FlyleafLib's issue). Yes if you update to FFmpeg.Autogen v6 then you can link v6 ffmpeg libraries fine. For the audio gaps I think now it's not a buffering issue (as your demuxer has buffer) but a 'hanging' issue of the application (affects the Thread.Sleep in FlyleafLib and instead of sleeping X seconds it sleeps more). You should try increasing the threadpool's threads. |
|
I had to reinstall flyleaf lib to get rid of that error, buts gone now. When you say Threadpool do you mean if I'm using a threadpool? or a thread pool that is part of flyleaf ? would it make a difference to start the player in a seperate thread from the ui thread? I noticed audio and video errors happen more if I drag the window around, so its definitely tied to the ui thread somehow? |
|
Would it make sense to move something into a thread (regarding flyleaf player)? I have a few other threads running (TS parser, Hardware Comms, etc), but VLC and Flyleaf related items are not in seperate threads. VLC doesn't seem to be affected by dragging window around while its playing, but on flyleaf audio stutter definitely goes bad when dragging the window around. Thanks again for all your efforts |
|
Yes, sounds more like a ThreadPool issue. Use something like this:- ThreadPool.GetMinThreads(out int workers, out int ports);
ThreadPool.SetMinThreads(workers + 6, ports + 6);I will consider also changing the main playback thread to high priority or have a config for that. |
|
Just updated the NuGet to v3.7.12. Now you can use playerConfig.Demuxer.AllowFindStreamInfo and changed also the threading to be outside from the thread pool. If you still have issues you might want to try playerConfig.Player.ThreadPriority to highest. |
|
Thanks John, I think I'm going to make a release of my tuner software with all the updates today and see what the feedback is. Its definitely a lot better. btw - when I install either one of the nuget packages (3..7.11 or 3.7.12) then I get the designer error. If I revert back to 3.7.10 then its not a problem. Nothing has been changed on my form constructor vs previous versions. I can still build it and run it, but everytime I go into the designer view I get a popup and then it looks like this:
I am running ffmpeg.autogen v6 as well btw - so not sure where the reference to 5 comes from in the error. |
|
I think is this commit 005359b#diff-b0818c5cd53759dfc5c4321668c78615c583fe67afa371f428151baa0e593bf0 I've added in the Player which tries to load the assembly OnPaintBackground:- The library uses FFmpeg.AutoGen v5 and that's why it confuses the designer (when you use v6 on your app). I will remove the reference on the Player for now. |
|
Let me know if v3.7.13 fixed the designer issue. And when you say a lot of better (did you notice difference with the thread instead of task? or does the UI/dragging still affects the playback?) (Btw, before you release it you might want to disable the embedded functionality such as Key Bindings, Mouse Bindings etc.) |
|
Sadly still same issue:
The biggest change was the buffer fixes you did earlier. The rest maybe slightly improves it, but dragging the window around I can still get the audio to stutter. But all in all its a lot more usable now, than what it was yesterday. ThreadPriority doesn't seem to make a noticeable difference. |
|
Can you try adding using FFmpeg.AutoGen; to you application in the mainwindow (before the control loads?) so it will load the v6 and hopefully it will stop complaining? (I can't reproduce it) |
|
Sorry, Doesn't seem to make a difference. Don't worry about it for now, everything is still working, its just annoying messages popping up when opening up the designer. I'll have a look tomorrow and maybe reinstall the packages again. Thanks for all the help with everything, much appreciated. |
|
Yes I see it now at your repo, I will check it tomorrow as well. Thanks for helping with the bugs ;) |
|
Hey @tomvdb, found and fixed also the issue with the designer. You should really consider updating to .Net 7/8. Probably, I will drop support for .Net framework in the near future (as also Vortice dependencies already dropped it). I'm closing this long thread and feel free to open a new issue! |
|
Hi John, Thanks for that. It seems to be working fine with the latest package. Much appreciated. Thanks for all the help, I'll keep you updated. Yes, upgrading is on the roadmap, but need to sort out a few other packages first that play so well with newer .Net Thanks! |
Hi John,
I've spent a good chunk today tracing the stuttering audio I'm having playing some (not all ts streams) as received by by tuner app. This post is more a general "this is what I found" to see if you have any ideas.
So I get these messages in the debug when the audio glitches happen
20.40.10.044 | Info | [#1] [Player ] aDistanceMs = -93
20.40.10.644 | Info | [#1] [Player ] aDistanceMs = -21
20.40.11.405 | Info | [#1] [Player ] aDistanceMs = -110
20.40.12.004 | Info | [#1] [Player ] aDistanceMs = -38
20.40.20.251 | Info | [#1] [Player ] aDistanceMs = -8
20.40.29.484 | Info | [#1] [Player ] aDistanceMs = -77
Which seems to make sense. Audio data is coming in slow, or something is being blocked.
(1) I have confirmed that the TS data is good, if I dump it into a file and then use ffplay to play it back, then all is well, no issues. The data I dumped was not just from the receiver, but directly the data I sent via the Read override in the stream opening in OpenAsync.
(2) I'm reading the TS data from the hardware, putting it into a buffer and then created my own stream class (inheriting Stream) and I passed that to your OpenAsync function.
The read function that is overridden basically reads from the queue when I get a read request, but sometimes I haven't received all the data yet, so I'm holding the read function (and I suspect that's probably where the problem comes from). Not because the buffer on your side is running empty, but rather that I'm blocking by holding it. The video still plays fine however.
I do have some questions around the read function.
public override int Read(byte[] buffer, int offset, int count)So, it doesn't seem like buffer size changes at any time, but count does seem to get less over time and then reset to a bigger number again.
This is some debug of what I see:
Buffer: Len: 2097152,0,1707152
21.16.58.678 | Info | [#1] [Player ] aDistanceMs = -567
21.16.58.719 | Info | [#1] [Player ] aDistanceMs = -143
Buffer: Len: 2097152,0,1697152
Buffer: Len: 2097152,0,1687152
Buffer: Len: 2097152,0,1677152
Buffer: Len: 2097152,0,1667152
Buffer: Len: 2097152,0,1657152
Buffer: Len: 2097152,0,1647152
Buffer: Len: 2097152,0,1637152
21.16.59.878 | Info | [#1] [Player ] aDistanceMs = -790
21.16.59.918 | Info | [#1] [Player ] aDistanceMs = -368
Buffer: Len: 2097152,0,1627152
Buffer: Len: 2097152,0,1617152
where first value is buffer length, second value is offset, and third value is count.
(a) I'm assuming count is more or less that is left in a buffer in the library?
(b) What does aDistanceMs mean exactly? Is something being blocked, or is it running out of data?
(c) I've added my stream class below, maybe I'm doing it completely wrong anyways
I've played with config.Player.MinBufferDuration thinking that it should build up a bit more of a buffer before it starts playing, but doesn't seem to make any difference.
Assuming that my assumption on the count parameter is correct (in that its the space left in a buffer in the library), i'm assuming there is enough space and that I'm blocking some thread. I have tried to just return 0 instead of holding it, but after a couple times of returning 0 then it stops anyways
This is my stream class
Any tips, ideas or aha moments in your own time will be greatly appreciated,
Thanks,
Tom
The text was updated successfully, but these errors were encountered: