Optimizing RTSP Broadcast with FlyLeaf Player: Challenges in FPS Adjustments and Data Stream Speed

[fatihbahceci]

Hello, I am using the FlyLeaf player by feeding it with my own customized stream. (With a recorded RTSP broadcast)

For more information see #348

While doing this, there are scenarios where I need to neither speed up nor slow down the video. For these scenarios, I send my frames at certain intervals.
For example, for 20 FPS, I send 20 frames per second.
When it's 2x, I send 40 frames per second, and at 4x, it's 80 frames... and so on.
Therefore, for instance, when I send 320 frames per second at 16x, if the player uses it at (let's say) 25 FPS, when I change the FrameRate/Speed,

it doesn’t switch to the new data before the previous one finishes,
when I send data at the speed I want to achieve, the FlyLeaf player doesn’t play this data at the correct speed. (like playing at 1x when I do 2x, or playing at 2x when I do 4x)
Also, when we set the FlyLeaf player to catch up (player.Speed = 16f -MAX-), it loses video fluidity and shows only one frame every 5-6 seconds.

For these reasons, I think one of the following options could solve the problem.

1. Force the FlyLeaf to instantly render the data by sending it a high FPS (like 320) and making it settle with the data I send.
2. Turn off the FPS control mechanism in FlyLeaf and make it instantly render the incoming data.

I couldn't find an example related to FPS adjustment in the FlyLeaf codes, nor a member in the source code where I can set this.

I would appreciate your solution suggestion regarding this.
Thanks in advance.

[SuRGeoNix]

Hi @fatihbahceci, I'm not what you are trying to achieve. Everything that you say though it points me to MaxLatency as what you are asking is how Flyleaf can follow the the FPS/speed of the data?

[fatihbahceci]

Hello! Thanks for your response. What I'm trying to achieve is, when I send data to the player, I want it to render all the data immediately in succession and without waiting, not based on a fixed FPS.

Because, even if the player operates at, say, 30 FPS, I actually send data according to 60 FPS at time X, 120 FPS at time Y, and 15 FPS at time Z.
I already set the time between these frames while feeding them. Therefore, I need to either disable the FPS mechanism of the player or manually change it during playback, so that the player plays the frames I send at the correct speed.

Additionally, is it possible for me to clear the player's cache in some cases?

[SuRGeoNix]

@fatihbahceci I think what you need it must be set at your stream or through FFmpeg. What format/fps does you stream announce? Did you check FFmpeg options so you can force the stream's fps? You can call Player.Flush() for clearing the cache.

[fatihbahceci]

Hello, let me first ask a brief question and then explain why I am asking it. You mentioned, 'I think what you need it must be set at your stream or through FFmpeg.' I couldn’t find a place in the config class where I can set the fps through FFmpeg.

public PlayerConfig     Player      { get; set; } = new PlayerConfig();
public DemuxerConfig    Demuxer     { get; set; } = new DemuxerConfig();
public DecoderConfig    Decoder     { get; set; } = new DecoderConfig();
public VideoConfig      Video       { get; set; } = new VideoConfig();
public AudioConfig      Audio       { get; set; } = new AudioConfig();
public SubtitlesConfig  Subtitles   { get; set; } = new SubtitlesConfig();

Could you briefly tell me where to set the FFmpeg settings? I couldn’t find a scenario related to this in the examples of the project.

As for why I am asking, I had explained it before, but let me explain again. You will probably remember when reading. I have an application with the following tasks:

1. Connect to RTSP cameras and record the data to disk.
2. Provide live footage to connected clients.
3. Show previously recorded footage to connected clients.
4. Respond to requests like Forward, Backward, Fast Forward/Backward, Play From Time.

As you might remember, in Discussion #348, I had created a class that feeds the Player for this purpose. This class connects to our server and receives live/playback data frame by frame according to requests like 1x, 2x, 4x, etc.

Then I contacted you about this, and you provided very useful information and made new developments/bug fixes for this purpose. You informed me with the message, 'I've fixed some issues with h264/hevc formats yesterday and also added a few improvements. I just released a new package for you to test as I suspect that you were facing the issue that I've fixed. If not, let me know so I can look deeper into it.'

It should be one of these https://github.com/SuRGeoNix/Flyleaf/commits/master/?since=2023-12-01&until=2023-12-08.
I would like to thank you again for this rapid development.

As for the data properties and the server's operation,

The h264/hevc data coming via RTSP is recorded on the disk in raw form and sent to live clients in the same way.

So, it’s sufficient to feed the player with the data as it is.

Before FlyLeaf, we were rendering the data using our own resources with directdraw.
And here, we were directly rendering the incoming data without any concern for FPS.
Because our server was setting the FPS itself and sending as many frames to the client as needed.
The advantage of this was as follows. For example, when we wanted to do Forward/Backward at speeds like 2x, 4x, 8x, all we had to do was to send 40 frames instead of 20 or, with a little trick, omit some frames in between.
Now, however, as far as I understand, in FlyLeaf, since the FPS information is fixed (probably), the frame information I send is processed within the framework of these FPS rules.
If I can change the FPS value instantly by increasing or decreasing it through the ffplay settings you mentioned, or if I specify a high FPS from the start, I hope it might be solved.
Or it could be resolved if I completely bypass the FPS issue and make it render whatever I send immediately.

I am also open to considering any other solutions you might suggest.

[SuRGeoNix]

@fatihbahceci Sorry but I can't remember our previous conversations, hope you understand :)
I just realized you were talking about NoTimestamp formats (like h264/h265). I've added this code in VideoStream.cs

 if (Demuxer.Name == "h264" || Demuxer.Name == "hevc")
 {
     FixTimestamps = true;

     if (FPS == 0)
     {
         FPS = 25;
         FrameDuration = (long) (10000000 / FPS);
     }
 }

Which means that probably your stream does not announce properly the FPS (if it possible to announce it with those formats, I don't know) so it will se the default 25 Fps. If that's your issue I can expose that and you can set this manually.

[fatihbahceci]

I think manually changing the FPS will solve my problem. If I set the player to 300 FPS, I hope to turn it into a beast that can chew up and render every incoming Frame :)
What do you think? Wouldn’t it be a good idea to manipulate the playback speed with FPS? :)

I have a question of interest here. By changing this property, will it be possible to decrease and increase the FPS during playback? Or once set initially, does it remain the same FPS until I reconfigure and restart the player? I have my own solution plans for both scenarios.

Edit: Also, I just realized that; In the RTSP broadcast, in some cases, I should be able to determine the FPS myself. This is because while some cameras can be set to 2 FPS, 10 FPS and 30 FPS. I would need to inform the Player of this in some way ahead of time. Therefore, even if I hadn't done FFWD or BFWD, I should have been able to set this value. Thanks in advance for this development.

[SuRGeoNix]

If your hardware can handle I guess it will work. OK so I will have later today a new config to adjust this value however you should consider also checking your encoding process that might be possible to set the FPS there,

[fatihbahceci]

Also, I just realized that; In the RTSP broadcast, in some cases, I should be able to determine the FPS myself. This is because while some cameras can be set to 2 FPS, 10 FPS and 30 FPS. I would need to inform the Player of this in some way ahead of time. Therefore, even if I hadn't done FFWD or BFWD, I should have been able to set this value. Thanks in advance for this development.

[SuRGeoNix]

@fatihbahceci I've a new NuGet for testing. Just set this

Config.Demuxer.ForceFPS = 300;

Note: this is only for NOTIMESTAMP formats