Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Video smear observed with RTT=90m and latency=400ms over a ethernet interface #147

Closed
kmudide opened this issue Oct 25, 2017 · 26 comments
Closed

Comments

@kmudide
Copy link

kmudide commented Oct 25, 2017

I have checked out master branch and started testing while simulating our network conditions with tc and metem.

Test results
Observed video smear couple of times in 10 minutes video

Test Scenario
Both PC-1 and PC-2 are connected over a Ethernet connection

sudo tc qdisc add dev eth1 root netem delay 90ms
sudo tc qdisc show dev eth1
qdisc netem 8001: root refcnt 2 limit 1000 delay 90.0ms

PC-1 (Ubuntu):
./stransmit -t:0 -s:1000 udp://:11111 srt://10.1.50.7:22222?mode=rendezvous&latency=400
On this same PC, vlc is streaming big_buck_bunny_720p_h264.mov to localhost:11111

PC-2 (Centos):
./stransmit -t:0 -s:1000 srt://10.1.50.240:22222?latency=400&mode=rendezvous udp://:2002
On this same PC gstreamer is listening at localhost:2002 and sending to TV through video OUT.

Stats:
======= SRT STATS: sid=268422821
PACKETS SENT: 0 RECEIVED: 479188
LOST PKT SENT: 0 RECEIVED: 0
REXMIT SENT: 0 RECEIVED: 3775
RATE SENDING: 0 RECEIVING: 1.89605
BELATED RECEIVED: 0 AVG TIME: 0
REORDER DISTANCE: 0
WINDOW: FLOW: 8192 CONGESTION: 8192 FLIGHT: 0
RTT: 90.259ms BANDWIDTH: 2814.82Mb/s
BUFFERLEFT: SND: 12288000 RCV: 12003000
======= SRT STATS: sid=268422821
PACKETS SENT: 0 RECEIVED: 480047
LOST PKT SENT: 0 RECEIVED: 0
REXMIT SENT: 0 RECEIVED: 3776
RATE SENDING: 0 RECEIVING: 1.89702
BELATED RECEIVED: 0 AVG TIME: 0
REORDER DISTANCE: 0
WINDOW: FLOW: 8192 CONGESTION: 8192 FLIGHT: 0
RTT: 90.252ms BANDWIDTH: 3167.52Mb/s
BUFFERLEFT: SND: 12288000 RCV: 12214500
======= SRT STATS: sid=268422821
PACKETS SENT: 0 RECEIVED: 481013
LOST PKT SENT: 0 RECEIVED: 0
REXMIT SENT: 0 RECEIVED: 3778
RATE SENDING: 0 RECEIVING: 1.89554
BELATED RECEIVED: 0 AVG TIME: 0
REORDER DISTANCE: 0
WINDOW: FLOW: 8192 CONGESTION: 8192 FLIGHT: 0
RTT: 90.231ms BANDWIDTH: 1544.24Mb/s
BUFFERLEFT: SND: 12288000 RCV: 12265500

@kmudide
Copy link
Author

kmudide commented Oct 25, 2017

Can anyone suggest how I can improve video quality by changing any other parameters?

@ethouris
Copy link
Collaborator

ethouris commented Oct 26, 2017

In general, please don't forget that by default SRT does "TLPKTDROP", that is, if the lost packet is not recovered in time (that is, before the next-to-lost packet becomes ready to play), it's dropped. If you are experiencing this too often, this simply means that your network is not fast enough for the current value of the latency (default 120ms). Increasing the latency will make the video more delayed towards the sending site, but will give the receiver more time to recover lost packets and this way decrease the chance for "Too Late Packet Drop" to fire.

Turning off TLPKTDROP is rather a bad idea because it will tear the sending speed in case when a packet lingers too long with recovering, and this way you'll see scraps on the video anyway.

If you experience a UDP packet reordering (also by testing through netem), you can also try the lossmaxttl option. This allows to put extra delay before requesting a retransmission, so that it waits a while longer for a packet seen as lost to come later out of order.

If you want to know what really happens at the receiver side, make sure that you compile with logs enabled (use ENABLE_LOGGING=1 or compile in debug mode), and try to use at least the "note" level in logs (-loglevel:note in stransmit). The event of dropping packets will be displayed with "DROPSEQ" info.

@ethouris
Copy link
Collaborator

One more thing here. I can see that the source stransmit sending reads the input from UDP. It's possible at least in theory that the packets are being dropped already there. Because I can't see any packet lost reported in the stats.

If you can, please try to repeat the test with feeding the source data to stransmist through the pipe (file://con) and see if you can observe any problems.

@kmudide
Copy link
Author

kmudide commented Oct 27, 2017

Ethouris,

Ok, I will test today and update it here. I am testing master branch. Is it stable enough? Suggest me, otherwise.

@ethouris
Copy link
Collaborator

At least master it's considered stable, so you should test there. Thanks.

@kmudide
Copy link
Author

kmudide commented Oct 27, 2017

Ethouris,

I see devastating results. Receiver side's SRT is stop printing stats after few seconds. Nothing is played by VLC, which is listening at localhost:11111 on receiver side PC. 

Here are the commands that I used. Let me know, if I am wrong.

Sender side command:
cat ../big_buck_bunny_720p_h264.mov | ./stransmit -t:0 -s:1000 file://con srt://10.1.50.7:22222?mode=rendezvous&latency=400&

Receiver side command:
./stransmit -t:0 -s:1000 srt://10.1.50.240:22222?mode=rendezvous&latency=400 udp://:11111&

======= SRT STATS: sid=342177881
PACKETS SENT: 0 RECEIVED: 9094
LOST PKT SENT: 0 RECEIVED: 327
REXMIT SENT: 0 RECEIVED: 39486
RATE SENDING: 0 RECEIVING: 14.4428
BELATED RECEIVED: 403 AVG TIME: 1.68622e+16
REORDER DISTANCE: 0
WINDOW: FLOW: 8192 CONGESTION: 8192 FLIGHT: 0
RTT: 50.179ms BANDWIDTH: 3316.58Mb/s
BUFFERLEFT: SND: 12288000 RCV: 1258500
======= SRT STATS: sid=342177881
PACKETS SENT: 0 RECEIVED: 10537
LOST PKT SENT: 0 RECEIVED: 327
REXMIT SENT: 0 RECEIVED: 39486
RATE SENDING: 0 RECEIVING: 15.8946
BELATED RECEIVED: 515 AVG TIME: 1.84467e+16
REORDER DISTANCE: 0
WINDOW: FLOW: 8192 CONGESTION: 8192 FLIGHT: 0
RTT: 50.15ms BANDWIDTH: 2961.71Mb/s
BUFFERLEFT: SND: 12288000 RCV: 301500
======= SRT STATS: sid=342177881
PACKETS SENT: 0 RECEIVED: 11784
LOST PKT SENT: 0 RECEIVED: 327
REXMIT SENT: 0 RECEIVED: 39486
RATE SENDING: 0 RECEIVING: 16.9262
BELATED RECEIVED: 515 AVG TIME: 1.84467e+16
REORDER DISTANCE: 0
WINDOW: FLOW: 8192 CONGESTION: 8192 FLIGHT: 0
RTT: 50.216ms BANDWIDTH: 3824.18Mb/s
BUFFERLEFT: SND: 12288000 RCV: 322500
======= SRT STATS: sid=342177881
PACKETS SENT: 0 RECEIVED: 11784
LOST PKT SENT: 0 RECEIVED: 327
REXMIT SENT: 0 RECEIVED: 39486
RATE SENDING: 0 RECEIVING: 16.1542
BELATED RECEIVED: 515 AVG TIME: 1.84467e+16
REORDER DISTANCE: 0
WINDOW: FLOW: 8192 CONGESTION: 8192 FLIGHT: 0
RTT: 50.216ms BANDWIDTH: 3824.18Mb/s
BUFFERLEFT: SND: 12288000 RCV: 1822500
======= SRT STATS: sid=342177881
PACKETS SENT: 0 RECEIVED: 11784
LOST PKT SENT: 0 RECEIVED: 327
REXMIT SENT: 0 RECEIVED: 39486
RATE SENDING: 0 RECEIVING: 15.4495
BELATED RECEIVED: 515 AVG TIME: 1.84467e+16
REORDER DISTANCE: 0
WINDOW: FLOW: 8192 CONGESTION: 8192 FLIGHT: 0
RTT: 50.216ms BANDWIDTH: 3824.18Mb/s
BUFFERLEFT: SND: 12288000 RCV: 3322500
======= SRT STATS: sid=342177881
PACKETS SENT: 0 RECEIVED: 11784
LOST PKT SENT: 0 RECEIVED: 327
REXMIT SENT: 0 RECEIVED: 39486
RATE SENDING: 0 RECEIVING: 14.8037
BELATED RECEIVED: 515 AVG TIME: 1.84467e+16
REORDER DISTANCE: 0
WINDOW: FLOW: 8192 CONGESTION: 8192 FLIGHT: 0
RTT: 50.216ms BANDWIDTH: 3824.18Mb/s
BUFFERLEFT: SND: 12288000 RCV: 4822500
======= SRT STATS: sid=342177881
PACKETS SENT: 0 RECEIVED: 11784
LOST PKT SENT: 0 RECEIVED: 327
REXMIT SENT: 0 RECEIVED: 39486
RATE SENDING: 0 RECEIVING: 14.2098
BELATED RECEIVED: 515 AVG TIME: 1.84467e+16
REORDER DISTANCE: 0
WINDOW: FLOW: 8192 CONGESTION: 8192 FLIGHT: 0
RTT: 50.216ms BANDWIDTH: 3824.18Mb/s
BUFFERLEFT: SND: 12288000 RCV: 6322500
======= SRT STATS: sid=342177881
PACKETS SENT: 0 RECEIVED: 11784
LOST PKT SENT: 0 RECEIVED: 327
REXMIT SENT: 0 RECEIVED: 39486
RATE SENDING: 0 RECEIVING: 13.6617
BELATED RECEIVED: 515 AVG TIME: 1.84467e+16
REORDER DISTANCE: 0
WINDOW: FLOW: 8192 CONGESTION: 8192 FLIGHT: 0
RTT: 50.216ms BANDWIDTH: 3824.18Mb/s
BUFFERLEFT: SND: 12288000 RCV: 7822500
======= SRT STATS: sid=342177881
PACKETS SENT: 0 RECEIVED: 11784
LOST PKT SENT: 0 RECEIVED: 327
REXMIT SENT: 0 RECEIVED: 39486
RATE SENDING: 0 RECEIVING: 13.1543
BELATED RECEIVED: 515 AVG TIME: 1.84467e+16
REORDER DISTANCE: 0
WINDOW: FLOW: 8192 CONGESTION: 8192 FLIGHT: 0
RTT: 50.216ms BANDWIDTH: 3824.18Mb/s
BUFFERLEFT: SND: 12288000 RCV: 9322500
======= SRT STATS: sid=342177881
PACKETS SENT: 0 RECEIVED: 11784
LOST PKT SENT: 0 RECEIVED: 327
REXMIT SENT: 0 RECEIVED: 39486
RATE SENDING: 0 RECEIVING: 12.6832
BELATED RECEIVED: 515 AVG TIME: 1.84467e+16
REORDER DISTANCE: 0
WINDOW: FLOW: 8192 CONGESTION: 8192 FLIGHT: 0
RTT: 50.216ms BANDWIDTH: 3824.18Mb/s
BUFFERLEFT: SND: 12288000 RCV: 10822500

@kmudide
Copy link
Author

kmudide commented Oct 27, 2017

In one occasion, receiver side SRT is printing stats with lots of lost packets.

======= SRT STATS: sid=192438995
PACKETS SENT: 0 RECEIVED: 20958
LOST PKT SENT: 0 RECEIVED: 56578
REXMIT SENT: 0 RECEIVED: 39631
RATE SENDING: 0 RECEIVING: 6.06545
BELATED RECEIVED: 486 AVG TIME: 1.84467e+16
REORDER DISTANCE: 0
WINDOW: FLOW: 8192 CONGESTION: 8192 FLIGHT: 0
RTT: 50.211ms BANDWIDTH: 8521.74Mb/s
BUFFERLEFT: SND: 12288000 RCV: 12100500
======= SRT STATS: sid=192438995
PACKETS SENT: 0 RECEIVED: 27685
LOST PKT SENT: 0 RECEIVED: 104728
REXMIT SENT: 0 RECEIVED: 39635
RATE SENDING: 0 RECEIVING: 5.43699
BELATED RECEIVED: 486 AVG TIME: 1.84467e+16
REORDER DISTANCE: 0
WINDOW: FLOW: 8192 CONGESTION: 8192 FLIGHT: 0
RTT: 50.199ms BANDWIDTH: 6406.79Mb/s
BUFFERLEFT: SND: 12288000 RCV: 12106500
======= SRT STATS: sid=192438995
PACKETS SENT: 0 RECEIVED: 33283
LOST PKT SENT: 0 RECEIVED: 152913
REXMIT SENT: 0 RECEIVED: 39639
RATE SENDING: 0 RECEIVING: 4.95023
BELATED RECEIVED: 486 AVG TIME: 1.84467e+16
REORDER DISTANCE: 0
WINDOW: FLOW: 8192 CONGESTION: 8192 FLIGHT: 0
RTT: 50.221ms BANDWIDTH: 5796.61Mb/s
BUFFERLEFT: SND: 12288000 RCV: 12165000
======= SRT STATS: sid=192438995
PACKETS SENT: 0 RECEIVED: 39778
LOST PKT SENT: 0 RECEIVED: 201135
REXMIT SENT: 0 RECEIVED: 39644
RATE SENDING: 0 RECEIVING: 4.75676
BELATED RECEIVED: 486 AVG TIME: 1.84467e+16
REORDER DISTANCE: 0
WINDOW: FLOW: 8192 CONGESTION: 8192 FLIGHT: 0
RTT: 50.201ms BANDWIDTH: 3931.04Mb/s
BUFFERLEFT: SND: 12288000 RCV: 12279000
======= SRT STATS: sid=192438995
PACKETS SENT: 0 RECEIVED: 47322
LOST PKT SENT: 0 RECEIVED: 257439
REXMIT SENT: 0 RECEIVED: 39648
RATE SENDING: 0 RECEIVING: 4.60916
BELATED RECEIVED: 486 AVG TIME: 1.84467e+16
REORDER DISTANCE: 0
WINDOW: FLOW: 8192 CONGESTION: 8192 FLIGHT: 0
RTT: 50.195ms BANDWIDTH: 8960Mb/s
BUFFERLEFT: SND: 12288000 RCV: 12229500

@ethouris
Copy link
Collaborator

Ok, I can see, we need some good guide for live video streaming using SRT.

I'll try to explain the problem short: stransmit is a transmitter, not a player. If you want to make a live stream transmission out of the file, you need some intermediate tool, which will control the speed of the data. Stransmit can read from file, of course, but it reads it always as fast as possible because it has no idea about the timestamps in the TS stream, and therefore can't synchronize the sending speed with them.

Tools such as VLC or ffmpeg can do this "live stream playing" for you; you should use them and make them stream to the standard output, which you can connect by pipe to the standard input of stransmit. Just make sure that you set the payload size as 1316. There was some instruction shown already in one of the issues; I'll try to dig this up and put into some documentation.

@ethouris
Copy link
Collaborator

I found some instructions how to do it in Issue #86. There's some option that states the live transmission, and this pkt_size=1316 added to the output URI is very important.

@kmudide
Copy link
Author

kmudide commented Oct 27, 2017

If you don't pass packet size, stransmit sets it to DEFAULT_CHUNK, which is defined as 1316. Here is the snippet of the code from stransmit.cpp.
if ( chunk == 0 )
chunk = DEFAULT_CHUNK;

Do I have to set VLC or ffmpeg's output stream size to 1316 as well?

@ethouris
Copy link
Collaborator

You have to make sure that the data will be sent in 1316-bytes portions. It's more significant for UDP because when stransmit reads UDP, it simply gets up to 1316 bytes in one call, so if there is more available, it will be lost. From the pipe, stransmit shouldn't lose anything, but if chunks on the input are bigger, the timing for the data may get serious discrepancy, which will lose all the advantages of SRT and its "delivery time smoother".

DEFAULT_CHUNK is 1316. The new version is more strict about the size and this size is also added to the developer API file (srt.h), as well as it's the default maximum chunk size for live mode.

@kmudide
Copy link
Author

kmudide commented Oct 27, 2017

Ok, I will try to test and update here. Thanks.

@kmudide
Copy link
Author

kmudide commented Oct 30, 2017

VLC is sending 1316 byes, which matches to SRT packet size, to localhost:11111

@ethouris
Copy link
Collaborator

ethouris commented Nov 1, 2017

Try to make VLC send to standard output instead and see what happens. Try also to read the stream via UDP from VLC and save it to a file directly. It has happened several times that I've observed disturbances when too many applications at a time do live streaming over UDP on the same machine.

@kmudide
Copy link
Author

kmudide commented Nov 2, 2017

Ethouris,

I am using following command to send a movie file to stdout and then piping it to stransmit via stdin.

vlc ../big_buck_bunny_720p_h264.mov --sout '#transcode{vcodec=h264,scale=Auto,acodec=mpga,ab=128,channels=2,samplerate=44100}:file{mux=mp4,dst=/dev/stdout,no-overwrite}' | ./stransmit file://con srt://10.1.30.7:22222?mode=rendezvous&latency=300

I see that the vlc is playing 10 minute video in just couple minutes and receiver side vlc does not display anything.
stransmit reading faster than it supposed to, from the stdin?

Any insight

@ethouris
Copy link
Collaborator

ethouris commented Nov 2, 2017

You know, this is a process involving multiple parts, any of which may fail to work.

Start with: vlc ./big_.... (whatever set of options) | ffplay -. (Or maybe vlc has also some possibility to read from stdin).

You can also use a dummy pipe from stransmit (both arguments specified as file://con) and intercept the pipe as specified above. At least it will read the input the same way as it would in case of any other transmission. When this works ok, you can also see if ./stransmit file://con output.ts also works as it should.

Also use -v option to see if any data are transferred at all. It will also inform you about any connection problems.

Stransmit cannot read faster than the input data are delivered. When the data are not available at the time when stransmit is trying to read them, it's waiting until they are available.

@kmudide
Copy link
Author

kmudide commented Nov 2, 2017

I followed your suggestions and noticed that the ffplay does not play anything either. I think some command line arguments in vlc is wrong.

vlc ../big_buck_bunny_720p_h264.mov --sout '#transcode{vcodec=h264,scale=Auto,acodec=mpga,ab=128,channels=2,samplerate=44100}:file{mux=mp4,dst=/dev/stdout,no-overwrite}' | ffplay -

@kmudide
Copy link
Author

kmudide commented Nov 2, 2017

After I changed mux to ts from mp4, the following command works and ffplay plays movie.

vlc ../big_buck_bunny_720p_h264.mov --sout '#transcode{vcodec=h264,scale=Auto,acodec=mpga,ab=128,channels=2,samplerate=44100}:file{mux=ts,dst=/dev/stdout,no-overwrite}' | ffplay -

However, I see same issue if feed it to stransmit via stdin.

vlc ../big_buck_bunny_720p_h264.mov --sout '#transcode{vcodec=h264,scale=Auto,acodec=mpga,ab=128,channels=2,samplerate=44100}:file{mux=ts,dst=/dev/stdout,no-overwrite}' | ./stransmit file://con srt://10.1.30.7:22222?mode=rendezvous&latency=300

@ethouris
Copy link
Collaborator

ethouris commented Nov 2, 2017

As I said, try dummy pipe first, then stransmit to a file. And use -v option with stransmit. Last resort, turn on logs.

@kmudide
Copy link
Author

kmudide commented Nov 2, 2017

You mean like this?

vlc | stransmit file://con file://con | ffplay -

@kmudide
Copy link
Author

kmudide commented Nov 2, 2017

The following command works fine too.

vlc ../big_buck_bunny_720p_h264.mov --sout '#transcode{vcodec=h264,scale=Auto,acodec=mpga,ab=128,channels=2,samplerate=44100}:file{mux=ts,dst=/dev/stdout,no-overwrite}' | stransmit file://con file://con | ffplay -

@ethouris
Copy link
Collaborator

ethouris commented Nov 2, 2017

Try this rendezvous then with -v option. You'll see at least if they connect to one another.

@kmudide
Copy link
Author

kmudide commented Nov 2, 2017

I am always using rendezvous mode. I enabled -s option on receiving side stransmit and see printing stats and the bit rate is around mbps.

@maxsharabayko maxsharabayko added this to the v.1.3.3 milestone Mar 8, 2019
@maxsharabayko maxsharabayko modified the milestones: v.1.3.3, v.1.3.4 May 29, 2019
@maxsharabayko
Copy link
Collaborator

@kmudide Please collect CSV stats with srt-live-transmit. Using command-line options -s:<bitrate/(8*1500)> -statsout:srtstats.csv -pf:csv.
Use srt-live-transmit -help to get usage help.

@maxsharabayko
Copy link
Collaborator

Please also bear in mind buffer size and maxbw configuration.
The size should be at least Bufrcv >= bps × RTTsec / (8 × (1500 - 28)) + (latencysec × bps)
Bear in mind #700.

Default receiver buffer size is 8192 × (1500-28) = 12058624 bytes or approximately 96 Mbits.
Default flow control window size is 25600 packets or approximately 300 Mbits.

Default maxbw is 30 Mbps, refer to #552.

@maxsharabayko maxsharabayko removed this from the v1.3.4 milestone Aug 9, 2019
@maxsharabayko maxsharabayko modified the milestone: v1.4.1 Aug 9, 2019
@maxsharabayko
Copy link
Collaborator

No activity. Closing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants