Timestamp correction for janus-pp-rec

[tbence94]

Summary

Devices can send wrong RTP timestamps after some kind of interruption.
For example mobile devices tend to send wrong timestamps after receiving a call.

This feature attempts to provide a solution for correcting wrong RTP timestamps in the recorded streams by adding some optional flags to janus-pp-rec.

Problem

Use case

We managed to reproduce this issue with both iOS and Android devices by accepting a call while streaming.
The recorded and converted video stream was OK but the audio stream after the call was not in sync with the video.

This was due to the lack of silence in the audio stream. There should have been a gap in the timestamps at the call.

Example

Let's say there was a 5s long call at 10s and the whole video is 30s long.

The video stream looks like this (normal):

• timestamps go from 0s to 10s
• there is a gap in the RTP packets
• the next timestamp is 15s then it continues to grow to 30s

But the audio stream looks like this (faulty):

• timestamps go from 0s to 10s
• there is NO a gap in the RTP packets!
• the timestamps continue to grow from 10s then it continues to grow to 25s

When we convert this using janus-pp-rec the output file will have a 30s long video with 5s black-screen in it starting at 10s.
But there is no gap in the audio timestamps so if we made any sound at 16s (in reality) it will be replayed at 11s in the converted file.

How to detect?

We can detect issues like this by comparing the RTP timestamps to the arrival time of the packets.
(Introduced here: #1719)

There is a sudden jump in latency after events like the one described in the use case section.

Solution

How to correct?

The current implementation provides a solution which compares the current latency (arrival time - timestamp) to the
moving average latency of the last 10 (default) packets. If the current latency is higher then then the moving average
multiplied with (the number provided as the restamping flag value divided by 1000) than we calculate an offset based
on the latency which will be added to the RTP timestamps starting from the current packet.

I also introduced a minimum threshold to avoid correction of smaller "jumps" in latency which might be cuased by network issues.

Although this is not a perfect solution it helps to create something that might be closer to the thing that was seen and heard by the people watching the live stream.

NOTE: So far we've managed to restore 7 rooms for our clients using this method.

Flags and environment variables

In order to give more freedom to the users of janus-pp-rec my implementation has more flags than just a single threshold.
With these flags they can more easily describe the kind of "jump" they wish to correct in the stream.

restamp

  -r, --restamp=count (default=0)

  JANUS_PPREC_RESTAMP

If the latency of a packet is bigger than the moving_average_latency * (<restamp>/1000) the timestamps will be corrected.
Disabled if 0

NOTE: this is the "main switch" of this feature. The feature only work if this value is bigger than 0.

restamp-packets

  -c, --restamp-packets=count (default=10)

  JANUS_PPREC_RESTAMP_PACKETS

Number of packets used for calculating moving average latency for timestamp correction.

restamp-min-th

  -n, --restamp-min-th=milliseconds (default=500)

  JANUS_PPREC_RESTAMP_MIN_TH

Minimum latency of moving average to reach before starting to correct timestamps.
If the current latency is below this threshold the timestamps will not be changed.
Below the threshold we ignore the moving average.

[januscla]

Thanks for your contribution, @tbence94! Please make sure you sign our CLA, as it's a required step before we can merge this.

[tgabi333]

👍

[lminiero]

Thanks for the contribution and the detailed insight on this, it's very appreciated!
We won't be able to review for a week or so, but a couple of quick things I noticed:

1. indentation is broken: as per the project code style, we use tabs, not spaces;
2. since you added new command line arguments, janus-pp-rec.1 (the man file) should be updated too.

[tbence94]

1. indentation is broken: as per the project code style, we use tabs, not spaces;
2. since you added new command line arguments, janus-pp-rec.1 (the man file) should be updated too.

Thanks for your insights! I'll try and fix those things as soon as I can.

[tbence94]

I updated the manual page as requested and tried to fix the coding style as best as I could.
But CLion wasn't too helpful with the .editorconig so I am not sure it's all good...
Let me know if you spot any more coding style issues.

[atoppi]

@tbence94 thanks for the effort and the very good presentation of the implementation.
This is very interesting but I'd like to compare on a real sample the results of this restamping algorithm with the audio skew detection (ASD).
As far as I understand those features rely on the same data but operate in a slight different way, e.g.:

• you work in the time domain with double variables, while ASD works in RTP domain with integers
• you use a O(n) moving avg by back tracking previous N packets, while ASD evaluates a O(1) average with weights

Have you tried to solve the original issue with ASD ? Would you mind sharing a sample in order to validate the algorithm ?

[tbence94]

Thank you for the comment.

About audioskew and how did we get here...

As a matter of fact we tried to solve the issue with --audioskew at first because of an answer you gave for tgabi333 in a Helpdesk ticket (audio mjr file length #803681).

@atoppi Maybe you can still find the ticket and check the example .mjr files in it.
I also compared the audioskew with restamping on that one.

We implemented a solution using where we checked the audio and video duration with ffprobe.
And tried to minimize the difference in duration using the --audioskew flag with different values and with.
At smaller gaps (like the one in the ticket) we managed to convert something that was bad at the phone calls, but came in sync with the video again after the calls ended. It seemed like the audio was "stretched" to cover the gap.

But we also had streams with larger gaps (more than 10s) and multiple gaps (2-3x 10s).
Although we managed to create an audio stream which approximately had the same duration as the video, but after
the wrong parts the audio was not in sync with the video.

First we experimented with the values you recommended [40, 400]. But those values often didn't work for these large gaps.
With the solution described above, we found that the best audioskew value to was usually somewhere around the duration difference in milliseconds.

Since these lipsync problems seem to be high priority issues for our clients we needed needed a solution where these kind of gaps were filled with silence instead of "streching" the audio. Therfore I created this restamping solution.

Answers

You work in the time domain with double variables, while ASD works in RTP domain with integers
I usually work with JavaScript and other high level langaues. It is possible that I use the wrong types by mistake.
But it still seems to work just fine. :)
However if you think I should use some other type somewhere I am happy to help.

You use a O(n) moving avg by back tracking previous N packets, while ASD evaluates a O(1) average with weights
In my defense my moving average solution only goes back if the current latency which is still calculated in an O(1) fashion was bigger then a given threshold (See --restamping-min-th).

So we are not going back constantly to get the mavg. Only if there really seems to be a "big enough" jump in latency.
On those (possibly rear) occasions we have to go back, becuase we did not gather the mavg.

I think we could sacrifice some memory and create a solution where we store the N previous latency values so we could avoid back tracking. I created this version becuase I thought it would be easier to calculate this way (only when we need it).

Have you tried to solve the original issue with ASD ?
Yes. (See above)

Would you mind sharing a sample in order to validate the algorithm?
I'll ask my boss if we could share something more than the reference to the old ticket (audio mjr file length #803681).
But it would be better if someone else could help and reproduce more examples and test scenarios for this.

[lminiero]

I'll ask my boss if we could share something more than the reference to the old ticket (audio mjr file length #803681).

If you expect people to test this and provide feedback, we'll need an mjr file that is publicly available.

[lminiero]

PS: apologies for the late reply, we've been busy with the IETF meeting.

[tbence94]

If you expect people to test this and provide feedback, we'll need an .mjr file that is publicly available.

Hello. Thanks for your reply.

I can attach a short example. It includes the converted (original / faulty) result.

As you can see there is a part when you here him counting (around ~15s) but his lips are not moving or he is saying something totally different. And when he gets to the counting part in the video the audio already finished an there is only silence after that.

If I remember correctly we produced this by calling the phone while streaming but we did not take the call.
Then the audio continued after we declined the call.

I don't have too much time myself but let me know if I can help somehow to speed this up and I'll try to find time for this.
Thanks!

restamping-example.zip

[lminiero]

Thanks for the sample! We'll look into this, and do a more in depth review of your PR as well 👍

[atoppi]

This seems a valid enhancement in general. Still I have left some comments, check them out.

[lminiero]

@tbence94 any feedback on Alessandro's note above? Thanks!

[tbence94]

Sorry, but I am really swamped with work right now. I will try my best to find some time to answer...

[tbence94]

Hi @atoppi @lminiero !
I answered your questions. I am sorry that it took so long. Thanks for your patience!

[atoppi]

Thanks @tbence94 for the changes.
Now it LGTM 👍

[atoppi]

@tbence94 sorry I just remembered why I was mentioning seq nums fixing.
When trying to restamp the mjr you shared, the outputs says:

[WARN] Lost a packet here? (got seq 596 after 595, time ~21s)
[WARN] [FILL] pos: 000595, writing silences (count=476)

This is because when dealing with opus, janus-pprec evaluates the number of silence packets according to the RTP ts difference.

janus-gateway/postprocessing/pp-opus.c

Lines 76 to 83 in fee03d4

	if(tmp->prev != NULL && ((tmp->ts - tmp->prev->ts)/48/20 > 1)) {
	JANUS_LOG(LOG_WARN, "Lost a packet here? (got seq %"SCNu16" after %"SCNu16", time ~%"SCNu64"s)\n",
	tmp->seq, tmp->prev->seq, (tmp->ts-list->ts)/48000);
	ogg_packet *op = op_from_pkt((const unsigned char *)opus_silence, sizeof(opus_silence));
	/* use ts differ to insert silence packet */
	int silence_count = (tmp->ts - tmp->prev->ts)/48/20 - 1;
	pos = (tmp->prev->ts - list->ts) / 48 / 20 + 1;
	JANUS_LOG(LOG_WARN, "[FILL] pos: %06"SCNu64", writing silences (count=%d)\n", pos, silence_count);

As you can see that will lead to a confusing situation where:

• seq nums are okay (got seq 596 after 595)
• pp-rec fills the audio with silence packets (writing silences (count=476))

I guess this is not a big deal, since the user is being informed of the restamping algorithm kicking in.

[WARN] Timestamp gap detected. Restamping packets from here. Seq: 596

I'm not sure this would need a fix somewhere in the PR on in pprec code though.
Maybe someone else might have a suggestion.

[tbence94]

@atoppi
I don't see the problem. 😕 Maybe I misunderstood something.
What is the issue? The "writing silences" part in the output? I thought that was the whole point of this PR. 😄

This purpose of this PR is to shift the timestamps and write silences. In this case the original video should contain silences there... And as you said we informed the user about the restamping. Plus it's still an optional feature which is disabled by default. So the user can passing in the flags should probably expect these silences.

[atoppi]

Yeah probably this not an issue, but still worth mentioning.

In audio skew when the rtp timestamps are being increased due to a slow media clock, we also adjust sequence numbers, in order to obtain a coherent "fake" packet loss before letting pp-rec generate silence packets.

As I said not a big deal, but it might become an issue in case pp-rec will handle some audio codecs by seq nums and not rtp timestamps like we did before on g711/g722. In that case no silence will be generated.

[lminiero]

Sounds like something we can address later on if issues arise: in the meanwhile, it looks good to me, so merging. Thanks again!

[ansh1101]

restamping1.zip
Attaching this sample recording which has sync issue and not being fixed by restamp option

i am trying to fix this with restamping options but it doesn't look like it does justice to all of the affected recordings,

Please share the detail on how to use the restamp options properly to fix these type of issues

[ansh1101]

any updates on last comment