Decklink: Use frame scheduling API to fix output audio drift

[DDRBoxman]

This code uses the deckling scheduling API to use frame / audio timestamps to make sure that audio / video don't drift apart. This also add in clock drift compensation for this API.

This needs more testing but seemed to be successful with a 24 hour stream on my end.

[tt2468]

How bad is the drift without the fix?

[jfarre20]

I'm having some pretty bad drift/desync on a 24/7 system using a declink output to a channel injection system. I've seen desync as high as 600msec after a few days of obs uptime. I hope this isnt dead.

[PatTheMav]

Does this still require more extensive testing? I don't have or use decklink hardware, so it would be welcome to hear from people who do.

[jfarre20]

I haven't tested it yet, Last time I tried it was still drifting so I'm sticking with my external ffmpeg to decklink method for now

[tt2468]

@PatTheMav AFAIK last time I tested it it had some pretty bad desync issues too.

[norihiro]

I tested how the clockAdjustment works on Decklink Quad.
clockAdjustment went incremented until the drift reached the inflection point.
Then, drift keeps going to 0 and finally the moving average of the drift keeps toggling within +/- 60us.

When testing, I added one small change 12a7a0d to avoid the timestampOffset somehow jumps.

[norihiro]

If further reduction of jitter is required, I'd like to propose a lag lead filter.

I've created two filter implementations to for the feedback of the clock drift to clockAdjustment.

1. Proportional filter a5f511d
   This is a simple filter that multiplying constant value to the drift and set to clockAdjustment.
   Since it's just proportional, the drift won't go to zero.
   I think jitter of this filter is not better than current implementation but it shows faster response and simple implementation.

2. Lag lead filter 5e71717
   The lag lead filter is widely implemented in analog PLL (phase locked loop).
   This filter has better suppression of jitter than the proportional filter while maintaining steady-state drift to zero.
   However, the code is more complicated than the proportional filter.

I'd like to provide figures how the buffer amount, the drift, and clockAdjustment behaves with these filters. The timescale is different from my above comment.

Proportional filter with constant 1/32768

Lag lead filter with time constant 120s (This is my recommendation.)
The response at the startup is slower but the jitter on clockAdjustment looks reasonably small enough.

[tt2468]

I think I definitely like the simplicity of the proportional filter a lot. The only thing I notice is that it appears to be writing new offsets to the card very often, which somewhat worries me as we don't know whether writing new configuration values correlates to wear of the card's storage. That was the main focus behind my method, though my method definitely has flaws worth fixing.

Do you think it would be possible in a simple manner to apply the filter more aggressively while drift is still above a certain deviation? I also see that the average value is ignored. Perhaps it would be good to begin using the average value once drift has settled to a sustainable amount?

[norihiro]

we don't know whether writing new configuration values correlates to wear of the card's storage

Do you have a chance to ask Blackmagickdesign?
If the feature is designed for genlock to the host computer, it should accept a lot of settings and should not write into a non-volatile memory.

I also see that the average value is ignored. Perhaps it would be good to begin using the average value once drift has settled to a sustainable amount?

I was thinking to drop the moving-average instance. If keeping it, it might be worth to try always use it.

Btw, CalculateAndCorrectDrift might be better to be called from WriteAudio instead of DisplayVideoFrame. The frame rate usually varies between 24 fps to 60 fps but the common audio sample rate is 44.1 kHz and 48 kHz in OBS so far.

[norihiro]

The implementation of proportional filter is revised 59bcf19, which is at my branch.

• Averaging the error from 280 points (same as tt2468's implementaiton)
• Proportional filter
  • Added a constant CLOCK_ADJUST_DIVISOR = 8192. Since this is a proportional filter, the steady state error can be calculated from clockAdjustment * CLOCK_ADJUST_DIVISOR, which means at most 1.04 ms at the maximum clockAdjustment.
• Hysteresis to avoid frequent change to the device. I choose CLOCK_ADJUST_HYSTERESIS=1 from my experiments.

This figure is my experiments with Decklink Quad at CLOCK_ADJUST_DIVISOR=8192, DRIFT_AVERAGE_SAMPLES=280, CLOCK_ADJUST_HYSTERESIS=1.

[tt2468]

Tested your method on my end, and indeed it works very well. Not understanding why it always sits at ~-200us of drift, but at least it's steady and not really notable.

[PatTheMav]

@norihiro, @tt2468, @DDRBoxman so what is the state of this? The adjusted proportional filter seems to work well from the feedback I read (it looks like it uses slight adjustments to slowly oscillate in on a good clock alignment).

[PatTheMav]

Last call for this PR - it's over 6 years old by this point, is in draft stage, heavily outdated and my last CTA from 2023 went unanswered. If I don't see any further progress here I'll close it next week.

[norihiro]

As discussed above, my proposal should work fine. However, are there a demand to fix this issue? If so, I may create another PR but I guess no many users are waiting. IMO, just closing this PR should be fine.

[jfarre20]

I currently have my workaround with a local instance of FFMPEG connecting to the RTMP server that OBS is streaming to, but I am waiting for a fix still. This is for a slide show machine that outputs to a tv channel insertion box over composite.

[PatTheMav]

As discussed above, my proposal should work fine. However, are there a demand to fix this issue? If so, I may create another PR but I guess no many users are waiting. IMO, just closing this PR should be fine.

Then I'd suggest opening a new PR with the proportional filter - from the limited domain knowledge I have and based on @tt2468's comment it seems to achieve the stability that people are looking for.