libobs: Implement asynchronous sample rate conversion for audio by norihiro · Pull Request #6351 · obsproject/obs-studio

norihiro · 2022-04-18T16:06:07Z

Description

This PR implements asynchronous sample-rate conversion by utilizing a soft compensation feature in libswresample. A digital approach of asynchronous sample-rate conversion was proposed in 1, a PDF is available here. Also described in 2. The soft compensation in libswresample uses similar technique as 1. In addition, a traditional lag-lead filter is implemented to eliminate errors at steady state.

As described in the description of the PR #6133, next_audio_ts_min should be same as data->timestamp + conv_frames_to_time(sample_rate, in.frames). However, just overwriting next_audio_ts_min cannot compensate insufficient or too many number of samples. Instead, the difference between these values will be fed to the soft compensation.

A lag-lead filter is widely used in a conventional PLL design. In the feedback loop, the lag-lead filter is employeed for these purposes.

to eliminate errors at the steady state
to suppress jitter induced by fluctuations such as CPU workload and USB transfer.

The unity gain frequency of the feedback loop is set to 1/180 Hz so that the fluctuations faster than 3 minutes will be suppressed
and I believe users won't notice transition at the startup.

This PR also changes built-in plugins to enable the feature by default.

Note: Though the implementation looks working, I'd like to clean the code to make it more readable.

Motivation and Context

Since most of audio sources are asynchronous,
the number of samples provided from the audio sources is sometimes insufficient or too many,
which causes audio glitch as #4600 and audio/video desync.

Fix #4600.

How Has This Been Tested?

The code is tested on Fedora 34 with MOTU M2 and recording a tone signal.

Also analyzed step response and frequency response of the feedback system with my SPICE model.

Frequency response of the open loop transfer function. Gain curve crosses 0 dB at 5.55 mHz (inverse of 3 minutes) and shows suppression for higher frequency components to eliminate fluctuation such as USB and load of the host. At 5.55 mHz, the phase curve shows 60 degree phase margin so that the feedback system is stable.

Simulated and measured step response of the system. This figure shows step response how the compensation is applied (vc1) and another internal node in the lag-lead filter (vc2), measured with MOTU M2 with Fedora 34. It estimated roughly 4.5 samples are are missing for every second. The implementation gradually started to add samples and reached maximum compensation -5.3 samples per second after 80 seconds from the start, then goes to the steady state compensating 4.5 samples per second. The other node vc2 slowly track vc1 so that it eliminates offset at the steady state. The curves of measured and simulated model are well matched except between 100 seconds to 300 seconds. I think we can rely on the model so that the gain and phase curves are well simulated and conclude the feedback system has enough stability.

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)

Checklist:

My code has been run through clang-format.
I have read the contributing document.
My code is not on the master branch.
The code has been tested.
All commit messages are properly formatted and commits squashed where appropriate.
I have included updates to all appropriate documentation.

digitalchou · 2022-04-20T16:51:52Z

Hi, if you need a macos intel tester Ill be glad!! Figthing with this bug till i bought a new Maudio Mtrack DUO 48kHz soundcard, but it sound a little worse than mi UMC404-192kHz one(wich dont exactly match the clock in 48kHz mode and drops a few ms of audio every 2:20 min aprox)
Thanks for your work, your source color minitor is one of the best obs plugins ever!!

norihiro · 2022-04-21T03:01:35Z

Hi, if you need a macos intel tester Ill be glad!!

I appreciate your comment. You can download a package for macOS by this instruction. Since the code is not signed and notarized, you might need to adjust security settings on your macOS.
With this package, you will see an option Enable Asynchronous Compensation for Audio Input Capture.

tt2468 · 2022-06-26T05:22:21Z

Other than for testing purposes, is there any reason to want to have async compensation be a configurable value in sources? To me it seems like it should always be on unless there is a good reason to want it off.

norihiro · 2022-06-26T10:06:29Z

Other than for testing purposes, is there any reason to want to have async compensation be a configurable value in sources? To me it seems like it should always be on unless there is a good reason to want it off.

It's a good point.

I want to provide the option to revert to the old behavior just in case it works wrong.
If async-SRC is enabled, a low-pass filter is conformed to cut off very high frequency. Someone might want to avoid the low-pass filter instead of avoiding glitches.
Some audio interfaces support genlock. If a user somehow do genlock to OBS, async-SRC is unnecessary.

Anyway, libobs need to have the switch to have async-SRC. Obviously, media source and VLC source have to disable the async-SRC.

Btw, so far I have one thing I need to work on. When I turn off async-SRC, I guess the audio resampler is still running even though it's unnecessary. Other behavior looks fine for me but I don't know the owner of this project is happy to have this PR. Edited: this is fixed.

tt2468 · 2022-07-25T12:16:56Z

Other than for testing purposes, is there any reason to want to have async compensation be a configurable value in sources? To me it seems like it should always be on unless there is a good reason to want it off.

It's a good point.

I want to provide the option to revert to the old behavior just in case it works wrong.

If async-SRC is enabled, a low-pass filter is conformed to cut off very high frequency. Someone might want to avoid the low-pass filter instead of avoiding glitches.

Some audio interfaces support genlock. If a user somehow do genlock to OBS, async-SRC is unnecessary.

Anyway, libobs need to have the switch to have async-SRC. Obviously, media source and VLC source have to disable the async-SRC.

Btw, so far I have one thing I need to work on. When I turn off async-SRC, I guess the audio resampler is still running even though it's unnecessary. Other behavior looks fine for me but I don't know the owner of this project is happy to have this PR. Edited: this is fixed.

Seems like a UI chat question. If every source should be able to enable/disable conversion, then perhaps it should not be up to the source itself and instead be configured through the advanced audio dialog. Or, if that's too complicated still for users, we just make more assumptions and handle the rest inside of obs-audio.

norihiro · 2022-07-26T01:40:54Z

The reason I set default to enable is based on my assumption that most users won't have genlock and not care audio. The option should be disabled by professional users who have genlock outside obs for example.

This figure shows how the compensation is working for long period, measured on Fedora 35 with MOTU M2, 48000 Hz. The compensation becomes steady state after around 300 seconds has passed.

Looking at the behavior after 300 seconds have passed, I think the compensation is stable enough. Cumulative probability of the compensation value is shown in the figure below. Its standard deviation is 0.097 S/s. 99% points are within 0.50 S/s range. The minimum is -4.86 S/s, the maximum is -4.19 S/s. Giving a 442 Hz tone, pitch variation will be between 441.9966 Hz and 442.0027 Hz. I don't think even well-trained musicians can notice.

This commit implements a property to enable asynchronous sample rate conversion (SRC) in the PR obsproject/obs-studio#6351. This commit relies on the PR so that the implementation is disabled by default. Assuming the user of this plugin has enough knowledge for asynchronous sample rate and genlock, the option is disabled by default.

zbynekdrlik · 2024-10-07T12:09:18Z

When we could test it?

norihiro · 2024-10-07T16:24:37Z

Rebased and the build flow has passed. You can test it from the summary page.

moonlightes · 2024-12-23T07:31:17Z

Hi. Is this change missing in the current version of obs 31? If so, is it expected to be added soon?

Fenrirthviti · 2024-12-23T07:44:20Z

This has not been merged yet, so it is not missing from any release.

We prefer not to provide estimations on PR reviews as the project priorities are constantly in flux.

pkviet

The code is very clean and well written.
I appreciated the explanations with reference to the research behind the compensation filter.
As discussed in another PR, FFmpeg has the aresample filter which has both hard and soft compensation. Can you summarize the differences and see if this can suggest improvements to your current implementation ?
Also could this filter help solve the desync issues with all audio outputs (chiefly, decklink and ndi)
(currently there is a low latency audio buffering option which sets oai->max_buffering_ms to 20 ms and triggers :
https://github.com/obsproject/obs-studio/blob/master/libobs/obs.c#L1612
and following)
Sidenote: please rebase the PR

libobs/media-io/audio-resampler-ffmpeg.c

libobs/media-io/lag_lead_filter.h

libobs/obs-source.c

pkviet · 2026-01-31T13:04:27Z

plugins/linux-alsa/alsa-input.c

 #endif
+
+	async_compensation = obs_data_get_bool(settings, "async_compensation");
+	obs_source_set_async_compensation(data->source, async_compensation);


in theory this is a no-op, if async_compensation value has not changed. So it's OK.
But in general, we store source settings as members of the base data struct, and update it only if its value has changed, so this saves a call. I will readily agree that here it does not make much difference though.
(same remark for other sources where you followed the same pattern)

pkviet · 2026-01-31T13:11:03Z

libobs/media-io/audio-resampler-ffmpeg.c

+		lag_lead_filter_tick(&rs->compensation_filter, rs->input_freq, in_frames);
+		double drift = lag_lead_filter_get_drift(&rs->compensation_filter);
+
+		ret = swr_set_compensation(rs->context, -(int)(drift * 65536 / rs->input_freq), 65536);


where does that figure come from ?

pkviet · 2026-01-31T13:19:56Z

libobs/media-io/audio-resampler-ffmpeg.c

+	if (!rs->compensation_filter_configured) {
+		// Parameters are determined experimentally by SPICE simulation
+		// to have these characteristics.
+		// - Unity gain frequency: 1/180 Hz (inverse of 3 minutes)


did you experiment with shorter and longer times ? say 1 min so 1/60 Hz and 5 min so 1/300 Hz ?
Regarding this curve:
https://user-images.githubusercontent.com/780600/180902267-e95694da-7aed-46c2-839b-36c5fe097f15.png

how is it modified ?
Is the sample/sec negative compensation due to input lag ? have you tested the reverse condition (through a simulation for instance)
More generally, can you explain why the filter converges to a plateau ? it seems to mean that the input source is intrinsically lagging if the filter must always compensate ?

An API `audio_resampler_compensate` is added that activates resampling compensation by calling `swr_set_compensation`.

norihiro · 2026-02-18T05:40:49Z

Rebased to resolve conflicts.

This commit implements asynchronous sample-rate conversion for audio source.

Since the audio from most sound devices are asynchronous, it requires to be compensated so that the audio source provides exactly necessary amount of samples. The option is enabled by default. Also update linux-alsa.

When `use_device_timing` is not set, the audio from the device is asynchronous. It requires to be compensated to avoid audio glitch. The option is masked if `use_device_timing` is set. The option is enabled by default for wasapi_input_capture.

Since the audio from most sound devices are asynchronous, it requires to be compensated so that the audio source provides exactly necessary amount of samples. The option is enabled by default.

Since the audio from the device is asynchronous, it requires to be compensated so that the audio source provides exactly necessary amount of samples. The option is enabled by default.

norihiro force-pushed the async-sample-rate-conversion branch from 54a2b75 to aa38840 Compare April 18, 2022 16:09

norihiro marked this pull request as draft April 18, 2022 16:10

cg2121 added Bug Fix Non-breaking change which fixes an issue Enhancement Improvement to existing functionality labels Apr 18, 2022

norihiro force-pushed the async-sample-rate-conversion branch 4 times, most recently from 9eeb15e to ec47a17 Compare April 20, 2022 15:01

norihiro mentioned this pull request Apr 20, 2022

aja: Subtract packet time from audio timestamp #6359

Merged

6 tasks

jp9000 added the Seeking Testers Build artifacts on CI label Apr 21, 2022

norihiro force-pushed the async-sample-rate-conversion branch from f5aec3a to c0392ec Compare April 21, 2022 02:08

norihiro force-pushed the async-sample-rate-conversion branch 2 times, most recently from 3821723 to 0a36bd0 Compare April 27, 2022 08:40

norihiro force-pushed the async-sample-rate-conversion branch from 0a36bd0 to 3d37efd Compare July 15, 2022 10:28

norihiro marked this pull request as ready for review July 16, 2022 15:54

norihiro mentioned this pull request Jul 25, 2022

Asynchronous sample rate converter pkviet/obs-studio#77

Merged

16 tasks

norihiro force-pushed the async-sample-rate-conversion branch from 3d37efd to 55d57fc Compare July 26, 2022 01:22

norihiro force-pushed the async-sample-rate-conversion branch from 55d57fc to e78a06f Compare July 26, 2022 03:20

norihiro force-pushed the async-sample-rate-conversion branch 3 times, most recently from c906af2 to 4caef71 Compare September 27, 2022 15:58

dojima mentioned this pull request Nov 27, 2022

Fix random audio dropouts and corruption of audio data (2/2) #5993

Open

6 tasks

WizardCM added this to the OBS Studio 29.1 milestone Dec 10, 2022

norihiro force-pushed the async-sample-rate-conversion branch from 4caef71 to 57b3fcc Compare December 20, 2022 22:54

norihiro force-pushed the async-sample-rate-conversion branch from 3e3bb30 to 2c9f064 Compare April 13, 2024 02:20

RytoEX modified the milestones: OBS Studio (Next Version), OBS Studio (Next Major Version) Jun 7, 2024

norihiro mentioned this pull request Aug 23, 2024

libobs: Add experimental API to retrieve next audio timestamp on source #11176

Open

6 tasks

norihiro force-pushed the async-sample-rate-conversion branch from 2c9f064 to c705ee8 Compare October 7, 2024 16:01

ipatix mentioned this pull request May 9, 2025

linux-pulseaudio: Implement audio output #10495

Open

6 tasks

mindscarp mentioned this pull request Jul 7, 2025

Random short audio dropouts when recording or streaming #4600

Open

ogmkp mentioned this pull request Jan 17, 2026

Mic stutter/cutting for over 3 years of recording videos on linux/pipewire #12835

Open

norihiro mentioned this pull request Jan 20, 2026

libobs/audio-monitoring: Add audio sync feature for audio-only playback #12974

Open

6 tasks

pkviet approved these changes Jan 31, 2026

View reviewed changes

norihiro marked this pull request as draft February 18, 2026 05:23

norihiro force-pushed the async-sample-rate-conversion branch from c705ee8 to 6ab3caa Compare February 18, 2026 05:25

libobs/media-io: Add compensation to audio-resampler

c7de322

An API `audio_resampler_compensate` is added that activates resampling compensation by calling `swr_set_compensation`.

norihiro force-pushed the async-sample-rate-conversion branch 2 times, most recently from 7be8ff5 to 13cf845 Compare February 18, 2026 05:43

norihiro added 7 commits February 18, 2026 14:49

libobs: Implement async compensation in source

c81368c

This commit implements asynchronous sample-rate conversion for audio source.

libobs/media-io: Report estimated frequency of sample rate compensation

a976a76

linux-pulseaudio: Add option to activate async compensation

98fa88d

Since the audio from most sound devices are asynchronous, it requires to be compensated so that the audio source provides exactly necessary amount of samples. The option is enabled by default. Also update linux-alsa.

mac-capture: Add option to activate async compensation

c88f145

Since the audio from most sound devices are asynchronous, it requires to be compensated so that the audio source provides exactly necessary amount of samples. The option is enabled by default.

aja: Add option to activate async compensation

1646bb7

Since the audio from the device is asynchronous, it requires to be compensated so that the audio source provides exactly necessary amount of samples. The option is enabled by default.

decklink: Add option to activate async compensation

c88796e

Since the audio from the device is asynchronous, it requires to be compensated so that the audio source provides exactly necessary amount of samples. The option is enabled by default.

norihiro force-pushed the async-sample-rate-conversion branch from 13cf845 to c88796e Compare February 18, 2026 05:49

Uh oh!

Comments

Conversation

norihiro commented Apr 18, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Motivation and Context

How Has This Been Tested?

Types of changes

Checklist:

Uh oh!

digitalchou commented Apr 20, 2022

Uh oh!

norihiro commented Apr 21, 2022

Uh oh!

tt2468 commented Jun 26, 2022

Uh oh!

norihiro commented Jun 26, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tt2468 commented Jul 25, 2022

Uh oh!

norihiro commented Jul 26, 2022

Uh oh!

zbynekdrlik commented Oct 7, 2024

Uh oh!

norihiro commented Oct 7, 2024

Uh oh!

moonlightes commented Dec 23, 2024

Uh oh!

Fenrirthviti commented Dec 23, 2024

Uh oh!

pkviet left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pkviet Jan 31, 2026

Choose a reason for hiding this comment

Uh oh!

pkviet Jan 31, 2026

Choose a reason for hiding this comment

Uh oh!

pkviet Jan 31, 2026

Choose a reason for hiding this comment

Uh oh!

norihiro commented Feb 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

13 participants

norihiro commented Apr 18, 2022 •

edited

Loading

norihiro commented Jun 26, 2022 •

edited

Loading

pkviet left a comment •

edited

Loading