Add audio plugin hosting via Carla

[falkTX]

Description

Adds support for audio plugins to OBS (focus is on VST3 for now, as per the bounty in #5074).
As requested in the bounty, all plugins are loaded in a separate process with communication done via shared memory (with semaphores for synchronization).

The plugin support is done mostly through Carla, but everything is directly integrated into OBS.
While the focus is on VST3, the same code can be used for loading LADSPA, LV2, VST2, CLAP and JSFX plugins.

Motivation and Context

Having access to audio plugins within OBS simplifies a lot of complex setups that before hand had to rely on external tools, or keep only using VST2 plugins through the "old" obs-vst plugin.

See #5074 for discussion around the VST3 feature request.

How Has This Been Tested?

I have tested mainly on Linux during development, with occasional switching to Windows 10 and macOS to check on their status.

Do note the plugin bridging as used by Carla is already in use in other applications like zrythm, so aside for new bugs due to new OBS-related and VST3 hosting implementation I expect most things to just work.

On Linux I regularly checked through valgrind to catch for runtime memory issues and leaks.
(I did not check debug the remote plugin process much though, but have tested the same hosting code running non-bridged)

Types of changes

This PR adds a new "input source" and "input filter", for now called "Audio Plugin" but I am welcome to a more OBS-branding-like name.

These 2 come from the same binary and code, the "input source" creates its own thread for the audio while "input filter" uses the typical filter_audio method. They also both work the same way, with the same exact options.

Note that in regards to the bounty only the filter is necessary, but I find allowing generator plugins as sound source to be extremely useful.

Plugin options

By default there will be 4 widgets:

• Buffer size (direct buffer from OBS is default, with 128, 256 and 512 fixed buffer + latency extra options)
• Select plugin... (to open the plugin list dialog)
• Load file... (opens a file selection dialog, for hand-picking a VST2/3 dll or CLAP binary)
• Reload (can be used in case of plugin crash or timeout, reloads the plugin as necessary)

If a plugin is loaded that provides a custom UI, a "Show custom UI" button will be shown too.

After these buttons OBS will show plugin parameters, if any are exposed by it.
Changes in parameters from OBS side will be transmitted to the plugin and vice-versa.

Screenshot for reference, using master_me:

Checklist:

• My code has been run through clang-format.
• I have read the contributing document.
• My code is not on the master branch.
• The code has been tested.
• All commit messages are properly formatted and commits squashed where appropriate.
• I have included updates to all appropriate documentation.

Assuming the PR gets merged at some point, where is the official place for documentation that needs to be updated?

Current status

This pull request is pretty much complete. With a few things worth noting:

• Use of this feature requires updated obs-deps release (forced in this PR for now, for easier testing)
• Automated/nightly macOS builds (once obs-deps is resolved) use ad-hoc signing, which prevents OBS from loading any plugin from the system
• List of plugin folders is not configurable, it will just use the common folders from official plugin specs
• The VST3 hosting is somewhat new in Carla, it is possible some plugins don't work properly (let me know if you find any please)
• It is still unclear to me how to handle unequal IO between OBS and plugins. For example, if a 4 input goes into a stereo plugin (that can only do stereo, not more) what are we supposed to do, mixdown to stereo, pass only first 2 channels, something else...?

Feedback welcome, also regarding the UI/UX. I am open to tweaking names, labels etc to fit things better in OBS.

[falkTX]

on macOS the artifacts from OBS org actions have the same issue I had locally, making it impossible to load and thus test any plugins.

example output:

[carla] Scanning /Library/Audio/Plug-Ins/VST3/Surge XT Effects.vst3...
2023-05-17 05:02:45.274 carla-discovery-native[34869:8856080] Error loading /Library/Audio/Plug-Ins/VST3/Surge XT Effects.vst3/Contents/MacOS/Surge XT Effects:  dlopen(/Library/Audio/Plug-Ins/VST3/Surge XT Effects.vst3/Contents/MacOS/Surge XT Effects, 0x0106): tried: '/Library/Audio/Plug-Ins/VST3/Surge XT Effects.vst3/Contents/MacOS/Surge XT Effects' (code signature in <D3871979-4FB9-349D-B0E5-20443BB2A516> '/Library/Audio/Plug-Ins/VST3/Surge XT Effects.vst3/Contents/MacOS/Surge XT Effects' not valid for use in process: mapping process has no Team ID and is not a platform binary)
[carla] discovery: Failed to load VST3 bundle executable

guidance for this is appreciated.

[falkTX]

And just found a way.
After installing OBS.app, run:

codesign --remove-signature /Applications/OBS.app/Contents/MacOS/*

and re-run OBS. now it can list and load plugins under macOS again.

[falkTX]

for the windows build, at least under wine, it fails to load/find the carla-bridge plugin and there are no debug messages printed about it. dont know why yet, will try with a real windows system later in the week.

[gxalpha]

Assuming the PR gets merged at some point, where is the official place for documentation that needs to be updated?

The checkbox is mostly meant for developer documentation if libobs/frontend APIs are changed, so I don’t think it really applies. That said, we’d probably want to have some user documentation in the knowledge base come release.

[tytan652]

• List of VST3 folders is not configurable for now, it will just use the common folders from VST3 spec

From a quick test, the Flatpak does not show VST_PATH plugins (I have a LinuxAudio extension that provide VSTs (2 and 3)), the plugin must rely on such env vars since those are mostly the only way on Flatpaks.

[tytan652]

the env var to use here is VST3_PATH, as VST_PATH typically referes to VST2 plugins (we should not mix the 2). not that steinberg cares though, steinbergmedia/vst3sdk#67 is still open after all.

but try with VST3_PATH and let me know if that works for you

The manifest then needs to be updated to support VST 3, it requires me a pull the branch and modify the manifest to test then.

[falkTX]

Just checked on Windows, there is a crash when loading the carla-bridge plugin, but not really sure what it means just yet.
Used msys64 gdb to get a backtrace.

Thread 1 received signal SIGTRAP, Trace/breakpoint trap.
0x00007ffb5a94ecb3 in ntdll!RtlIsZeroMemory () from C:\Windows\SYSTEM32\ntdll.dll
(gdb) bt
#0  0x00007ffb5a94ecb3 in ntdll!RtlIsZeroMemory () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ffb5a957a72 in ntdll!RtlpNtSetValueKey () from C:\Windows\SYSTEM32\ntdll.dll
#2  0x00007ffb5a957d5a in ntdll!RtlpNtSetValueKey () from C:\Windows\SYSTEM32\ntdll.dll
#3  0x00007ffb5a95d991 in ntdll!RtlpNtSetValueKey () from C:\Windows\SYSTEM32\ntdll.dll
#4  0x00007ffb5a869640 in ntdll!RtlAllocateHeap () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x00007ffb5a865d21 in ntdll!RtlFreeHeap () from C:\Windows\SYSTEM32\ntdll.dll
#6  0x00007ffb5810e97b in ucrtbase!_free_base () from C:\Windows\System32\ucrtbase.dll
#7  0x00007ffac39cb77b in obs_module_ver () from C:\msys64\home\user\obs-studio-windows-x64-f2c0abf4c\obs-plugins\64bit\carla-bridge.dll
#8  0x00007ffac39c1b29 in obs_module_load () from C:\msys64\home\user\obs-studio-windows-x64-f2c0abf4c\obs-plugins\64bit\carla-bridge.dll
#9  0x00007ffac416d3c1 in obs_init_module () from C:\msys64\home\user\obs-studio-windows-x64-f2c0abf4c\bin\64bit\obs.dll
#10 0x00007ffac416c96b in obs_key_to_name () from C:\msys64\home\user\obs-studio-windows-x64-f2c0abf4c\bin\64bit\obs.dll
#11 0x00007ffac416d1e6 in obs_find_modules2 () from C:\msys64\home\user\obs-studio-windows-x64-f2c0abf4c\bin\64bit\obs.dll
#12 0x00007ffac416d489 in obs_load_all_modules2 () from C:\msys64\home\user\obs-studio-windows-x64-f2c0abf4c\bin\64bit\obs.dll

The obs_module_ver call is something put in place by OBS_DECLARE_MODULE macro, so it seems weird to result in a crash...

EDIT: this has been fixed now, there was a buffer overflow

[PatTheMav]

Are these macros required? Only use macros if there is no other language feature to achieve the same (e.g. const character array for constant strings).

[falkTX]

The macros allow to reuse the same code for other OBS plugins, which I already have written for testing but are not part of this PR. The idea is to build a few other variants that reuse the same base, but are separate OBS plugins.
One of them is "carla patchbay" which brings the full GUI along as seen in https://user-images.githubusercontent.com/1334853/235544415-6ae2c9e5-3cbf-47b2-8fc0-013f0ec8276d.png

I also dont know if calling it "Carla" is okay or not, you might prefer something more like "OBS audio plugin".

[PatTheMav]

The macros allow to reuse the same code for other OBS plugins, which I already have written for testing but are not part of this PR. The idea is to build a few other variants that reuse the same base, but are separate OBS plugins. One of them is "carla patchbay" which brings the full GUI along as seen in https://user-images.githubusercontent.com/1334853/235544415-6ae2c9e5-3cbf-47b2-8fc0-013f0ec8276d.png

Understandable, but this code is targeted to become part of the obs-studio codebase and should be designed for that. If this were a 3rd party "audio plugin" template, then this would make sense, but for a core plugin this configurability is not needed and should be removed.

I also dont know if calling it "Carla" is okay or not, you might prefer something more like "OBS audio plugin".

I don't mind keeping the name of the technology in place, maybe others have an opinion on this.

[falkTX]

note that I already pushed the 2 OBS modules in this PR, so what I said above regarding carla-bridge and carla-patchbay reusing the same code already applies.
the same code is built twice, but uses a compiler macro to tweak it just enough to be different. the alternative is to have duplicate files where they only differ for 2-3 simple strings. is that better than the current approach?

[PatTheMav]

I have to admit that the entire architecture of this plugin goes above my head right now because there's multiple binaries, libraries, modules - it goes beyond the complexity of any other plugin we have right now.

Do you have an overview of all these moving parts somewhere and what the differences between them are? Because I don't currently understand why we need 2 plugins with the same code and what their differences would be.

[falkTX]

sorry for the confusion, this is just reusing some files for a common api adjusting obs <-> carla, then implementing that for both carla-bridge and carla-patchbay plugins.
the way to deal with getting audio from OBS, optionally putting it in a buffer, feeding that buffer into an audio plugin (lv2/vst3/etc) and then getting that audio back to feed it back into OBS... all this is something that both carla-bridge and carla-patchbay have to do, so it makes sense to share the base details or their implementation.

we basically have here:

• carla-bridge
• carla-patchbay

the carla-bridge is the main target of this PR and the old bounty, it allows OBS to load VST3 plugins but because carla handles more than that we get CLAP, LADSPA, JSFX and VST2 all in one. while other OBS things exist that deal with LV2 or VST2 plugins, they do not always work the best and are fairly limited.
my approach is to consolidate it all into a single nice package.
this OBS plugin has its own frontend code in order to deal with a proper plugin dialog with a searchbar, filters, favorites etc.
carla-utils is used for handling the plugin scanning (which internally calls the discovery tool as needed, audio plugins are skipped if previously scanned and in cache) and also to deal with some of the IPC details.
we still need to do some of the heavy lifting, sending and receiving IPC messages, but the actual complexity is mostly handled on carla side, including the audio plugin processing.

the carla-bridge handling is quite simple in its handling, in terms of what it can do. the plugin scanning and dialog can be a bit complex, but all processing is just dealing with IPC, then a bit of code for exposing parameters and saving plugin state.

but if you mostly meant to ask about code, well most of it is shared and the GUI parts are a bit "extra" as they are unique.
I would divide the code into 4 sections.

1. common code

• common.c/h - common C code between the plugins
• qtutils.h/cpp - small qt utilities not exposed through OBS APIs, plus any other needed Qt/C++ related bits
• carla-wrapper.h - provides a sort of "base" API,
• carla.c - bridges the carla-wrapper.h API into OBS, alike a Carla <-> OBS translation. these last 2 are the important files on the common code

2. carla-bridge only code

• carla-bridge.hpp/cpp - this implements the carla IPC bridge, in a way suitable for OBS. it is almost a generic implementation though, could be thought as a kind of "private API"
• carla-bridge-wrapper.cpp - this implements the "base" specified on point 1, calling into bridge details

3. carla-patchbay only code

• carla-patchbay-wrapper.c - similar to carla-bridge-wrapper.cpp above, but implements the public carla plugin API instead of handling bridge details. this code does nothing special by itself, it simply calls an API exposed by carla

4. carla-bridge GUI code

• pluginlistdialog.cpp/hpp/ui
• pluginrefreshdialog.hpp/ui

this is just the UI code to deal with the plugin dialog, no audio processing happens here, and it is all based on Qt code

[PatTheMav]

Thank you for the write-up, will read this carefully later tonight or tomorrow.

[falkTX]

I think I found the issue on Windows, an out of bounds write, fixed in 691e6df
Have not tested yet, but confirmed the issue with a local build of the same code.

And hopefully next build is fully working for flatpak, I imported the same code that other parts of OBS use to check if it is running under flatpak.

[falkTX]

good news, can confirm plugins work under flatpak!

but with a big caveat - it now needs some setup in order to see system installed plugins.
the page at https://github.com/flathub/org.freedesktop.LinuxAudio.BaseExtension seems to have info about this, but as I have never really used flatpak (besides doing these quick checks) I dont feel like I am the best person to handle that part.

can we ping the person(s) responsible for the OBS flatpak stuff please?

[falkTX]

ok the flatpak stuff was rather easy, just adding a few flags on the main json config.
then also installing some plugins via flatpak, all seems to work here.

[tytan652]

something is off with the rpath, trying the flatpak I can see this error:

error: os_dlopen(/app//lib/obs-plugins/carla-bridge.so->/app//lib/obs-plugins/carla-bridge.so): libcarla_utils.so: cannot open shared object file: No such file or directory

looking at the build logs, this part seems very suspicious:

Not at all, it would be the same outside of a Flatpak, just /usr rather than /app.

-- Installing: /app/lib/obs-plugins/carla-bridge.so
-- Set runtime path of "/app/lib/obs-plugins/carla-bridge.so" to "/app/lib"

I extracted the package contents using ostree, can verify that /app/lib/carla exists and the needed lib is there. Also was good to confirm that the needed binaries are present, in /app/bin. But this custom setup from flatpak side breaks carla, which assumes that libcarla_utils is in the same directory as the bridge and discovery binaries.. not to mention forcing RPATH after build..

The RPATH is "forced" Flatpak or not, it is how the CMake works. Flatpak do nothing custom, it just builds inside a runtime that is not the host.

[pkviet]

The plugin is now loaded in windows with current state of the PR.
I had several issues though:

• i had to specify manually CarlaUtils_INCLUDE_DIR, CarlaUtils_LIBRARY, CarlaUtils_DISCOVERY_NATIVE, CarlaUtils_BRIDGE_NATIVE; I had assumed that the finder would have looked in the path provided by CMAKE_PREFIX_PATH (where my deps folder is located);
• i had an insta crash on startup, which was because the carla-bridge-native.exe and carla-discovery-native.exe had not been copied by cmake in the rundir folder. This needs to be fixed.
• once this was solved, the plugin did appear BUT I was unable to load any VST3 nor VST2.
  I got an error message saying that the bridge could not be found.
  
• I noticed that the vst2/vst3 enumeration missed a lot of my VSTs on first load of the plugin; but on second load, the list was complete.

[falkTX]

Nice, thanks for testing.
Indeed we still need to readjust after the latest OBS cmake related changes, the carla stuff is not always found properly.

I think I know why the windows bridge is not found, can push that soon but dont have a windows machine to verify such changes at the moment.

can someone from OBS please tag a release on https://github.com/obsproject/obs-deps so we can begin to have this PR officially working?
should I open a ticket on obs-deps side about this?

[pkviet]

The project managers are in vacation. There'll be a tag when they're back in a few days.
People can use obs-deps artefacts already.
Alternately there's a json in root folder you could also change to point to your obs-deps fork. So you can tag yourself. That'd be a temporary commit to be removed later.
Check for instance my stats graphs pr.

[falkTX]

good to know, thanks.
so yeah I think I will indeed go with the option of pointing to a obs-deps fork, so we can begin to do some testing on windows and macOS. as an extra this allows us to update the carla dep as needed too, in case something is buggy, need to see if I can reproduce the crash when the tools are missing.

[micsthepick]

Hi falkTX, great work here, I recently moved my daily driver to Pop! OS, and now I'm testing out your Carla host! Glad to see that JSFX support is here!

I've encountered some issues: It seems that JSFX plugins are still loaded from ~/.config/REAPER/Efffects, in the "Audio Plugin Generator/Source" despite setting the JSFX search path elsewhere in the Patchbay or Rack.

[falkTX]

I've encountered some issues: It seems that JSFX plugins are still loaded from ~/.config/REAPER/Efffects, in the "Audio Plugin Generator/Source" despite setting the JSFX search path elsewhere in the Patchbay or Rack.

I dont quite follow this.. there is no setting for specifying the path of the plugins, it will use the defaults as defined by the plugin spec. for JSFX there is no spec so it uses the reaper's home folder.

If you set that path to something else in the Carla tool, that won't apply here as it is not Carla, it is just a plugin list dialog that was originally based on it.
Settings for OBS and Carla are completely independent.

[falkTX]

Did a refresh of the PR against latest master branch, aligning with the requested changes so far too.

I am not able to get the bridge tools/binaries to end up being part of the windows package though. I see stuff in there like obs-amf-test.exe but no carla-discovery-native.exe. So there must be some specific way to add an imported tool that already exists on the filesystem as a runtime dependency of the plugin.
The important part is that the carla tools are external things, not being built here on this repo but on obs-deps. CMake seems to have troubles marking such external tools as a dependency, they just gets ignored...

[falkTX]

A similar thing happens in macOS, the app bundle is missing carla-utils framework.

error: os_dlopen(/Applications/OBS.app/Contents/PlugIns/carla-bridge.plugin/Contents/MacOS/carla-bridge->/Applications/OBS.app/Contents/PlugIns/carla-bridge.plugin/Contents/MacOS/carla-bridge): dlopen(/Applications/OBS.app/Contents/PlugIns/carla-bridge.plugin/Contents/MacOS/carla-bridge, 0x0105): Library not loaded: '@rpath/carla-utils.framework/Versions/A/carla-utils'
  Referenced from: '/Applications/OBS.app/Contents/PlugIns/carla-bridge.plugin/Contents/MacOS/carla-bridge'
  Reason: tried: '/Applications/OBS.app/Contents/MacOS/../Frameworks/carla-utils.framework/Versions/A/carla-utils' (no such file), '/Applications/OBS.app/Contents/MacOS/../Frameworks/carla-utils.framework/Versions/A/carla-utils' (no such file), '/Applications/OBS.app/Contents/MacOS/../Frameworks/carla-utils.framework/Versions/A/carla-utils' (no such file), '/Applications/OBS.app/Contents/MacOS/../Frameworks/carla-utils.framework/Versions/A/carla-utils' (no such file), '/System/Library/Frameworks/carla-utils.framework/Versions/A/carla-utils' (no such file)

warning: Module '/Applications/OBS.app/Contents/PlugIns/carla-bridge.plugin/Contents/MacOS/carla-bridge' not loaded

Dont know how to proceed here, any help or guidance appreciated.

The good news is that manually copying carla-utils.framework from the referenced obs-deps works as intended.
So it is just a matter of solving this missing dep copy.

[micsthepick]

I've encountered some issues: It seems that JSFX plugins are still loaded from ~/.config/REAPER/Efffects, in the "Audio Plugin Generator/Source" despite setting the JSFX search path elsewhere in the Patchbay or Rack.

I dont quite follow this.. there is no setting for specifying the path of the plugins, it will use the defaults as defined by the plugin spec. for JSFX there is no spec so it uses the reaper's home folder.

If you set that path to something else in the Carla tool, that won't apply here as it is not Carla, it is just a plugin list dialog that was originally based on it. Settings for OBS and Carla are completely independent.

no worries, it was a bit confusing, but all my issues with JSFX have been resolved - I can place the plugins in that folder, or use the Patchbay plugin to change OBS Carla's paths. Am hoping that the latest commits will remove some of the carla related issues I've encountered (instability).

[micsthepick]

[Fenrirthviti]

After fairly lengthy and difficult internal discussions, we have decided that using Carla as a host for VST3 (and other) plugins is not the right path forward for the project.

In the interests of maintaining full transparency, we have spoken with falkTX and are paying out partial bounty on this for the work and effort, and want to publicly thank them for their patience on this. We will be posting an update to the bounty itself soon, and will reopen for new submissions at that time.

[falkTX]

Thanks for reaching out and getting a sorta conclusion to this PR.
It is of course not what I was hoping for when I started it, but can understand the situation.

I do not want to have the effort go to waste though, so at some time later this year I will revive this PR as its own external OBS plugin. That way it still lives on, and also allows for direct inclusion of carla as git submodule which was rejected early on (much easier to deal with regarding packaging).
If someone wants to collaborate on this carla-obs external plugin thing, I am open to it too. Specially since the middle wrapping code between OBS and Carla is pretty much done, what is needed now is setting up an external plugin project and add the bits from this PR. Worst case I will do it myself sometime during spring.

Thanks again and best wishes to the OBS project team.
And happy new year!