Pipewire support

[matt335672]

I'm starting this discussion as it's inevitable that it will come up in issues, now that Fedora 34 has moved to native pipewire support.

[matt335672]

This is an early stage discussion. Some of the information I'm presenting below may not be entirely correct, and there may be other options which I haven't even though of yet. I'm making no apologies for this - I'd rather start the discussion going as soon as possible. Please feel free to jump in and move this discussion forward.

A summary of the situation as I see it is as follows.

Audio support in xrdp (v0.9.17 is the current version as I write this) is provided for Pulseaudio (PA) by a combination of the pulseaudio-xrdp-module and functionality in the xrdp-chansrv executable.

There are actually two pulseaudio xrdp modules, for incoming and outgoing data streams. For expeciency these are both referred to as 'the PA module' below.

When this works, it works reasonably well. However, over time the following disadvantages of this approach have become clearer:-

• The PA module uses internal PA interfaces.
  • Not completely stable over time - the PA module has to use conditional compilation to support multiple versions.
  • Building the PA module requires pulseaudio sources to be installed and configured. This can be a real challenge on some systems.
  • It's not practical for distros to wrap xrdp pulseaudio support.
  • Updating PA can result in a non-working or partially working PA module - this is hard to diagnose.
• Only PA can easily be supported with this technique.

To support other audio systems, we can either continue with the existing design (i.e. write other modules), or re-think how the audio is processed. The problem with writing yet more modules is the maintenance effort scales with the number of modules. PA isn't going away any time soon, and we do not yet know how successful Pipewire will be over time. If we wrote a Pipewire module we'd still have to support the existing PA module for many years yet.

A better solution might be to explore using sound protocols over the network (e.g. RDP/SDP/SAP) to provide access to xrdp sound devices. I haven't researched this in any detail yet - comments from people who know about such things would be extremely welcome!

[Kishi85]

Would it be possible to use a local loopback network connection that works for both Pipewire-Pulseaudio (https://gitlab.freedesktop.org/pipewire/pipewire/-/wikis/Config-PulseAudio#network-support) and Pulseaudio itself (https://www.freedesktop.org/wiki/Software/PulseAudio/Documentation/User/Network/, likely using https://roc-streaming.org/toolkit/docs/running/pulseaudio_modules.html as Pipewire-Pulse does not support RTP for Network sink/source yet)?

The new xrdp audio module would have to simply provide the interfaces (auth protocol/source/sink) expected and supported by both of the network modules of Pipewire-Pulseaudio and Pulseaudio itself (probaly the ROC variants) and XRDP would only have to set up those sources/sinks at session startup.

Advantage: Would support any Backend that supports the chosen network audio protocol and also remove the really complicated step of having to compile a version-dependent PA module using PA sources, which would be simpler and easier to package even if it would require an additional package providing ROC support for native PulseAudio.

Just a thought that came to mind when looking at the Pipewire-PulseAudio docs unfortunately I'm lacking the abilty to implement this myself, otherwise i'd probably just draft up a pull request. Hope it helps anyone planning to pick this up.

[Kishi85]

Would it be possible to use a local loopback network connection that works for both Pipewire-Pulseaudio (https://gitlab.freedesktop.org/pipewire/pipewire/-/wikis/Config-PulseAudio#network-support) and Pulseaudio itself (https://www.freedesktop.org/wiki/Software/PulseAudio/Documentation/User/Network/, likely using https://roc-streaming.org/toolkit/docs/running/pulseaudio_modules.html as Pipewire-Pulse does not support RTP for Network sink/source yet)?

The new xrdp audio module would have to simply provide the interfaces (auth protocol/source/sink) expected and supported by both of the network modules of Pipewire-Pulseaudio and Pulseaudio itself (probaly the ROC variants) and XRDP would only have to set up those sources/sinks at session startup.

Advantage: Would support any Backend that supports the chosen network audio protocol and also remove the really complicated step of having to compile a version-dependent PA module using PA sources, which would be simpler and easier to package even if it would require an additional package providing ROC support for native PulseAudio.

Just a thought that came to mind when looking at the Pipewire-PulseAudio docs unfortunately I'm lacking the abilty to implement this myself, otherwise i'd probably just draft up a pull request. Hope it helps anyone planning to pick this up.

I've been doing some more digging on this and it seems that Pipewire can simply talk to a remote Pulseaudio server via the Pipewire Pulse Tunnel Module (https://docs.pipewire.org/page_module_pulse_tunnel.html) that means that with a minimalistic implementation of a Pulse server within xrdp (listening on a unix socket) it should be possible to easily attach both pipewire and pulseaudio to xrdp without any compilation or other hacks (via loopback/unix socket).

Also that minimalistic implemenation might also be able to reuse code from Pipewire's PulseAudio Server Module (https://docs.pipewire.org/page_module_protocol_pulse.html / https://github.com/PipeWire/pipewire/tree/master/src/modules/module-protocol-pulse).

Still a bit much for my limited ability to implement it myself but maybe it gives someone more experienced a way to implement this.

[toneboy]

Hi,
I'm very interested in the pipewire project, but I'm using debian buster (a distro without systemd). I tried and failed to build from source.

I was wondering if the current pipewire will ever be backported to debian buster?

Thanks very much

[dsonck92]

One possible option I've seen applications use (noson) that need to provide an output and input is to spawn a null sink and receive or send audio through it. This should be compatible with both pulse and pipewire.

In practice, this would mean a client would start and register 2 null sinks. One sink is for audio playback, and the client listens on the monitor output for audio to transmit through RDP. The other sink will receive the audio from RDP and the user of the desktop can select that monitor to capture audio.

I do have to note that the ergonomics might not be great with this, especially for the RDP source, as I think monitor capture endpoints are usually filtered out by clients. Although a quick test with pipewire and teams shows that it does support taking the monitor source as mic input.

[dsonck92]

As an extension on this, it would possibly be interesting to also have a client implementing the same for JACK, but then directly expose the in/out to the JACK graph. Incidentally, this would be usable (but not ergonomic) with pipewire as well with its JACK protocol implementation. But this would help out those that are on JACK natively.

[matt335672]

From 22.10, Ubuntu no longer seems to support PA modules at all - neutrinolabs/pulseaudio-module-xrdp#89

[Hiero32]

I tried to support pipewire in two ways.

1. I wrote pipewire module. This approach is close to module-xrdp-sink.c/module-xrdp-source.c.

Attached module-xrdp-pipewire.tar.gz includes four files.

 module-xrdp-pipewire.c  : module which supports both xrdp-sink and xdrp-source.
 pw-cli_0358_mod.c       : This is modified version of pw-cli.c
                           This is not necessary for pipewire 0.3.59 and above, which version of pw-cli can be used.
 pipewire-xrdp.desktop   : desktop file to run load_pw_modules.sh. Placed in /etc/xdg/autostart
 load_pw_modules.sh      : shell script to start module-xrdp-pipewire. Placed in /usr/local/libexec/pipewire-module-xrdp

I picked "unix domain socket" related portion from module-xrdp-sink.c and module-xrdp-source.c here,
then merged them into module-pipe-tunnel.c in pipewire source tree.
And modified module-pipe-tunnel.c to support both xrdp-sink and xdrp-source in a single instance, and renamed it to module-xrdp-pipewire.c

built and installed on Ubuntu 22.10, Debian 12 and Pop_OS! 22.04.

Following steps are to compile/link and install them.

$ sudo apt install build-essential pkg-config libspa-0.2-dev libpipewire-0.3-dev pulseaudio-utils
$ cc -o libpipewire-module-xrdp-pipewire.so module-xrdp-pipewire.c -shared -fPIC $(pkg-config --cflags libpipewire-0.3)
$ sudo cp libpipewire-module-xrdp-pipewire.so $(pkg-config --variable=moduledir libpipewire-0.3)/.
$ sudo chmod 644 $(pkg-config --variable=moduledir libpipewire-0.3)/libpipewire-module-xrdp-pipewire.so
$ sudo mkdir -p /usr/local/libexec/pipewire-module-xrdp
$ sudo cp load_pw_modules.sh /usr/local/libexec/pipewire-module-xrdp/.
$ sudo chmod 755 /usr/local/libexec/pipewire-module-xrdp/load_pw_modules.sh
$ sudo cp pipewire-xrdp.desktop /etc/xdg/autostart/.
$ sudo chmod 644 /etc/xdg/autostart/pipewire-xrdp.desktop

If installed pipeware version is 0.3.58(Ubuntu 22.10 is the case at this moment), following three steps are necessary.

$ cc -o pw-cli_0358_mod pw-cli_0358_mod.c $(pkg-config --cflags --libs libpipewire-0.3)
$ sudo cp pw-cli_0358_mod /usr/local/libexec/pipewire-module-xrdp/.
$ sudo chmod 755 /usr/local/libexec/pipewire-module-xrdp/cli_0358_mod

Thats all.

Works on both Ubuntu 22.10, Denbian 12 and Pop_OS! 22.04.

This is modification of the file(module-pipe-tunnel.c) in pipewire tree.
Pipewire opens API(SPA (Simple Plugin API)) to make plugin/module.
There are development packeges libspa-0.2-dev and libpipewire-0.3-dev on Debian/Ubuntu and pipewire-devel on Fedora.
To compile/link module, only header files, which are included in above packeges, are nessesary.
It is not necessary to get full source code.

Fedora37:

$ sudo dnf group install "C Development Tools and Libraries" "Development Tools"
$ sudo dnf upgrade pipewire
$ sudo dnf install pipewire-devel

== Following steps are mostly same as Ubuntu

$ cc -o libpipewire-module-xrdp-pipewire.so module-xrdp-pipewire.c -shared -fPIC $(pkg-config --cflags libpipewire-0.3)
$ sudo cp libpipewire-module-xrdp-pipewire.so $(pkg-config --variable=moduledir libpipewire-0.3)/.
$ sudo chmod 755 $(pkg-config --variable=moduledir libpipewire-0.3)/libpipewire-module-xrdp-pipewire.so
$ sudo mkdir -p /usr/local/libexec/pipewire-module-xrdp
$ sudo cp load_pw_modules.sh /usr/local/libexec/pipewire-module-xrdp/.
$ sudo chmod 755 /usr/local/libexec/pipewire-module-xrdp/load_pw_modules.sh
$ sudo cp pipewire-xrdp.desktop /etc/xdg/autostart/.
$ sudo chmod 644 /etc/xdg/autostart/pipewire-xrdp.desktop

2. use GStreamer approach.

GStreamer supports pulseaudio with plugin named pulsesrc.
pulsesrc is also supported by pipewire.
Using gstreamer's appsink plugin can be used to write code to send stream to xrdp.
Attached file xrdp_appsink_test.c is a small source code, but works as xrdp-sink.
It necessary to just build as follows.

Ubuntu:
$ sudo apt install libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev
$ gcc -o xrdp_appsink_test xrdp_appsink_test.c $(pkg-config --libs --cflags gstreamer-app-1.0)

Fedora:
$ sudo dnf install gstreamer1-devel gstreamer1-plugins-base-devel
$ gcc -o xrdp_appsink_test xrdp_appsink_test.c $(pkg-config --libs --cflags gstreamer-app-1.0)

If module-xrdp-sink / unload-module module-xrdp-source are working, stop them as follows.
When using pulseaudio
$ pactl unload-module module-xrdp-sink
$ pactl unload-module module-xrdp-source
When using pipewire
$ ./load_pw_modules.sh -d

Then, just run xrdp_appsink_test.

$ ./xrdp_appsink_test

work on
Ubuntu 22.10 : libpipewire 0.3.58
Fedora 37 : libpipewire 0.3.63
Pop!_OS 22.04: libpipewire 0.3.63
Ubuntu 22.04 : pulseaudio 15.99.1
Ubuntu 20.04 : pulseaudio 13.99.1
Ubuntu 18.04 : pulseaudio 11.1

Disadvantage of this approach is:
This is an application, not module.
Once streaming to xrdp starts, it does not stop sending.
When audio application pause/stop/exit, silence audio is sent to xrdp client continuously.
It consumes network flow(44100 x 8 x 4 = 1,411,200bps). If xrdp is configured to support mp3lame, the rate is 128,000bps.
This approach does not work well for xrdp source(input. e.x. mic ), because there is no way to stop receiving stream when no record application is working.

3. RTP/SAP

Pulseaudio has a rtp send module: module-rtp-send.
Pipewire supprts rtp on version 0.3.60 and after: module-rtp-sink
It is easy to test RTP using these module.
destination of RTP below is IP address/port (192.168.10.2:43210 in this example) of client.

on xrdp server:

PULSEAUDIO:

pactl load-module module-null-sink sink_name=rtp format=s16le channels=2 rate=44100
pactl load-module module-rtp-send source=rtp.monitor destination=192.168.10.2 port=43210 mtu=320 channels=2 rate=44100 inhibit_auto_suspend=only_with_non_monitor_sources

PIPEWIRE: version 0.3.60 and after

pw-cli -m load-module libpipewire-module-rtp-sink sap.ip=192.168.10.2 destination.ip=192.168.10.2 audio.rate=44100 stream.props={node.name="xrdp-rtp-sink" media.class="Audio/Sink"}
pactl set-default-sink xrdp-rtp-sink

on xrdp client:

Using VLC: see this url:
https://docs.videolan.me/vlc-user/3.0/en/advanced/streaming/sap_session.html#connecting-to-a-sap-stream

Using ffplay:
ffplay sap://

Using gstreamer:
$ gst-launch-1.0 udpsrc port=43210 caps="application/x-rtp,media=audio,clock-rate=44100,encoding-name=L16,channels=2" ! rtpL16depay ! queue ! audioconvert ! audioresample ! autoaudiosink
gstremaer does not support SAP.

When worked for video applicatios, lip sync does not work well.
This is a bit unstable dependent on network conditions. Playback speed and pitch drifts.
To use RTP, it is necessary for users to install RTP client and start it. I think it is not convenient for most users.
I don't think RTP is realistic.

As a result, I think it is the best way to write a pipewire module to support full function.

module-xrdp-pipewire.tar.gz
xrdp_appsink_test.tar.gz

[matt335672]

Great work, @Hiero32! In particular I appreciate your analysis of the approaches available.

I'll have a look at this in more detail when I've got more time - at the moment I'm trying to get sesman is a reasonable state for the next major xrdp release, and that's got to be a priority.

In the meantime, if anyone else cares to look at this, please add your experiences here.

[wtay]

Would you like to propose the xrdp module in PipeWire upstream?

[jsorg71]

Hi Wim, you seem to be involved with pipwire, what you you suggest? One big problem with pulseaudio is that they had a nice modal system but no devel package to build them out of tree. Does pipewire allow out of tree modules?
At first I think the modules will change a lot but maybe when stable they can be upstream.
In the future I can see using pipwire for wayland display streams.

[wtay]

Yes, you can develop modules out of tree just fine with pipewire.

[Hiero32]

@wtay,
>> Would you like to propose the xrdp module in PipeWire upstream?
Yes.
This module is the combination of xrdp and PipeWire .
I do not have an idea which is better this (or same type) module is handled by xrdp or PipeWire .
Any way, if PipeWire handles it, it is very good.
By the way, does the next major xrdp release keep the same API for sound i/o to/from this type of module?

[matt335672]

There are no plans to change the interface between chansrv and the sound module at the moment. I think it's safe to assume it won't be changing for a bit.

[elboulangero]

@Hiero32 In the end, do you have any plan to propose this module for inclusion Pipewire? Or is there any benefit that it remains out-of-tree, in a separate repo, as it is now at https://github.com/neutrinolabs/pipewire-module-xrdp? Thanks in advance for your reply.

[matt335672]

@Hiero32 - I've started looking at making this a distributable module.

When I've got something worth sharing I'll let you know.

[matt335672]

All,

Here's an initial repo with @Hiero32's module in it:-

https://github.com/matt335672/pipewire-module-xrdp

I've built the module on Ubuntu 22.10 and Fedora 38. I've done no other testing yet, but I'm out of time today.

Comments/Issues/PRs, etc welcome.

[matt335672]

Thanks @madmalkav

I've recorded your comment as an issue here:-

matt335672/pipewire-module-xrdp#3

[Hiero32]

I have found out the cause of this issue, and sent a pull request.

[matt335672]

All merged - works great. Thanks both.

[elboulangero]

Just passing by to say thanks! In Kali Linux we now provide a Hyper-V image pre-configured for Enhanced Session Mode (ie. xRDP connection). So far we relied on the Pulseaudio xrdp module for sound to work, however we just switched to Pipewire, so no more sound...

@matt335672 I just found your repo, packaged it for Kali already: https://repology.org/project/pipewire-module-xrdp/versions. We have users who reported that it works (at https://bugs.kali.org/view.php?id=8342)

So big thanks to you and @Hiero32 (and others involved) !!

[matt335672]

That's great news @elboulangero - thanks for the info.

@Hiero32 has done all the hard work here - I've just spent a bit of time adding a configure script.

[matt335672]

@metalefty - given the feedback above, I think we should make this repo sit under https://github.com/neutrinolabs rather than my account, and maybe make a release. What do you think?

[JoeSalmeri]

@matt335672

I've been following this for a while. Happy to hear that sound is coming back to xrdp with pipewire.

Any idea what future version of xrdp will have this pipewire support or when it will be coming to distros like openSUSE Tumbleweed ?

Thanks!

[matt335672]

Hi @JoeSalmeri

Sorry - I 've got no news on that.

@metalefty - are you in a position to copy https://github.com/matt335672/pipewire-module-xrdp to https://github.com/neutrinolabs/pipewire-module-xrdp ? I don't have permissions to do that.

Once we've got an official release, it will be up to the distros how they package the driver up and distribute it.

[metalefty]

Could you give me admin permission on your repository? Then I could transfer your repo to neutrinolabs I think

[JoeSalmeri]

@matt335672

I inquired about this on the Tumbleweed mailing list to see if/when they could include the new module.

One of the replies I received said that the pipewire-pulseaudio module is supposed to allow applications written for pulseaudio to work using pipewire.

I suspect that module is for apps that are running locally and not an app like xrdp which is doing sound redirection, but I could be wrong.

I went ahead and tried the recommendation to see if it would actually get sound working but I still ran into the same "No output or input devices found" for the xrdp session.

Here's a link to that discussion....

https://lists.opensuse.org/archives/list/factory@lists.opensuse.org/thread/6IF6Z67FGZN5267YRI42WYRUNUAPUIH4/

I also asked the person that replied there to join this discussion.

Thanks!

Joe

[matt335672]

I've posted to the discussion. We'll see where it goes.

[JoeSalmeri]

@matt335672

Thank you!

I very much appreciate it!

Have a blessed day!

[JoeSalmeri]

@matt335672

FYI, someone ( Jason Craig ) on the TW Factory Mail list ( mentioned above ) compiled the new pipewire module and created an RPM package and I installed it on one of my test Tumbleweed machines and tested it out and it is working PERFECTLY!

Will this new pipewire module become part of the xrdp package such that when TW updates to a new version of xrdp it will be included or is it going to remain a separate package ?

I am asking because I created this bug report for TW to get the new module added

https://bugzilla.suse.com/show_bug.cgi?id=1214295

but there seems to be some confusion about how to get it added...

[matt335672]

I've just added you as a collaborator. I think you'll get an email. Then I should be able to make you an admin.

[metalefty]

Hmm, I need to be owner to transfer repos. However, repos under individual accounts can have only collaborators and organizations can have more than 2 owners if I understand correctly.

I'll create neutrinolabs/pipewire-module-xrdp with your write privilege. After that, can you push your repo also to the neutrinolabs repo instead of transferring repo? Is it fine to you?

[matt335672]

Yes - that's fine. I've come to much the same conclusion.

Thanks.

[metalefty]

Created.

https://github.com/neutrinolabs/pipewire-module-xrdp

[matt335672]

Contents now pushed.

[matt335672]

The PR history isn't there, but I don't think there's an easy way to get that over. With only 19 commits, I don't think it's that critical!

[JoeSalmeri]

Will the new pipewire module be part of the base XRDP package or will it be a separate package ?

Thanks!

[matt335672]

It's up to the packagers. We supply the module separately, but that doesn't mean they can't ship together. We have no desire to control or restrict what the packagers do in this area - its their distro.

I believe Debian used to ship the pulseaudio package as part of their xrdp deb but that' was removed for v0.9.8. See https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=794692. The same considerations do not apply to the pipewire module.