sound improvements: refactorings, cleanups,

[totaam]

Issue migrated from trac ticket # 849

component: sound | priority: critical | resolution: fixed

2015-04-30 15:39:32: antoine created the issue

---

Follow up from #400, #669 - related to #835.

• would be nice to share more code between the client and server

• handle the sound sequence mechanism for "microphone" just like we do for "speaker"
• use a cython or ctypes wrapper for accessing the pulse properties rather execing a subprocess
• avoid the osx pool autorelease warnings?
• setup process tagging in subprocess only?
• allow us to use python3 / gstreamer 1.0 in the subprocess from a python2 caller (requires execing a process to get the list of codecs supported..)

[totaam]

2015-04-30 15:41:02: antoine changed status from new to assigned

[totaam]

2015-04-30 15:41:02: antoine edited the issue description

[totaam]

2015-04-30 15:41:02: antoine changed title from sound improvements: to sound improvements: refactorings, cleanups,

[totaam]

2015-05-12 11:00:07: antoine commented

---

Some fixes (backported already, see #669#comment:43) and some changes worth recording (some of which we may want to backport): r9328.

[totaam]

2015-05-18 06:39:26: antoine commented

---

r9407 allows us to try to use gstreamer 1.x with python2 for probing and debugging (not really usable yet).

In order to completely move to a subprocess for all the gstreamer bits, we need to ensure that (almost) all calls to [/browser/xpra/trunk/src/xpra/sound/gstreamer_util.py] go through the subprocess wrapper.
Either by adding new commands to the wrapper for probing (ie: getting the list of plugins) or by ensuring the data is exported back to the client / server via info packets from the subprocess.

Grepping the source:

• bug report tool: can continue to use direct calls, should the info found in the client (if present and if we have any)
• session info: only needs version information, we can include that in info (small cost)
• ui client base: uses gstreamer util for populating caps and for validating command line options. Both need to use the wrapper, it will be a tad slower, but this is a one-off.
• server base and server source: same as client, one-off.
• the "main" script queries the plugins when the "help" option is used, no problem in using a subprocess to get that

Packaging updates:

• switch to gstreamer 1.x for Fedora and Stretch (and maybe Jessie?)
• win32: no idea about the availability of the gi bindings, will need to figure that out
• osx: we already use a subprocess wrapper script, should be possible to tweak the environment to make it load gstreamer 1.x

[totaam]

2015-05-25 14:23:19: totaam commented

---

• some fixes were needed to avoid some new warnings with gstreamer 1.x: r9501
• r9519 tries to continue on overrun, we try to re-fill the pipeline to a usable level - will need thorough testing on OSX as the restart code was written for handling overruns on OSX..
  If this works out, we can enhance the logic to take the average value and range of the queue values to try to minimize its size (helps with synchronize sound with video frames #835).

[totaam]

2015-05-27 14:48:02: antoine commented

---

• cleanups in r9527
• r9528 changes the default appsink settings (and maybe we should backport the drop=true property?)
• r9529 makes it easier to tweak the appsink settings, ie (showing the defaults):

XPRA_SOURCE_APPSINK="appsink name=sink emit-signals=true max-buffers=10 drop=true sync=true async=true qos=true" \
    xpra start :10

[totaam]

2015-05-27 16:03:45: antoine uploaded file sound-autotune-queue.patch (7.7 KiB)

experimenting with auto-tuning the queue to minimize overruns and underruns whilst keeping the overall delay low

[totaam]

2015-05-28 07:08:40: antoine uploaded file palib.patch (13.0 KiB)

support for using palib instead of execing pactl and parsing the output

[totaam]

2015-05-28 07:09:19: antoine uploaded file palib.tar.xz (13.1 KiB)

fixed palib: add setup.py for installation, ensure we can call it multiple times without causing problems with signals

[totaam]

2015-05-28 07:11:23: antoine commented

---

See the work in progress patches above based on the code found in pypacl: we want the palib parts which are nice, but not the pypactl bits which are not finished and not usable. This would require packaging it, which shouldn't be too hard (GPLv3+), a setup.py has already been added.

[totaam]

2015-05-28 10:50:51: antoine uploaded file python-palib.spec (1.6 KiB)

specfile

[totaam]

2015-05-28 10:51:34: antoine uploaded file palib-1.0.tar.xz (12.5 KiB)

updated source with py3k fixes

[totaam]

2015-05-28 13:04:59: antoine uploaded file palib-1.0.tar.2.xz (13.0 KiB)

updated source with py3k fixes and server_info support

[totaam]

2015-05-28 13:22:17: antoine commented

---

As of r9537, the code will now try to load palib and print a warning before falling back to execing pactl as before.
I have posted beta RPM builds of python-palib.

Everything seems to work, and faster too. It is cleaner / safer than the previous code which relies on execing "pactl" and parsing strings (and hoping they won't change in the future!).
We also get more information out of the pulseaudio server:

$ ./xpra/sound/pulseaudio_util.py 
device.alsa_input.pci-0000_00_14.2.analog-stereo                 : Built-in Audio Analog Stereo
device.alsa_input.usb-046d_HD_Pro_Webcam_C920_15475ECF-02-C920.analog-stereo : HD Pro Webcam C920 Analog Stereo
device.alsa_output.pci-0000_00_14.2.analog-stereo                : Built-in Audio Analog Stereo
device.alsa_output.pci-0000_00_14.2.analog-stereo.monitor        : Monitor of Built-in Audio Analog Stereo
devices                                                          : 4
pulseaudio.cookie                                                : 394338217
pulseaudio.default_sink_name                                     : alsa_output.pci-0000_00_14.2.analog-stereo
pulseaudio.default_source_name                                   : alsa_input.pci-0000_00_14.2.analog-stereo
pulseaudio.host_name                                             : desktop
pulseaudio.id                                                    : 1000@0d37226ca0064aaab1db9e66016c45f5/2311
pulseaudio.server                                                : {0d37226ca0064aaab1db9e66016c45f5}unix:/run/user/1000/pulse/native
pulseaudio.server_name                                           : pulseaudio
pulseaudio.server_version                                        : 6.0
pulseaudio.user_name                                             : antoine
pulseaudio.wrapper                                               : palib

about python palib: this library is old (2010), the code is ugly, badly indented, etc..
But it works.

Ideally, we could re-write the bits we use from palib using libpulseaudio - which is also ugly, less friendly (the code is generated automatically from the library) - but since we use very little and don't actually use the mainloop, this would be cleaner.

[totaam]

2015-06-15 20:39:43: antoine commented

---

As of r9633 + r9636 (+some minor fixes in r9634 + r9635), we can run gstreamer 1.x from the GTK2 client or server (or even gstreamer 0.10 from the GTK3 client if one wanted to do so - apart from testing, I can't imagine why you would want to do that).

All calls to the sound subpackage now go through a subprocess, which can use a different gobject / glib library.

By default, we still run sound with the same python version as the client or server that controls it, but this can be overriden on posix with:

XPRA_SOUND_COMMAND="/usr/bin/python3 /usr/bin/xpra" /usr/bin/python2 xpra start ...

OSX and win32 would require a lot more work to support this as we would need to package two versions of the python interpreters (2.7.x and 3.x), each with all the dependencies.

---

Here are some new useful hidden subcommands, used by some xpra help command line options and for populating the list of codecs on startup:

$ python2 /usr/bin/xpra _sound_query sources
sources=pulsesrc,autoaudiosrc,alsasrc,osssrc,oss4src,audiotestsrc

$ python2 /usr/bin/xpra _sound_query encoders
encoders=mp3,flac,wavpack,wav

$ python2 /usr/bin/xpra _sound_query decoders
decoders=mp3,flac,wavpack,wav

Note how you get different results when running with python3 (where we now disable flac to avoid a bug):

$ python3 /usr/bin/xpra _sound_query encoders
encoders=mp3,wavpack,wav

[totaam]

2015-07-08 10:34:04: antoine uploaded file sound-autotune-queue-v2.patch (6.9 KiB)

updated patch

[totaam]

2015-07-08 11:18:19: antoine commented

---

palib leaks file descriptors and is unmaintained, we need something better.
(see #912)

[totaam]

2015-08-06 11:23:08: antoine uploaded file sound-show-queue-graph.patch (7.2 KiB)

shows the min/max/cur values of the sound buffer as a graph on session info

[totaam]

2015-08-06 14:47:16: antoine uploaded file queue-levels-graph.png (9.0 KiB)

the new queue level graph shown in the session info window

[totaam]

2015-08-06 15:04:02: antoine commented

---

I've wasted time on pulseaudio for nothing, see #912#comment:4 ...

But as of r10227, we now show the sound buffer level and min/max limits in use:
[[Image(queue-levels-graph.png)]]

I want to do a bit more testing on various platforms to see how this behaves, what range of values we get, etc..
@afarr: feel free to provide feedback on this.

Then it shouldn't be too hard to adjust the levels in a more gradual way.
We ought to be able to figure out how low we can get the level by keeping a history of the level range we observe.
Actually lowering the level is a bit more difficult:

• we can drop packets - as we now do when we hit overruns, but how many milliseconds this drops will vary - and is not very predictable, and this can be more than we want...
• playing with the queue limits as we do now, for both underruns and overruns, which drops packets after they've been parsed - which means that we are dealing with time limits at this point not just raw packets (this is more in line with what we're trying to achieve)
• changing the playback speed - but I couldn't get that to work when I tried: looks like we're supposed to use seek events, maybe this conflicts with a live source

Problems:

• wav is not playing nice (fills the buffer way too quickly) - not sure we care, could just drop this encoding
• testing network conditions is hard...
• bound to be platform quirks (osx...)

[totaam]

2015-08-07 13:52:46: antoine uploaded file sound-source-jitter.png (22.5 KiB)

example of sound source jitter causing overruns and underruns

[totaam]

2015-08-07 13:52:58: antoine commented

---

As of r10231, it is a bit easier to stress test the sound level:

XPRA_SOUND_SOURCE_JITTER=700 xpra start ..

Will introduce random jitter in the delivery of sound buffers (given in milliseconds, the random value will be between 1 and the value specified). This will cause the client side levels to go up, and occasionally hit overruns:
[[Image(sound-source-jitter.png)]]

[totaam]

2015-08-07 14:32:04: antoine uploaded file 100ms-precision.png (13.7 KiB)

switch to 100ms sampling so we can see more clearly the changes

[totaam]

2015-08-07 14:33:10: antoine commented

---

The data sampling is still asynchronous, but as of r10234 we run it every 100ms which makes the graph data more precise:
[[Image(100ms-precision.png)]]

[totaam]

2015-08-10 09:59:05: antoine uploaded file sound-tune.patch (10.2 KiB)

try to tune the queue dynamically to minimize buffering

[totaam]

2015-08-10 13:25:21: antoine uploaded file lower.png (18.4 KiB)

shows how we gradually lower the buffer level by lowering the max level

[totaam]

2015-08-10 13:25:44: antoine uploaded file underruns.png (28.1 KiB)

harmless underruns

[totaam]

2015-08-11 06:18:20: antoine uploaded file win7-lower-buffer.png (18.2 KiB)

a very clear example of what we're trying to do with the max level: lower the current level

[totaam]

2015-08-11 06:59:53: antoine commented

---

Recap: the point of these changes is to try to keep the buffer level low as this helps with #835.

Dealing with underruns is relatively easy: we just temporarily increase the min-level which causes data to accumulate in the buffer (visually, the green line goes up and back down again)
Going below "min" usually triggers an underrun, but that doesn't always causes the sound to stutter or stop.
Here's what it looks like:
[[Image(underruns.png)]]

[[BR]]

---

Dealing with overruns is much harder:

• we want to be able to cause overruns to lower the buffer level to make the queue drop some buffers, it looks like this:
  [[Image(win7-lower-buffer.png)]]
• if we get too many overruns then we need to raise the max level, because overruns cause the sound to crackle

Those two behaviours conflict with each other..
The events are asynchronous and multi-threaded, which makes it very hard to tie things together. For example, the "Level" can go above the "max" threshold without triggering an overrun if the pipeline consumes the buffers in time...
Changing the "max" level also causes the sound pipeline to stutter, so we can't change it too often.

Here's what we try to do in r10250:

• We keep track of how many overruns we hit and when. If the number increases (only keeping track of the last 60 seconds) then we raise the limit to try to prevent more overruns.
• We lower the max level if the range of values of the current level is small enough that it should fit in a smaller space (the vertical range of the blue line) - which often causes an overrun, and:
• When we hit an overrun, we temporarily lower the max level, which drains some of the buffer contents
  (this lowering causes the range to go up... which can cause the max level to go back up, but by that point the work has been done already)

New tunables:

• XPRA_SOUND_GRACE_PERIOD=2000 number of milliseconds to wait for before we do anything with underruns and overruns, allows the pipeline time to settle down a bit - probably find as it is. (could eventually be replaced by an event, like the one we get when the decoder is processing the codec and gives us the codec name - meh)
• XPRA_SOUND_MARGIN=50 (from 0 to 200), tunes how aggressively we try to lower the max level (from 0=no margin=aggressive to 200 which lets the buffer level go wild) - this one may well need tuning. Too low and we get too many overruns, too high and the buffer level is too high...
• XPRA_SOUND_FAKE_OVERRUN no longer exists..

Testing notes / some ideas of things to try to see what effect they have:

• to see what the sound subprocess does, run the client with XPRA_SOUND_SUBPROCESS_DEBUG=1 xpra attach ...
• try different apps that generate sound (ie: flash, html5, music player)
• start and stop the sound (ie: music player) will often trigger underruns
• start and stop from the tray
• change the volume up and down
• useful to try: various codecs, various OS, various python versions (ie: to test gstreamer 1.x with python3, see comment:9 : XPRA_SOUND_COMMAND=python3 /usr/bin/xpra - we want to switch to python3 / gstreamer 1.x sooner rather than later, see Please update to GStreamer 1.x #903)
• various network conditions (see also comment:12)
• various application workloads (heavy video will use more bandwidth, add latency and tax the cpu)
• various encodings (again: changes the latency/throughput of the network connection)
• there's also the new fault injection code to test: XPRA_WRAPPER_FAULT_INJECTION_RATE=200 xpra attach ...
• I have not tested OSX - I hope it doesn't cause too many problems, there were deadlocks before when we hit too many overruns... (could be related to the leak warning we see on osx - migrate osx build to using jhbuild and modulesets #533#comment:69)
• it would be worth testing just before and after r10250 to get a better baseline to compare against

During my testing I found (mostly with mp3, and also with vorbis):

• Gb lan can keep the buffer below ~120ms
• wifi bumps it up to ~250ms
• win32 is a tad higher than Linux

PS:

• r10249 + r10252 + r10255: minor tweaks, the current sound caps can be seen with xpra info | grep caps
• gstreamer 1.x compat fixes: r10251 + r10253
• r10275 re-enables opus (only with gstreamer 1.x) and speex: a good time to take a look at those again and see if they are better than mp3 for our use case (and also re-test wavpack / flac - though I am seeing overruns with those..)
• r10276 fixes vorbis - beware though: we can't mix python2/gstreamer-0.10 servers with python3/gstreamer-1.x clients without getting a stream of warnings: GStreamer-CRITICAL **: gst_segment_to_stream_time: assertion 'segment->format == format' failed (not much we can do about this I am afraid... I'll have to write some code to skip this combination since the warning is deep in gstreamer code and we can't really avoid it since gstreamer is the one that is generating the packet metadata that causes the problems)
  And here's the proof:

• this works (0.10 to 0.10, 1.0 to 0.10, 1.0 to 1.0):

gst-launch-0.10 -q audiotestsrc ! vorbisenc ! gdppay ! filesink location=/dev/stdout | gst-launch-0.10 filesrc location=/dev/stdin ! gdpdepay ! vorbisdec ! autoaudiosink
gst-launch-1.0 -q audiotestsrc ! vorbisenc ! gdppay ! filesink location=/dev/stdout | gst-launch-0.10 filesrc location=/dev/stdin ! gdpdepay ! vorbisdec ! autoaudiosink
gst-launch-1.0 -q audiotestsrc ! vorbisenc ! gdppay ! filesink location=/dev/stdout | gst-launch-1.0 filesrc location=/dev/stdin ! gdpdepay ! vorbisdec ! autoaudiosink

• this doesn't (0.10 to 1.0), not sure why it works when we do this same pipeline from our code using python gstreamer...:

gst-launch-0.10 -q audiotestsrc ! vorbisenc ! gdppay ! filesink location=/dev/stdout | gst-launch-1.0 filesrc location=/dev/stdin ! gdpdepay ! vorbisdec ! autoaudiosink

• the graphs only show the client side buffer levels, which is only half of the problem - to evaluate the server side latency you will still need to rely on visual inspection using one of the sync videos (see synchronize sound with video frames #835#comment:1)

[totaam]

2015-10-24 15:05:45: ashtonbarr8800 changed owner from afarr to ashtonbarr8800

[totaam]

2015-10-24 15:05:45: ashtonbarr8800 commented

---

I tried with XPRA_SOUND_SOURCE_JITTER=500 on a random mp3 playing website with chrome (​[http://www.lesmp3.com/] no video playing), and it looks like the max level adjusts pretty well to the "topography" of the sound buffer levels (in the neighborhood of 500 ms). the min level doesn't seem to be adjusting at all in this case though... for the most part rolling at a flat 0 ms.

[totaam]

2015-12-05 10:48:59: antoine changed owner from ashtonbarr8800 to afarr

[totaam]

2015-12-05 10:48:59: antoine commented

---

Note: the min level are only adjusted temporarily to prevent underruns, as we can to keep the queue levels low.
@afarr: still waiting for reproducible test cases, or evidence that we should close or even do more on this ticket. Re-assigning to you since @ashtonbarr8800 is not updating this either.

[totaam]

2015-12-15 02:19:55: afarr uploaded file ongoing-wav-with-videos.png (216.0 KiB)

wav, listening to videos on youtube, with latency (frame total) at +/-157

[totaam]

2015-12-15 02:23:32: afarr uploaded file ongoing-wav-once-latency-drops.png (222.1 KiB)

wav, listening to videos on youtube, after latency (frame total) drops back to +/- 105

[totaam]

2015-12-15 02:31:33: afarr uploaded file wav-once-125ms-100ms-23percent-ended.png (96.9 KiB)

wav, at time when 125ms 100ms 23percent tc delay ended

[totaam]

2015-12-15 02:37:19: afarr uploaded file ongoing-wav-once-disabled-125ms-100ms-23percent.png (83.7 KiB)

wav as latency drops after ending tc delay

[totaam]

2015-12-15 02:39:36: afarr uploaded file wav-with-all-tc-disabled-but-rising-latency.png (91.2 KiB)

wav, as latency rises without tc being used

[totaam]

2015-12-15 03:02:23: afarr commented

---

As long as I have been having issues with osx clients with no mp3 (#970 - comment 19) and fedora 21 servers with no vorbis (#1052) - decided to test the wav heuristics a bit.

Looks like there are issues when using wav codec with latency (total frames) of 150 or more.

Testing using a tc setting of tc qdisc add dev eth0 root netem delay 125ms 100ms 23%, I ran into the following overrun and less overrun graphs with wav:

With latency higher-

[[BR]]

[[Image(ongoing-wav-with-videos.png)]]

[[BR]]

As latency drops-

[[BR]]

[[Image(ongoing-wav-once-latency-drops.png)]]

[[BR]]

Then, once the tc delays are stopped-

[[Image(wav-once-125ms-100ms-23percent-ended.png​)]]

[[BR]]

And, as the latency settles back toward usual-

[[Image(ongoing-wav-once-disabled-125ms-100ms-23percent.png)]]

[[BR]]

And then, interestingly, as the latency rises again without turning to tc:

[[Image(wav-with-all-tc-disabled-but-rising-latency.png)]]

[[BR]]

Not sure how often wav should be expected to be used in a network setting with latencies like this, but it definitely looked like an area where the heuristics seem to be missing a beat.

Checking a number of different tc settings with a connection using mp3, however, showed little of any (new) interest. I'll go through more thoroughly and then also try with a windows client a bit (with mp3 and wav and vorbis, if I can set up a server that will run it) and see if I find anything interesting.

For the most part though, it looks like the overrun buffer heuristics are doing well with un-crazy scenarios.

[totaam]

2016-01-05 09:08:48: antoine commented

---

Follow ups: #970, #1074. #1075

[totaam]

2016-01-20 00:17:33: afarr commented

---

One last couple of notes (Some last couple of...) — I won't bother to clutter with more screenshots though.

Testing some more with 0.17.0 clients (win32 r11649 (1 change) = 11653; osx r11687) against a 0.17.0 server (fedora 23 r11692); using tc to induce delay and drop.

---

mp3

• mp3 w/delay: Buffers generally look good... no real sign of stutters/overruns until about 40 ms delay ... and even with 50 ms 50 ms 25% the buffers are mostly good and only occasional overruns/stutters (often accompanied by spinners... oddly better sync'd than with 3% drop). win32/osx client about the same. Hard to regularly induce 'hiccups' even at 80ms 50ms 35% delay.

• mp3 w/drop/loss: Win32/osx similar... Buffers look good up to 3%, where only very occasional 'hiccups' begin to be seen. At 5% they are still infrequent, but worse when they hit... at 7% not common, but often very bad when they hit, & often accompanied by spinners... At 7% sometimes plays well for a significant while, but when it hits some bad drop it becomes a storm.

---

vorbis

• vorbis w/delay: Win32/osx clients handle about the same - delay seems to be handled by max-buffer up to 40ms, at 50ms some occasional overruns/hiccups are noticeable if one listens long enough, though the Session Info graph doesn't seem to indicate any overruns... and even at 50ms 25ms 25% (or 50ms 50ms 25%) overruns/hiccups are rare to occasional (except where accompanied by spinners). Sound is still rather stable, with only relatively infrequent 'hiccups' with delay set as high as 80ms 50ms 35%.

• vorbis w/drop/loss: Drops of up to 3% are well handled by win32/osx clients, with the graphs indicating max buffer is adjusting pretty well. At 3% drop either client will occasionally have sound 'hiccups', if sound is running long enough (maybe 5 long minutes before any 'hiccups' are heard). At 4% drop the osx client seems to have a more noticeable uptick in 'hiccups' than the win32 client, with the max buffer looking like it's adjusting pretty closely (sometimes a 'hiccup is heard when the "level" and the "max" intersect, without the "level" seeming to cross the "max"), but by 5% drop they both begin to have some noticeable 'hiccups', though sound can often run for several minutes between with no problems. At 7-8% loss the buffer graphs still look pretty good, but sound will cut out with regular 'hiccups' (not to mention spinners).

---

opus (only packaged with the win32 client I was testing with)

• opus w/delay: Up to 40 ms, delay doesn't produce any sign of drops; at 50ms delay, the Session Info seems to be indicating overruns, but the sound still seems perfect to my not-very-sensitive ear. With 50ms 25ms 25% I start getting hiccups/stutters... mostly in conjunction with spinners, but even up to 80ms 50ms 35% the hiccups/stutters aren't very apparent ... until the tc hits a hot streak and the "level" graph starts dropping to 0 periodically (sometimes inducing spinners, sometimes just a sound stutter).

• opus w/loss/drop: With loss 3%, some very occasional 'hiccups' are heard, butffer still looks good though. At 4% the 'hiccups' come often enough to be noticeable. Buffers continue to look good and 'hiccups' don't seem to come much more often, or be much more severe, through 7% loss. At 8% spinners are common, and sound drops with them (but otherwise generally pretty good, and graphs likewise...)

[totaam]

2016-01-28 01:57:49: afarr changed status from new to closed

[totaam]

2016-01-28 01:57:49: afarr set resolution to fixed

[totaam]

2016-01-28 01:57:49: afarr commented

---

Ok, I don't think there's anything left to deal with in this ticket... any new issues will want a new ticket at this point.

Closing.

[totaam]

2017-11-02 13:52:42: antoine commented

---

See also: #1617.