Use python-rtmixer for low-latency audio recording

[tlecomte]

Friture has issues keeping up with audio input, which tends to produce overflows, that appear as glitches in the audio signal.

This has largely been unnoticed for years because Windows MME is very forgiveable in that regard. But Windows Directsound, for example, is much more sensitive. Similarly, Alsa or PulseAudio in Linux also show a lot of glitches (even if PortAudio might not report any overflow).

Here this is fixed by taking advantage of python-rtmixer, a library that sits on top of python-sounddevice (the PortAudio wrapper). Python-rtmixer provides an audio callback that is implemented in C and doesn’t invoke the Python interpreter, therefore avoiding waiting for things like garbage collection and the GIL.

Reference: https://python-rtmixer.readthedocs.io/en/0.1.0/index.html

The result is glitch-free audio signal even with Windows DirectSound.

We believe this solves #90

[keturn]

This could be a good way to determine if doing those PortAudio callbacks while under the GIL is the source of your troubles!

If that improves things, after reading rtmixer I think there's a way to meet the "C callback for PortAudio Stream" goal by taking some cues from rtmixer without adding the dependency wholesale.

Concerns:

• If the only part of rtmixer you're using is Recorder.record_ringbuffer, i.e. none of the scheduled playback and mixing from its Mixer, that's a small fraction of the rtmixer code.
• If Friture's troubles with keeping up are at all related to the cost of memory allocation or copying, this isn't helping. rtmixer's callback copies the input in to AudioBackend.ringbuffer, then AudioBackend.fetchAudioData does a new allocation every chunk and copies it when it concatenates the two areas of the read buffer, to be copied again when it arrives at Friture's main AudioBuffer.

Speaking of AudioBuffer, looking at the tools Friture already has at its disposal:

• AudioBuffer already uses a ring buffer implementation!
• The project already has a way to write C-callable functions (friture_extensions use Cython)

Of course, implementing that isn't as simple as just

cdef callback(in_data, frame_count, time_info, status, user_data) nogil:
    user_data.ringbuffer.push(in_data)

as I don't expect the existing friture.ringbuffer.push to be callable from a nogil Cython function. But since the AudioBuffer is all built on top of a numpy array already, it seems like it should be doable without much disruption to the interface its readers currently use.

("should be" is what famous last words are made of, I know.)

[tlecomte]

Thanks @keturn for your comment!

My tests show that using python-rtmixer really solves the problem. Following your reasoning, that indicates that the problem was not with memory allocation and copying, but really with the GIL and/or garbage collection pauses in the python callback of python-sounddevice.

It's true that I could probably implement a C or Cython callback directly in Friture. The balance between maintaining it myself and using python-rtmixer tends to push in favor of the latter. python-rtmixer has a fine set of wheels on PyPI which is perfect for Friture continuous integration. It is built on top of the PortAudio implementation of a ring buffer, which is hopefully solid and battle-tested. At this point, I think it's sufficient. It solves the immediate audio input issue.

Thanks again!