Does using this for "static" (not streaming) audio perform worse than Pyannote?

[benjamin-kirkbride]

This is probably a very ignorant question, but if I have a use case that involves both processing audio files, and also processing real-time streaming audio, should I use Pyannote for the files, and this for the streaming, or do I not lose anything by using this for both?

[juanmc2005]

Hi @benjamin-kirkbride,

I recommend you use pyannote directly for the files and diart for the streams.
pyannote.audio can exploit the entire conversation, while diart needs to act fast and can only rely on current and past audio.
This makes pyannote.audio better for "offline" cases where the entire conversation is pre-recorded.

[benjamin-kirkbride]

@juanmc2005 Thanks so much for the fast reply and the insight!

So I assume in some contexts then it would make sense to use diart for real time display, and then re-process using pyannote.audio after the stream has concluded. I assume that could also be used to fine-tune diart??

[benjamin-kirkbride]

Are there benchmarks for diart vs pyannote.audio for offline use?

[juanmc2005]

So I assume in some contexts then it would make sense to use diart for real time display, and then re-process using pyannote.audio after the stream has concluded. I assume that could also be used to fine-tune diart??

Absolutely, part of this has actually been the subject of one of my recent papers: "Continual Self-Supervised Domain Adaptation for End-to-End Speaker Diarization" (source code here), where we fine-tune the segmentation model on new conversations from a target domain without supervision.

However, our paper does not include an analysis on the improvements this can bring to diart in particular. I'm extremely curious to know though :)

Are there benchmarks for diart vs pyannote.audio for offline use?

The results table in the README shows diart (Ours) with a 5s latency as well as VBx + pyannote.audio for resegmentation in the offline case (VBx w/ overlap-aware segmentation).
Note that offline inference corresponds to an "infinite" latency, since you need to wait for the conversation to finish before you give a prediction.
For a direct comparison with pyannote.audio see the (offline) DER values reported here.

Here we also report the performance of diart under different latencies.

All of this should give you some idea of the performance-latency tradeoff.