Speaker Identification

[EzraEllette]

description

This PR adds speaker identification to screenpipe. Audio is segmented by speaker then transcribed. transcriptions now have a speaker_id column. new table speakers was added with name and metadata columns. speaker_embeddings table was created with a one-to-many relationship for speaker and embeddings.

related issue: #

type of change

• new feature

how to test

Run the speaker_identification test. run screenpipe-server/src/db.rs tests.
Use screenpipe.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
screenpipe	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Nov 13, 2024 5:47am

[louis030195]

might be unrelated to this PR, keep testing

(running two screenpipe at once might be related)

[louis030195]

@EzraEllette did you test on windows btw?

would be good that someone test on windows

[louis030195]

any downside to load / unload the model at every chunk?

[louis030195]

1 GB / 37 GB), Total CPU: 17%, NPU: N/A 2024-11-13T19:00:44.580527Z INFO screenpipe_audio::stt: Preparing segments 2024-11-13T19:00:44.580547Z INFO screenpipe_audio::stt: device: MacBook Pro Microphone (input), resampling from 48000 Hz to 16000 Hz 2024-11-13T19:00:45.288077Z ERROR screenpipe_audio::pyannote::segment: Failed to compute speaker embedding: The frames array is empty. No features to compute. 2024-11-13T19:00:45.288101Z ERROR screenpipe_audio::pyannote::segment: Failed to compute speaker embedding: Failed to compute speaker embedding: The frames array is empty. No features to compute. 2024-11-13T19:00:45.288174Z ERROR screenpipe_audio::pyannote::segment: Failed to compute speaker embedding: The frames array is empty. No features to compute. 2024-11-13T19:00:45.288177Z ERROR screenpipe_audio::pyannote::segment: Failed to compute speaker embedding: Failed to compute speaker embedding: The frames array is empty. No features to compute. 2024-11-13T19:00:45.330388Z ERROR screenpipe_audio::pyannote::segment: Failed to compute speaker embedding: The frames array is empty. No features to compute. 2024-11-13T19:00:45.330408Z ERROR screenpipe_audio::pyannote::segment: Failed to compute speaker embeddi

[louis030195]

 INFO screenpipe_server::resource_monitor: Runtime: 4028s, Total Memory: 2% (1 GB / 37 GB), Total CPU: 16%, NPU: N/A
2024-11-13T19:00:14.578918Z  INFO screenpipe_audio::stt: Preparing segments    
2024-11-13T19:00:14.578965Z  INFO screenpipe_audio::stt: device: MacBook Pro Microphone (input), resampling from 48000 Hz to 16000 Hz    
2024-11-13T19:00:14.998152Z ERROR screenpipe_audio::pyannote::segment: Failed to compute speaker embedding: The frames array is empty. No features to compute.
2024-11-13T19:00:14.998174Z ERROR screenpipe_audio::pyannote::segment: Failed to compute speaker embedding: Failed to compute speaker embedding: The frames array is empty. No features to compute.
2024-11-13T19:00:15.040711Z ERROR screenpipe_audio::pyannote::segment: Failed to compute speaker embedding: The frames array is empty. No features to compute.
2024-11-13T19:00:15.040735Z ERROR screenpipe_audio::pyannote::segment: Failed to compute speaker embedding: Failed to compute speaker embedding: The frames array is empty. No features to compute.
2024-11-13T19:00:15.128138Z ERROR screenpipe_audio::pyannote::segment: Failed to compute speaker embedding: The frames array is empty. No features to compute.
2024-11-13T19:00:15.128158Z ERROR scr

[louis030195]

unrelated but fun:

2024-11-13T18:58:16.221108Z  INFO screenpipe_audio::multilingual: detected language: "fr"    
2024-11-13T18:58:16.532145Z  INFO screenpipe_server::resource_monitor: Runtime: 3917s, Total Memory: 2% (1 GB / 37 GB), Total CPU: 31%, NPU: N/A
2024-11-13T18:58:18.758847Z  INFO screenpipe_audio::whisper:   0.0s-0.0s:     
2024-11-13T18:58:18.758879Z  INFO screenpipe_audio::whisper:   0.0s-6.0s:  Parce que c'est tellement abstrait pour les gens dans la tech.    
2024-11-13T18:58:18.758891Z  INFO screenpipe_audio::whisper:   6.0s-8.0s:  Dans la startup.    
2024-11-13T18:58:18.758911Z  INFO screenpipe_audio::whisper:   10.0s-14.0s:  Si on se retrouve dans YC, qu'est-ce qui va passer avec l'évaluation ?    

whisper detect my voice in french (i spoke english)

[louis030195]

a/MacBook Pro Microphone (input)_2024-11-13_19-16-47.mp4"    
2024-11-13T19:16:48.483339Z  INFO screenpipe_audio::multilingual: detected language: "en"    
2024-11-13T19:16:49.955738Z  INFO screenpipe_audio::whisper:   0.0s-0.0s:     
2024-11-13T19:16:49.955759Z  INFO screenpipe_audio::whisper:   0.0s-1.8s:  Well, like we were previously    
2024-11-13T19:16:49.977495Z  INFO screenpipe_server::core: device MacBook Pro Microphone (input) received transcription Some(" Well, like we were previously\n")    
2024-11-13T19:16:49.978565Z  INFO screenpipe_server::core: Detected speaker: Speaker { id: 90, name: "", metadata: "" }    
2024-11-13T19:16:49.978582Z  INFO screenpipe_server::core: device MacBook Pro Microphone (input) inserting audio chunk: "/tmp/spp/data/MacBook Pro Microphone (input)_2024-11-13_19-16-49.mp4"    
2024-11-13T19:16:50.616236Z  INFO screenpipe_audio::multilingual: detected language: "en"    
2024-11-13T19:16:51.325953Z  INFO screenpipe_audio::whisper:   0.0s-0.0s:     
2024-11-13T19:16:51.325974Z  INFO screenpipe_audio::whisper:   0.0s-30.0s:  Thank you.    
2024-11-13T19:16:51.402875Z  INFO screenpipe_server::core: device MacBook Pro Microphone (input) received transcription Some(" Thank you.\n")    
2024-11-13T19:16:51.403004Z ERROR screenpipe_server::core: Error processing audio result: error returned from database: (code: 1) zero-length vectors are not supported. 

there are a few Thank you (something with VAD i suppose) but maybe not more than on main

[EzraEllette]

Okay there are some bug fixes to make.