Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: Realtime API support #3722

Draft
wants to merge 21 commits into
base: master
Choose a base branch
from
Draft

feat: Realtime API support #3722

wants to merge 21 commits into from

Conversation

mudler
Copy link
Owner

@mudler mudler commented Oct 3, 2024 •
edited
Loading

Description

This PR fixes #3714

And also covers #191

Notes for Reviewers

Signed commits

  • Yes, I signed my commits.

@netlify Netlify
Copy link

netlify bot commented Oct 3, 2024 •
edited
Loading

❌ Deploy Preview for localai failed.

Name Link
🔨 Latest commit eb911ad
🔍 Latest deploy log https://app.netlify.com/sites/localai/deploys/67697b0e25156000084b55ac

@mudler mudler mentioned this pull request Oct 3, 2024
Open
@mudler mudler force-pushed the feat/realtime branch 3 times, most recently from 5435a07 to 2db3f3d Compare October 9, 2024 10:57
@github-actions GitHub Actions
Copy link

github-actions bot commented Oct 9, 2024

yamllint Failed

Show Output 
::group::gallery/arch-function.yaml
::error file=gallery/arch-function.yaml,line=66,col=22::66:22 [new-line-at-end-of-file] no new line character at the end of file
::endgroup::

Workflow: Yamllint GitHub Actions, Action: __karancode_yamllint-github-action, Lint: gallery

@mudler
Copy link
Owner Author

mudler commented Oct 14, 2024

Just for reference, openai-realtime-console seems quite nice for testing things out especially at this stage, I've opened up a PR upstream to include a Dockerfile and instructions on how to use it with a local server: openai/openai-realtime-console#59

github-advanced-security[bot]
github-advanced-security bot found potential problems Oct 16, 2024
View reviewed changes
core/http/app.go Fixed Show fixed Hide fixed
@mudler mudler force-pushed the feat/realtime branch 2 times, most recently from 1c61aad to 22579bc Compare October 18, 2024 14:06
@mudler mudler force-pushed the feat/realtime branch 2 times, most recently from 2886390 to deff060 Compare October 30, 2024 15:10
@mudler mudler changed the title WIP: realtime API stub feat: Realtime API support Oct 31, 2024
@mudler mudler force-pushed the feat/realtime branch 4 times, most recently from 7a9d6e8 to ee4ae33 Compare November 6, 2024 17:35
@mattkanwisher
Copy link
Contributor

whats best option here if we want to contribute just make forks of the branch and PRS against this?

@mudler
Copy link
Owner Author

mudler commented Nov 7, 2024

whats best option here if we want to contribute just make forks of the branch and PRS against this?

Yes, that would work just fine!

@mudler mudler force-pushed the feat/realtime branch 2 times, most recently from 60579fb to 4034562 Compare November 7, 2024 22:54
@mudler mudler mentioned this pull request Nov 8, 2024
Merged
1 task
@mudler mudler force-pushed the feat/realtime branch 6 times, most recently from 818122a to a1da931 Compare November 12, 2024 18:12
@mudler mudler added the enhancement New feature or request label Nov 30, 2024
@mudler mudler force-pushed the feat/realtime branch 2 times, most recently from 55e2224 to 8250878 Compare December 4, 2024 08:34
@mudler mudler mentioned this pull request Dec 6, 2024
Merged
1 task
@mudler mudler force-pushed the feat/realtime branch 2 times, most recently from 6b12fd4 to e30925c Compare December 15, 2024 09:09
mudler added 21 commits December 23, 2024 16:00
@mudler
WIP
dfe6d0e 
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
WIP
4e19aab 
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
debug
9987653 
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
aujdio
b17b42c 
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
Add model interface to sessions
25c1fe3 
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
setup ws upgrade
08a358b 
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
Fix route
f157314 
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
chore: simplify passing options to ModelOptions
6e468e9 
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
Load wrapper clients
3b4c1a7 
Testing with:

```yaml
name: gpt-4o
pipeline:
 tts: voice-it-riccardo_fasol-x-low
 transcription: whisper-base-q5_1
 llm: llama-3.2-1b-instruct:q4_k_m
```

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
refactor
f2e0d23 
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
feat: correctly detect when starting the vad server
b1c7fc0 
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
chore: extract realtime models into two categories
51d86b8 
One is anyToAny models that requires a VAD model, and one is
wrappedModel that requires as well VAD models along others in the
pipeline.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
feat(vad): hook vad detection
d76b654 
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
chore(vad): try to hook vad to received data from the API (WIP)
5c2a00f 
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
Attach context for VAD
05bdee6 
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
Move to debug calls
254dc12 
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
Fix lock handling
d9495ca 
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
wip(vad)
6e60879 
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
gRPC client stubs
b1f57a9 
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
wip
78d45e5 
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
WIP
eb911ad 
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Add support for realtime API
2 participants
@mudler @mattkanwisher