Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix: re-enable vLLM multiprocessing #116

Merged
merged 2 commits into from
Oct 31, 2024
Merged

fix: re-enable vLLM multiprocessing #116

merged 2 commits into from
Oct 31, 2024

Conversation

quitrk
Copy link
Collaborator

@quitrk quitrk commented Oct 31, 2024

  • revert to launching vLLM as a separate process
  • use vLLM router to expose OpenAI like APIs

fix: re-enable vLLM multiprocessing
dfae863 
* revert to launching vLLM as a separate process
* use vLLM router to expose OpenAI like APIs
saghul
saghul reviewed Oct 31, 2024
View reviewed changes
Copy link
Member

@saghul saghul left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good work! I left some questions / comments.

skynet/http_client.py Show resolved Hide resolved
skynet/env.py Outdated Show resolved Hide resolved
skynet/main.py Show resolved Hide resolved
skynet/modules/ttt/openai_api/app.py Outdated Show resolved Hide resolved
skynet/modules/ttt/openai_api/app.py Show resolved Hide resolved
skynet/modules/ttt/openai_api/app.py Show resolved Hide resolved
@quitrk quitrk merged commit 05e6f14 into master Oct 31, 2024
@quitrk quitrk deleted the tavram/proxy branch October 31, 2024 11:11
shooding added a commit to shooding/skynet that referenced this pull request Nov 26, 2024
@shooding
Merge branch 'master' into zh-tw
c9a5c0c 
* master: (61 commits)
  deps: use pypi provided silero vad, upgrade to latest
  fix: remove public key validation (jitsi#123)
  fix: downgrade vllm (jitsi#122)
  feat: add fallback folder when looking up public keys (jitsi#119)
  fix: add ffmpeg dependency for pytorch
  ref: bypass queueing jobs with invalid payload (jitsi#121)
  fix: replace examplar usage with label for app_id
  feat: add instrumentation for app_id (jitsi#118)
  fix: re-enable vLLM multiprocessing (jitsi#116)
  fix: update incorrect prompt example
  fix: healthchecks failing due to missing internal id (jitsi#115)
  feat(openai-api) use Ollama for local development
  feat: expose openai api endpoints from vllm (jitsi#112)
  feat: update text hint type prompting (jitsi#111)
  feat: add meeting hint type and use it as default (jitsi#110)
  feat: enable requests batching (jitsi#109)
  metrics: add full duration metric
  metrics: add a skipped job status which will not count towards duration metrics
  fix: catch exceptions when echoing fails
  feat: add support for echoing requests (jitsi#107)
  ...

# Conflicts:
#	Dockerfile
#	Makefile
#	requirements.txt
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@saghul saghul saghul approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@quitrk @saghul