Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: switch to using vllm #103

Merged
merged 1 commit into from
Oct 4, 2024
Merged

feat: switch to using vllm #103

merged 1 commit into from
Oct 4, 2024

Conversation

quitrk
Copy link
Collaborator

@quitrk quitrk commented Oct 3, 2024

  • when cuda is not available, we are still using llama.cpp as a fallback

feat: switch to using vllm
814403b 
* when cuda is not available, we are still using llama.cpp as a fallback
@quitrk quitrk marked this pull request as draft October 3, 2024 08:22
@quitrk quitrk marked this pull request as ready for review October 3, 2024 08:28
quitrk
quitrk commented Oct 3, 2024
View reviewed changes
pyproject.toml Show resolved Hide resolved
saghul
saghul reviewed Oct 3, 2024
View reviewed changes
Copy link
Member

@saghul saghul left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left some comments, mostly just questions, excellent stuff! Can you please also update the README with what AWQ model one should run when using vllm?

run.sh Show resolved Hide resolved
run.sh Show resolved Hide resolved
skynet/env.py Show resolved Hide resolved
skynet/modules/ttt/openai_api/app.py Show resolved Hide resolved
skynet/modules/ttt/openai_api/app.py Show resolved Hide resolved
skynet/modules/ttt/summaries/processor.py Show resolved Hide resolved
skynet/modules/ttt/summaries/processor.py Show resolved Hide resolved
skynet/modules/ttt/summaries/v1/models.py Show resolved Hide resolved
skynet/utils.py Show resolved Hide resolved
@quitrk
Copy link
Collaborator Author

quitrk commented Oct 3, 2024

Left some comments, mostly just questions, excellent stuff! Can you please also update the README with what AWQ model one should run when using vllm?

We're not running AWQ as it is not optimised anyway, we're running raw, and still has some decent performance improvement over llama.cpp with quantized

saghul
saghul reviewed Oct 4, 2024
View reviewed changes
skynet/env.py Outdated Show resolved Hide resolved
saghul
saghul reviewed Oct 4, 2024
View reviewed changes
run.sh Outdated Show resolved Hide resolved
@quitrk quitrk merged commit 94397c2 into master Oct 4, 2024
@quitrk quitrk deleted the tavram/vll branch October 4, 2024 11:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@saghul saghul saghul approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@quitrk @saghul