New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

可以与vllm集成吗？ #41

Open

zhangxii opened this issue May 6, 2024 · 1 comment

zhangxii commented May 6, 2024

您好，我看代码中加载模型都是用AutoModelForCausalLM的API，请问是否可以支持用vllm加载模型，提高推理速度？据我所知，vllm似乎也是通过PageAttention来管理内存空间中的 keys 和 values，这与infllm工作有冲突吗？

umie0128 commented Jan 17, 2025

有进展嘛

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment