New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Feature Request] Memory Offloading #314

Open

elephantpanda opened this issue Feb 28, 2023 · 0 comments

elephantpanda commented Feb 28, 2023

As per this: Flex Gen
and this: Big Models

This is a way to run really large models by splitting the model up into small pieces and only putting a piece of the model on the GPU at one time.

This would be a very useful thing for Barracuda to implement especially if we want it to work on lower end hardware.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment