Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add fusion-for-decoder-only for llama #733

Open
wants to merge 24 commits into
base: main
Choose a base branch
from
Open

Conversation

binxuan
Copy link

@binxuan binxuan commented Jul 28, 2023

This is a fusion-in-decoder implementation for decoder-only model in FasterTransformer which supports encoding N contexts in parallel and do generation based on these N contexts. This implementation is based on the great work from this repo. We currently re-use batch_size dimension to do this encoding operation. A followup technique report/paper will be released soon.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants