Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for language models #5

Open
HennerM opened this issue Jun 22, 2023 · 1 comment
Open

Add support for language models #5

HennerM opened this issue Jun 22, 2023 · 1 comment

Comments

@HennerM
Copy link
Collaborator

HennerM commented Jun 22, 2023

Previously reported in #2 (comment) by @aamir-s18

The backend currently only supports encoder-decoder models,
whereas the underlying library also has support for decoder-only models: https://github.com/OpenNMT/CTranslate2/blob/master/src/models/language_model.cc

This should be fairly straightforward to add. Ideally we want to auto-detect the type of the model, or alternatively specify in the configuration.

@aamir-s18
Copy link

It would be great to support all the models ctranslate2 supports now (like Whisper and Encoder only).

I considered the best way to abstract the inference since different model classes have other calling functions (like generate, translate ...). We could create a metaclass which takes over the handling of the inference and initialization, so we have a unified interface to talk with. Tbh, I need to find out how easy this is.

Let's specify the model type in the config.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

No branches or pull requests

2 participants