-
Notifications
You must be signed in to change notification settings - Fork 291
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Self hosted models #4913
Self hosted models #4913
Conversation
Authored 99% by James (thank you!) Signed-off-by: jamesmcnamara <james.mcnamara@sourcegraph.com> Co-authored-by: Stephen Gutekanst <stephen@sourcegraph.com>
Signed-off-by: Stephen Gutekanst <stephen@sourcegraph.com>
Signed-off-by: Stephen Gutekanst <stephen@sourcegraph.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I spent an embarassingly long time reviewing the exopenaicompatibile
file before realizing it was a rename.
I can't comment too heavily on the config driven provider creation, and without tests I have to rely on you to verify that it is working properly. I think there's some abstraction and testing that should be added to this, but in the effort to unblock this, I'll approve.
I also added a commit to fix some CI errors.
// TODO(slimsag): self-hosted-models: eliminate model-specific conditionals here entirely | ||
// by relying on ClientSideConfig appropriately. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
❤️
This PR update's cody commit. The most important LLM selection dropdown **with self hosted models** is expected to be visible for enterprise users now. The important PRs from Cody to be included: - sourcegraph/cody#4913 (self hosted models) - ~sourcegraph/cody#4706 ([a bug with paths](sourcegraph/cody#4706 (comment))? [slack thread](https://sourcegraph.slack.com/archives/C05AGQYD528/p1721382757890889))~ FIXED BY sourcegraph/cody#4955 ## Test plan 1. full QA
These are the initial client changes needed for Cody Enterprise customers to be able to use self-hosted models. Specifically, this PR pairs with https://github.com/sourcegraph/sourcegraph/pull/63899
The first commit here is mostly changes @jamesmcnamara made to help me wire through the
Model
to the actual location where we write autocomplete providers. The following commits are me introducing a new genericopenaicompatible
autocomplete provider, which will supersede/replace the older experimental one which was purely client-side.Combined with https://github.com/sourcegraph/sourcegraph/pull/63899 - these changes are enough to have at least Chat and Autocomplete working fairly well using a self-hosted Starcoder2 and Mixtral 8x7b Instruct model.
In this PR, the autocomplete provider is mostly tailored towards models in those families (starcoder and mixtral/mistral instruct models) - to unblock some customer use-cases - but all of the
Model
clientSideConfiguration
options piped down to this autocomplete provider should be enough to generalize this approach to any model in the future. In a follow-up PR, I will begin making use of thatclientSideConfiguration
to enable using other self-hosted models and other use-cases (letting the site admin customize prompting, etc.)Test plan