Self hosted models #4913
Merged
Self hosted models #4913
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Authored 99% by James (thank you!) Signed-off-by: jamesmcnamara <james.mcnamara@sourcegraph.com> Co-authored-by: Stephen Gutekanst <stephen@sourcegraph.com>
Signed-off-by: Stephen Gutekanst <stephen@sourcegraph.com>
Signed-off-by: Stephen Gutekanst <stephen@sourcegraph.com>
jamesmcnamara
approved these changes
Jul 18, 2024
There was a problem hiding this comment.
I spent an embarassingly long time reviewing the exopenaicompatibile file before realizing it was a rename.
I can't comment too heavily on the config driven provider creation, and without tests I have to rely on you to verify that it is working properly. I think there's some abstraction and testing that should be added to this, but in the effort to unblock this, I'll approve.
I also added a commit to fix some CI errors.
Comment on lines
+297
to
+298
| // TODO(slimsag): self-hosted-models: eliminate model-specific conditionals here entirely | ||
| // by relying on ClientSideConfig appropriately. |
pkukielka
pushed a commit
to sourcegraph/jetbrains
that referenced
this pull request
Jul 22, 2024
This PR update's cody commit. The most important LLM selection dropdown **with self hosted models** is expected to be visible for enterprise users now. The important PRs from Cody to be included: - sourcegraph/cody#4913 (self hosted models) - ~sourcegraph/cody#4706 ([a bug with paths](sourcegraph/cody#4706 (comment))? [slack thread](https://sourcegraph.slack.com/archives/C05AGQYD528/p1721382757890889))~ FIXED BY sourcegraph/cody#4955 ## Test plan 1. full QA
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
These are the initial client changes needed for Cody Enterprise customers to be able to use self-hosted models. Specifically, this PR pairs with https://github.com/sourcegraph/sourcegraph/pull/63899
The first commit here is mostly changes @jamesmcnamara made to help me wire through the
Modelto the actual location where we write autocomplete providers. The following commits are me introducing a new genericopenaicompatibleautocomplete provider, which will supersede/replace the older experimental one which was purely client-side.Combined with https://github.com/sourcegraph/sourcegraph/pull/63899 - these changes are enough to have at least Chat and Autocomplete working fairly well using a self-hosted Starcoder2 and Mixtral 8x7b Instruct model.
In this PR, the autocomplete provider is mostly tailored towards models in those families (starcoder and mixtral/mistral instruct models) - to unblock some customer use-cases - but all of the
ModelclientSideConfigurationoptions piped down to this autocomplete provider should be enough to generalize this approach to any model in the future. In a follow-up PR, I will begin making use of thatclientSideConfigurationto enable using other self-hosted models and other use-cases (letting the site admin customize prompting, etc.)Test plan