feat(enterprise): allow multiple models for enterprise customers #4780

jamesmcnamara · 2024-07-04T06:33:01Z

Allows for enterprises to support multiple models. Additionally contains some refactors for unifying further a single Model and ModelsService class.

Test plan

Added E2E test

- Add ModelTag enum to categorize models - Remove deprecated models from model options - Update ModelSelectField to only show non-deprecated models TODO: - replace uiGroup with tags - remove codyProOnly & uiGroupfields & usage from Model class

…rprise-models

…ody into jsm/cody-enterprise-models

- Add `isCodyProModel`, `isCustomModel`, `isLocalModel`, and `isOllamaModel` utility functions to identify different types of models - Update model tags to include `Local` and provide more context around the meaning of different tags - Update usage of these utility functions in the codebase

abeatrix

left some comments in line. It looks like the model/index.ts file was removed by accident so I wasn't able to test it locally, but most of the changes look good to me!

There is an error caused by the change from my other commits that was merged into your branch, I've started the PR (that target your current branch) with the changes to fix that: #4795

Feel free to merge that into this branch and make any additional changes as you need!

Great job!

abeatrix · 2024-07-05T14:46:04Z

agent/src/agent.ts

+                modelID = ModelsService.getDefaultChatModel(authStatus)
+            }
+            if (!modelID) {
+                throw new Error('agent::chat/restore: no chat model found')


This assumes the models are loaded before the chat is restore. I think we can just skip this check for chat/restore because of the model doesn't exists it will just fall back to use the default anyway, instead of breaking the chat with no way to fix it in the UI. Wdyt?

Sure. Except I don't see where it would fallback to the default, everywhere in SimpleChatProvider it just uses the modelId directly.

agent/src/agent.ts

abeatrix · 2024-07-05T14:52:38Z

lib/shared/src/models/dotcom.ts

        usage: [ModelUsage.Chat, ModelUsage.Edit],
        // Has a higher context window with a separate limit for user-context.
        contextWindow: expandedContextWindow,
-        deprecated: false,
-        uiGroup: ModelUIGroup.Balanced,
+        tags: [ModelTag.Gateway, ModelTag.Balanced, ModelTag.Free],


We can probably add a comment to this file or turn this file into a file we use for testing, since this file will soon be deprecated.

abeatrix · 2024-07-05T14:57:27Z

vscode/webviews/chat/models/chatModelContext.tsx

@@ -33,5 +33,5 @@ export function useChatModelByID(

 export function useCurrentChatModel(): Model | undefined {
    const { chatModels } = useChatModelContext()
-    return chatModels?.find(model => model.default) ?? chatModels?.[0]
+    return chatModels?.[0]


Before this PR, whatever that is set to the current default would be the current model. Did we make new changes where we would move the last used model to the front of the model list?

Yes it happens here. Trying to update the field default was kind of a violation of the immutability of Model and did lots of weird casting which I didn't love. And besides we alway fell back to the head of the list so I just settled on that convention.

lib/shared/src/models/index.ts

…ody into jsm/cody-enterprise-models

…rprise-models

jamesmcnamara · 2024-07-09T23:28:37Z

‼️ Hey @sourcegraph/cody-security, please review this PR carefully as it introduces the usage of an unsafe_ function or abuses PromptString.

Can be ignored, it's just a syntactic change to an existing function

abeatrix

I've left some comments inline. Most of them are non-blockers, except for some of the the ModelTags are not assigned correctly for the pre-definded dotcom models. Other than that they all look good to me, and the models are set correctly on reload and updated when I run it locally. Great job!!

lib/shared/src/models/dotcom.ts

lib/shared/src/models/tags.ts

vscode/src/chat/chat-view/SimpleChatPanelProvider.ts

vscode/webviews/Chat.tsx

abeatrix · 2024-07-10T00:22:35Z

lib/shared/src/models/index.ts

+        // BUG: There is data loss here and the potential for ambiguity.
+        // BUG: We are assuming the modelRef is valid, but it might not be.


maybe in a follow-up, we could create a helper function to validate if the modelRef is valid or not?

Definitely. But I can look into that in a subsequent PR as we do more end-to-end testing once all the backend changes are in-place.

lib/shared/src/models/index.ts

Co-authored-by: Beatrix <68532117+abeatrix@users.noreply.github.com>

…rprise-models

abeatrix · 2024-07-10T17:31:37Z

vscode/CHANGELOG.md

@@ -9,6 +9,7 @@ This is a log of all notable changes to Cody for VS Code. [Unreleased] changes a
 - Ollama: Added support for running Cody offline with local Ollama models. [pull/4691](https://github.com/sourcegraph/cody/pull/4691)
 - Edit: Added support for users' to edit the applied edit before the diff view is removed. [pull/4684](https://github.com/sourcegraph/cody/pull/4684)
 - Autocomplete: Added experimental support for Gemini 1.5 Flash as the autocomplete model. To enable this experimental feature, update the `autocomplete.advanced.provider` configuration setting to `unstable-gemini`. Prerequisite: Your Sourcegraph instance (v5.5.0+) must first be configured to use Gemini 1.5 Flash as the autocomplete model. [pull/4743](https://github.com/sourcegraph/cody/pull/4743)
+- Enterprise: Enabled support for multiple dynaic models if the Sourcegraph backend provides them. Requires the experimental flag `cody.dev.useServerDefinedModels` which currently returns a static list of models. Note: this won't work until the server side of this feature is merged.


Suggested change

- Enterprise: Enabled support for multiple dynaic models if the Sourcegraph backend provides them. Requires the experimental flag `cody.dev.useServerDefinedModels` which currently returns a static list of models. Note: this won't work until the server side of this feature is merged.

- Enterprise: Enabled support for multiple dynaic models if the Sourcegraph backend provides them. Requires the experimental flag `cody.dev.useServerDefinedModels` which currently returns a static list of models. Note: this won't work until the server side of this feature is merged. [pull/4780](https://github.com/sourcegraph/cody/pull/4780)

Ooh this actually reminds me that that changelog is no longer true. I'll update both.

abeatrix

This is awesome! I've manually tested the following and verified it works accordingly:
[x] Can switch between chat models
[x] Chat models are displayed with the correct icons
[x] Chat models are displayed correctly with the correct UI group assigned
[x] New chat panel defaulted to use the last selected model correctly
[x] Can switch between inline-edit models
[x] Inline-edit models are displayed with the correct icons
[x] New chat panel defaulted to use the last selected model correctly
[x] Reloaded editor and the last selected models are showing up as default correctly

Screen.Recording.2024-07-10.at.10.37.03.AM.mov

jamesmcnamara · 2024-07-10T18:03:00Z

Thanks @abeatrix for the thorough testing!

jamesmcnamara · 2024-07-10T21:27:28Z

@chrsmith Do you want me to hold off merging this until you get a chance to review it?

chrsmith

From my understanding of how things all fit together this looks good to me! I assume there will be some subtle details to work out as we do more testing with server-supplied LLM models. But overall this looks like a great step in the right direction.

I wouldn't block submission on any of my suggestions or nit picks. Just things to consider.

chrsmith · 2024-07-10T21:37:41Z

lib/shared/src/models/dotcom.ts

        usage: [ModelUsage.Chat, ModelUsage.Edit],
        contextWindow: expandedContextWindow,
-        deprecated: false,
-        uiGroup: ModelUIGroup.Accuracy,
+        tags: [ModelTag.Gateway, ModelTag.Accuracy, ModelTag.Recommended, ModelTag.Free],


Would you mind elaborating on why you added this refactoring? It's clearly an improvement over having fields like default and deprecated, and it's also a cleaner way to represent things like "UI Groups". (Read: 👍 💯 !)

I'm just wondering if this was a change that had been discussed on the Cody Core team for a while, and/or if it was just something you saw as beneficial in this PR. Since, depending on that, maybe we'd want to reflect this "model tags" approach in the backend, too.

This started as @abeatrix brain child, so I'll let her elaborate on it.

chrsmith · 2024-07-10T21:41:26Z

lib/shared/src/models/index.ts

-    // in the UI
-    public deprecated = false
+    /**
+     * The tags assigned for categorizing the model.


Would we want to add some sort of invariant to enforce certain standards for tags? e.g. enforcing that a model can only have one tag in the set { accuracy, speed, balanced }? I can see a high-quality model being both accurate and fast... but since those terms are subjective, it might just lead to that type of classification being meaningless over time.

WDYT?

chrsmith · 2024-07-10T21:42:11Z

lib/shared/src/models/index.ts

+        // BUG: There is data loss here and the potential for ambiguity.
+        // BUG: We are assuming the modelRef is valid, but it might not be.


Definitely. But I can look into that in a subsequent PR as we do more end-to-end testing once all the backend changes are in-place.

chrsmith · 2024-07-10T21:43:43Z

lib/shared/src/models/index.ts

+        await ModelsService.storage?.set(ModelsService.storageKeys[type], resolved.model)
+    }
+
+    public static canUserUseModel(status: AuthStatus, model: string | Model): boolean {


Nit: canUserUseModel is a bit clunky. What do you think about just canUseModel or isModelAvailable?

For me, that's asking "Is this model functionally usable" but the question is really "does this user have permissions for this model". I could see isModelAvailableFor but that might be just as clunky.

lib/shared/src/models/index.ts

chrsmith · 2024-07-10T21:55:55Z

lib/shared/src/models/index.ts

+export function capabilityToUsage(capability: ModelCapability): ModelUsage[] {
+    switch (capability) {
+        case 'autocomplete':
+            return []


Not sure how this particular function will impact things, but my understanding is that "Edit" actually means "Code Editing" AKA "autocompletions". (And renaming "edit" to "autocomplete" was an intentional break in the new server-side schema.)

So this might not be pedantically correct, but I'm not sure if it matters.

This is just trying to parse what was specified here to something the extension can understand.

From what I've seen on the front end, autocomplete models seem to be handled by configuration values and not from the ModelsService, but I'm not sure how that works for enterprise. I've tried to wade through the completions code but it's non-trivial to say the least.

Outside of autocomplete, each model may flag that they support being used for chat or for being used for code-editing (like generate unit tests, fixup etc), but it's possible that some models aren't available for both.

So I'm not sure if what you were describing was on the server side or current client side, but that's my brain dump. Given that, what do you think the mapping should be?

chrsmith · 2024-07-10T21:56:23Z

lib/shared/src/sourcegraph-api/rest/client.ts

    }
 }
+
+// TODO(jsm): delete these


Friendly reminder to delete these :D

chrsmith · 2024-07-10T21:57:15Z

vscode/CHANGELOG.md

@@ -9,6 +9,7 @@ This is a log of all notable changes to Cody for VS Code. [Unreleased] changes a
 - Ollama: Added support for running Cody offline with local Ollama models. [pull/4691](https://github.com/sourcegraph/cody/pull/4691)
 - Edit: Added support for users' to edit the applied edit before the diff view is removed. [pull/4684](https://github.com/sourcegraph/cody/pull/4684)
 - Autocomplete: Added experimental support for Gemini 1.5 Flash as the autocomplete model. To enable this experimental feature, update the `autocomplete.advanced.provider` configuration setting to `unstable-gemini`. Prerequisite: Your Sourcegraph instance (v5.5.0+) must first be configured to use Gemini 1.5 Flash as the autocomplete model. [pull/4743](https://github.com/sourcegraph/cody/pull/4743)
+- Enterprise: Enabled support for multiple dynaic models if the Sourcegraph backend provides them. Requires the experimental flag `modelsAPIEnabled` to be sent by the client config API. Note: this won't work until the server side of this feature is merged. [pull/4780](https://github.com/sourcegraph/cody/pull/4780)


It seems odd to add a caveat like "this won't work until..." to the changelog, since by the time we expect someone to actually read it in a release that should already be the case, right? (Also, @slimsag can keep me honest, but I believe the modelsAPIEnabled field is already available.)

We should just add the changelog entries when we're done with everything yeah.

And yes, modelsAPIEnabled already works today - it will be true as long as you add this to your site config:

"modelsConfiguration": {},

vscode/src/chat/chat-view/SimpleChatModel.ts

…rprise-models

…tion

…rprise-models

vscode/webviews/components/modelSelectField/ModelSelectField.tsx

jamesmcnamara · 2024-07-12T22:58:23Z

@chrsmith I started to work on the server sent defaults but I realized there was going to be a significant amount of work. Since this keeps getting outdated I'm going to merge what we have now, and then continue with the defaults work on another.

This update is to align with a [refactor](sourcegraph/cody#4780) in the vscode repo around model types to remove certain hard coded fields and replace them with ModelTags. This is very rough around the edges because I don't know how to access the generated code in that PR (nor am I aware of the context on this repo) so any feedback is welcome! ## Test plan Green CI --------- Co-authored-by: Piotr Kukielka <pkukielka@virtuslab.com>

jamesmcnamara and others added 18 commits July 2, 2024 18:03

WIP

e7a420b

remove codyProOnly and uiGroup

0f50258

Update bindings

e412283

Update bindings

a7c02d5

Merge branch 'bee/tags-model' into jsm/cody-enterprise-models

3741398

Merge branch 'main' of github.com:sourcegraph/cody into jsm/cody-ente…

0580d74

…rprise-models

removed getModel and setModel

ee4ba56

spring cleaning

52e954e

spring cleaning pt. ii

0a4076c

Merge branch 'main' of github.com:sourcegraph/cody into jsm/cody-ente…

4e0179f

…rprise-models

fixed some tests

b59d717

fixes for tests

34d53ba

fixed type error

3333b8d

fixed type error

db80ff8

Merge branch 'jsm/cody-enterprise-models' of github.com:sourcegraph/c…

3e26028

…ody into jsm/cody-enterprise-models

removed console

bd7d289

abeatrix reviewed Jul 5, 2024

View reviewed changes

abeatrix mentioned this pull request Jul 5, 2024

POC: add model tags and remove deprecated models #4772

Closed

jamesmcnamara added 6 commits July 6, 2024 20:48

Reverted missing file

ff5cbbe

Merge branch 'bee/cody-enterprise-models' of github.com:sourcegraph/c…

3a49cad

…ody into jsm/cody-enterprise-models

updated recordings

81099b3

Merge branch 'main' of github.com:sourcegraph/cody into jsm/cody-ente…

b2ecc25

…rprise-models

updated recordings

7a9c556

added free model

e89f8cd

jamesmcnamara marked this pull request as ready for review July 8, 2024 18:56

ModelsService only accepts a valid model as default

8ecbb10

jamesmcnamara force-pushed the jsm/cody-enterprise-models branch from 2053803 to 1bcbf5e Compare July 8, 2024 22:27

Added an e2e test for enterprise model selection

caaa209

abeatrix reviewed Jul 10, 2024

View reviewed changes

jamesmcnamara and others added 6 commits July 9, 2024 19:14

Update lib/shared/src/models/dotcom.ts

d19007b

Co-authored-by: Beatrix <68532117+abeatrix@users.noreply.github.com>

Apply suggestions from code review

65db69b

Co-authored-by: Beatrix <68532117+abeatrix@users.noreply.github.com>

regenerated and fixed type errors

2aebb13

Merge branch 'main' of github.com:sourcegraph/cody into jsm/cody-ente…

dbd57ca

…rprise-models

updated recording

0cbffb1

switched e2e tests to use new server-sent models config

fad1402

abeatrix reviewed Jul 10, 2024

View reviewed changes

abeatrix approved these changes Jul 10, 2024

View reviewed changes

Removed stub models and updated changelog

b8a3862

jamesmcnamara mentioned this pull request Jul 10, 2024

simple bug fixes with new model configuration system #4829

Closed

chrsmith approved these changes Jul 10, 2024

View reviewed changes

review comments

f5b3fb4

jamesmcnamara mentioned this pull request Jul 11, 2024

Chat: update chat model selection and default model on auth sync #4844

Merged

jamesmcnamara added 6 commits July 11, 2024 18:16

Merge branch 'main' of github.com:sourcegraph/cody into jsm/cody-ente…

c28cbc1

…rprise-models

test(e2e): refactor command-core test to use focusSidebar helper func…

c71f385

…tion

fixed type error

98fbd94

only set default model if the user took an action

bec077f

Merge branch 'main' of github.com:sourcegraph/cody into jsm/cody-ente…

82dd424

…rprise-models

fixed callsite

d7be1fa

abeatrix reviewed Jul 12, 2024

View reviewed changes

vscode/webviews/components/modelSelectField/ModelSelectField.tsx Show resolved Hide resolved

jamesmcnamara added 2 commits July 12, 2024 15:47

swapped ordering of setting default models

b85268d

added comment

47cd1fe

jamesmcnamara merged commit dc9ea0b into main Jul 12, 2024
18 of 19 checks passed

jamesmcnamara deleted the jsm/cody-enterprise-models branch July 12, 2024 22:58

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(enterprise): allow multiple models for enterprise customers #4780

feat(enterprise): allow multiple models for enterprise customers #4780

jamesmcnamara commented Jul 4, 2024 •

edited

Loading

abeatrix left a comment

abeatrix Jul 5, 2024

jamesmcnamara Jul 8, 2024

abeatrix Jul 5, 2024

abeatrix Jul 5, 2024

jamesmcnamara Jul 8, 2024

jamesmcnamara commented Jul 9, 2024

abeatrix left a comment

abeatrix Jul 10, 2024

chrsmith Jul 10, 2024

abeatrix Jul 10, 2024

jamesmcnamara Jul 10, 2024

abeatrix left a comment

jamesmcnamara commented Jul 10, 2024

jamesmcnamara commented Jul 10, 2024

chrsmith left a comment

chrsmith Jul 10, 2024

jamesmcnamara Jul 10, 2024

chrsmith Jul 10, 2024

chrsmith Jul 10, 2024

chrsmith Jul 10, 2024

jamesmcnamara Jul 10, 2024

chrsmith Jul 10, 2024

jamesmcnamara Jul 10, 2024

chrsmith Jul 10, 2024

chrsmith Jul 10, 2024

slimsag Jul 10, 2024

jamesmcnamara commented Jul 12, 2024

		// BUG: There is data loss here and the potential for ambiguity.
		// BUG: We are assuming the modelRef is valid, but it might not be.

	- Enterprise: Enabled support for multiple dynaic models if the Sourcegraph backend provides them. Requires the experimental flag `cody.dev.useServerDefinedModels` which currently returns a static list of models. Note: this won't work until the server side of this feature is merged.
	- Enterprise: Enabled support for multiple dynaic models if the Sourcegraph backend provides them. Requires the experimental flag `cody.dev.useServerDefinedModels` which currently returns a static list of models. Note: this won't work until the server side of this feature is merged. [pull/4780](https://github.com/sourcegraph/cody/pull/4780)

feat(enterprise): allow multiple models for enterprise customers #4780

feat(enterprise): allow multiple models for enterprise customers #4780

Conversation

jamesmcnamara commented Jul 4, 2024 • edited Loading

Test plan

abeatrix left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jamesmcnamara commented Jul 9, 2024

abeatrix left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

abeatrix left a comment

Choose a reason for hiding this comment

jamesmcnamara commented Jul 10, 2024

jamesmcnamara commented Jul 10, 2024

chrsmith left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jamesmcnamara commented Jul 12, 2024

jamesmcnamara commented Jul 4, 2024 •

edited

Loading