Skip to content

Conversation

@peterj
Copy link
Collaborator

@peterj peterj commented Mar 25, 2025

This PR updates the ModelConfig CRD and adds support for Anthropic and Azure OpenAI (still needs to be tested). I've also updated the Agent CRD to reference the ModelConfig + other changes that were necessary to dynamically create the model in autogen based on the ModelConfig resource.

There are also bunch of changes in the UI to support the backend changes.

Here's how to test this out:

  1. Create the secret wit the API key:
kubectl create secret generic kagent-anthropic -n kagent --from-literal ANTHROPIC_API_KEY=<key>
  1. Apply the ModelConfig referencing the secret and setting the provider to Anthropic:
apiVersion: kagent.dev/v1alpha1
kind: ModelConfig
metadata:
  name: claude-model-config
  namespace: kagent
spec:
  apiKeySecretName: kagent-anthropic
  apiKeySecretKey: ANTHROPIC_API_KEY
  model: claude-3-sonnet-20240229
  provider: Anthropic

Similar for Azure:

  1. Secret first:
kubectl create secret generic kagent-azureopenai -n kagent --from-literal AZURE_OPENAI_API_KEY=<key>
  1. Then the modelconfig with azure-specific config:
apiVersion: kagent.dev/v1alpha1
kind: ModelConfig
metadata:
  name: azureopenai-model-config
  namespace: kagent
spec:
  apiKeySecretName: kagent-azureopenai
  apiKeySecretKey: AZURE_OPENAI_API_KEY
  model: gpt-4o-mini
  provider: AzureOpenAI
  azureOpenAI:
    azureEndpoint: "https://<endpointname>.openai.azure.com/"
    apiVersion: "2025-03-01-preview"
    azureDeployment: "gpt-4o-mini"
    azureAdToken: <token>

closes: #174

Copy link
Contributor

@EItanya EItanya left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My main comment on this PR has to do with the use of map[string]interface{}. Totally makes sense why these would look like a good option coming from a JS/Python world. But typically in go you should create a type to represent it when using it within the code. map[string]interface{] is typically used to represent JSON you don't know the format of, but we know the format of all of the JSON in this case.

peterj added 3 commits March 26, 2025 10:15
Signed-off-by: Peter Jausovec <peter.jausovec@solo.io>
Signed-off-by: Peter Jausovec <peter.jausovec@solo.io>
Signed-off-by: Peter Jausovec <peter.jausovec@solo.io>
@Jmadih
Copy link

Jmadih commented Mar 26, 2025

Hello,
Maybe in the future, you could check out LiteLLM (litellm.ai) for managing all LLMs in the OpenAI format.

Signed-off-by: Peter Jausovec <peter.jausovec@solo.io>
Copy link
Contributor

@EItanya EItanya left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking better, but still have the same major comment about the API

peterj added 3 commits March 26, 2025 19:17
Signed-off-by: Peter Jausovec <peter.jausovec@solo.io>
Signed-off-by: Peter Jausovec <peter.jausovec@solo.io>
Signed-off-by: Peter Jausovec <peter.jausovec@solo.io>
@peterj
Copy link
Collaborator Author

peterj commented Mar 27, 2025

Looking better, but still have the same major comment about the API

I think I address all feedback with the latest commit.

@EItanya EItanya merged commit b3be109 into main Mar 27, 2025
7 checks passed
@EItanya EItanya deleted the peterj/moarmodels branch March 27, 2025 11:32
@EItanya EItanya mentioned this pull request Mar 27, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add support for AWS Bedrock and Azure Open AI

4 participants