Using Azure OpenAI Deployments

[xjw1001001]

Thanks for the great application and the update to support different web url!

I wonder how I can use Azure openAI endpoint?

Here is what I need to provide to AzureOpenAI client in langchain

openai_api_key = xxx
openai_api_version = 2023-05-15
azure_endpoint = https://iapi.companyname.com/xxxx
azure_deployment = "gpt-4o-2024-05-13"

Because there is only openAI API key and base URL available, so I wonder how I can make it useful for my company's deploy can use this package?

Thanks!

[anthony2261]

Hey @xjw1001001 , thanks for the kind words!

We're working on adding different providers instead of just the OpenAI compatible ones, but it will take a bit of time to get this done due to other priorities.

However, there is a temporary solution you could do for the time being, which is using a proxy service like LiteLLM's proxy server. It lets you setup any LLM Provider, and you would just need to set the Dataline base URL to it (for example http://localhost:4000/v1). See the screenshot for how the UI looks:

Keep in mind I haven't tested this setup but LiteLLM has worked for me in the past so I trust that this should work. Let me know if this helps or if you run into any issues!

[xjw1001001]

Sorry for the late reply as I just get the liteLLM set. Here is the liteLLM models I hosted:

But when I use the key sk-1234 and url http://localhost:4000/v1 (I also tried some other such as without v1 or 127.0.0.1:4000)
I got

My litellm is working:

Any help is appreciated! Thanks!

[anthony2261]

Hey @xjw1001001, can you please share these details:

• How are you running Dataline (executable, homebrew, Docker)?
• If you are running the Docker image:
  1- do you have the latest image?
  2- Does it work if you use http://host.docker.internal:4000/v1 instead of http://localhost:4000/v1 as the base URL?
  3- Can you share the logs of the backend when you click Continue in the browser?

[xjw1001001]

Docker. Thank you very much.
The new URL worked. But I had a new problem:

When I ask question in the chat, it said:

Error code: 404 - {'error': {'message': "litellm.APIError: AzureException APIError - Error code: 404 - {'statusCode': 404, 'message': 'Resource not found'}\nReceived Model Group=gpt-3.5-turbo\nAvailable Model Group Fallbacks=None", 'type': None, 'param': None, 'code': '404'}}

And what if I want to use gpt4o instead of gpt3.5? In the setting I did not see the model selection part:

I downloaded the docker image yesterday from the github instruction.

[anthony2261]

Glad that URL worked 😄 Now for the next part, I'm going to have to replicate it myself as I personally haven't used Azure OpenAI. Will let you know what I discover soon!

As for the model selection bit, we've discovered that not all models work with Dataline at this time. For example, gpt-4o was hallucinating data instead of using the tools to see the available tables and running an actual SQL command. gpt-4o-mini was surprisingly better, however.

In any case, it is possible for you to use the model you want. Since you have LiteLLM setup, you can name the "Public model name" gpt-3.5-turbo and name the "LiteLLM model" the model you want. Another way to do this via Dataline is to go to http://localhost:7377/docs, expand the PATCH /settings/info section, click on "Try it out" and put this in the request body and Execute:

{
  "preferred_openai_model": "model_name"
}

For now, let me find the issue with the Azure thing first

[anthony2261]

@xjw1001001 I think I might have found the problem... it's gpt-35-turbo, not gpt-3.5-turbo. They removed the dot . 🤦
Make sure it's gpt-35-turbo in the LiteLLM model name, but gpt-3.5-turbo in the public model name