[Roadmap] AzureAI support for large language models

[wxwatcher2004]

Why
Llama 3.1 405b just released and one of the only way to get access to it is through AzureAI hosting it.

Description
Support AzureAI for non OpenAI models as a cloud service.

Requirements
Here is the current token error when I try to use AzureAI on my Llama 3.1 server. [Issue] OpenAI/azure: Bad Request - "status": "Auth token must be passed as a header called Authorization"

[enricoros]

@wxwatcher2004 to replicate this, how do I setup a service, and where do you see the error? Please add some details so I can reproduce easily.

[wxwatcher2004]

So the issue occurs in the models page and I think is an issue with how Azure works. They have a connection side to OpenAI, I followed the instructions you have and found that to work correctly. The other open source models use serverless options. Best I can tell these are accessed through the model deployment URL and Key. I also tried https://x.cognitiveservices.azure.com/ configuration but that only showed the GPT4 model as well.

working OpenAI connection

Not working Llama 3.1 connection

Azure model setting screen