Enhancement: Implement Support for Cloudflare's AI Gateway for Azure endpoint

[ajkost]

Contact Details

No response

What features would you like to see added?

I propose integrating support for Cloudflare's AI Gateway, specifically for Azure OpenAI deployments, into LibreChat. Thank you

More details

key features:

• Analytics
• Real-time logs
• Caching
• Rate limiting

Which components are impacted by your request?

Endpoints

Pictures

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[danny-avila]

I believe you can already use it for the OpenAI endpoint as a reverse proxy. You can also look into LiteLLM as a reverse proxy service that supports this

[ajkost]

Thank you for quick response. It should work with openai, but doesn't work with azure openai, because I cannot specify the url, because it is constructed out of INSTANCE_NAME DEPLOYMENT_NAME and API_VERSION. What I expected is native support for cf's AI gateway for azure endpoints.

I will explore LiteLLM, but looks like another thing that needs to be deployed alongside the librechat., which seems to be a bit overkill just for this simple use case.

Anyways, thank you again.

[danny-avila]

@ajkost I understand better now. I think I can implement passing a custom url which should cover this use case and others.

[paychex-ssmithrand]

@danny-avila along the same lines, we use Azure API Management as a gateway to multiple Azure OpenAI deployments. Is the current thought to just allow a custom domain + path in its entirety instead of attempting to piece it together? Right now we have a modified version of api/utils/azureUtils.js that does this:

const genAzureEndpoint = ({ azureOpenAIApiInstanceName, azureOpenAIApiDeploymentName }) => {
  return `https://${azureOpenAIApiInstanceName}${azureOpenAIApiDeploymentName}`;
};

Where azureOpenAIApiInstanceName = apimgateway.azure-api.net/openai/deployments and the deployment name is the model name.

[danny-avila]

Is the current thought to just allow a custom domain + path in its entirety instead of attempting to piece it together?

Yes as another option. Shouldn't be too hard but need to make sure it plays nice with other configs.

[paychex-ssmithrand]

@danny-avila any movement on this particular ask? I wonder if the title should be changed to better reflect the general need rather than just for cloudflare.

[danny-avila]

@danny-avila any movement on this particular ask? I wonder if the title should be changed to better reflect the general need rather than just for cloudflare.

I would like to get it done soon as it wouldn't be a big update. It is not a big ask and I think it's crucial for use of Azure

[danny-avila]

This is on my slate for today finally! Sorry it took so long.

[ajkost]

no prob at all, tyt

[danny-avila]

Docs will be updated with the following after #1596 is merged

Using a Specified Base URL with Azure

The base URL for Azure OpenAI API requests can be dynamically configured. This is useful for proxying services such as Cloudflare AI Gateway, or if you wish to explicitly override the baseURL handling of the app.

LibreChat will use the AZURE_OPENAI_BASEURL environment variable, which can include placeholders for the Azure OpenAI API instance and deployment names.

In the application's environment configuration, the base URL is set like this:

# .env file
AZURE_OPENAI_BASEURL=https://example.azure-api.net/${INSTANCE_NAME}/${DEPLOYMENT_NAME}

# OR
AZURE_OPENAI_BASEURL=https://${INSTANCE_NAME}.openai.azure.com/openai/deployments/${DEPLOYMENT_NAME}

# Cloudflare example
AZURE_OPENAI_BASEURL=https://gateway.ai.cloudflare.com/v1/ACCOUNT_TAG/GATEWAY/azure-openai/${INSTANCE_NAME}/${DEPLOYMENT_NAME}

The application replaces ${INSTANCE_NAME} and ${DEPLOYMENT_NAME} in the AZURE_OPENAI_BASEURL, processed according to the other settings discussed in the guide.

You can also omit the placeholders completely and simply construct the baseURL with your credentials:

# .env file
AZURE_OPENAI_BASEURL=https://instance-1.openai.azure.com/openai/deployments/deployment-1

# Cloudflare example
AZURE_OPENAI_BASEURL=https://gateway.ai.cloudflare.com/v1/ACCOUNT_TAG/GATEWAY/azure-openai/instance-1/deployment-1

Setting these values will override all of the application's internal handling of the instance and deployment names and use your specified base URL.

Notes:

• You should still provide the AZURE_OPENAI_API_VERSION and AZURE_API_KEY via the .env file as they are programmatically added to the requests.
• When specifying instance and deployment names in the AZURE_OPENAI_BASEURL, their respective environment variables can be omitted (AZURE_OPENAI_API_INSTANCE_NAME and AZURE_OPENAI_API_DEPLOYMENT_NAME) except for use with Plugins.
• Specifying instance and deployment names in the AZURE_OPENAI_BASEURL instead of placeholders creates conflicts with "plugins," "vision," "default-model," and "model-as-deployment-name" support.
• Due to the conflicts that arise with other features, it is recommended to use placeholder for instance and deployment names in the AZURE_OPENAI_BASEURL

[ajkost]

Works like a charm 💪

I've tested

• gpt-4-vision-preview (with image upload)
• gpt-35-turbo
• gpt-4 (1106-preview)

and everything is working as expected. For the record, I use the model-as-deployment-name feature and also the ${INSTANCE_NAME} and ${DEPLOYMENT_NAME} variables.

Thank you again for your hard work; it is greatly appreciated! Keep up the great work. I can't wait for the RAG support.

[danny-avila]

Awesome glad to hear that! I do know that azure setup is somewhat cumbersome but this is why #1390 is on the roadmap. I would like to offer more dynamic configuration where you can set even different deployments/instances per model, as some are region locked.