Backend pool load balancing lab

[vieiraae]

Backend pool load balancing lab

Playground to try the built-in load balancing backend pool functionality of APIM to either a list of Azure OpenAI endpoints or mock servers.

Get started

Proceed by opening the Jupyter notebook, and follow the steps provided.

Add questions or share feedback bellow

[yash-husqvarna]

Hello,

I was successfully able to deploy the resources.
But when I ran the test the api using a direct http call I am receiving the following output.

▶️ Run: 1 / 20
⌚ 4.67 seconds
Response status: �[1;32m200 - OK�[0m
Response headers: {'Content-Length': '1080', 'Content-Type': 'application/json', 'Date': 'Mon, 18 Nov 2024 10:36:31 GMT', 'Access-Control-Allow-Origin': '*', 'Cache-Control': 'no-cache, must-revalidate', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'apim-request-id': '092379ec-0d0a-4fcd-b1bf-7e1f7bac4089', 'X-Content-Type-Options': 'nosniff', 'x-ms-region': 'East US', 'x-ratelimit-remaining-requests': '1', 'x-ratelimit-remaining-tokens': '1984', 'x-accel-buffering': 'no', 'x-ms-rai-invoked': 'true', 'X-Request-ID': '1f7690c9-495b-4e76-8fe0-dda3bef6c7f9', 'x-ms-client-request-id': 'Not-Set', 'azureml-model-session': 'turbo-0301-cca926c4'}
x-ms-region: �[1;31mEast US�[0m
Token usage: {'completion_tokens': 59, 'prompt_tokens': 32, 'total_tokens': 91} 

💬  No, sorry, I don't have the capacity to tell time. I'm just a digital assistant programmed to be sarcastic and unhelpful. Although, I'm sure you have a fancy little device in your pocket or on your wrist that could tell you the time with a quick glance. 

▶️ Run: 2 / 20
⌚ 1.27 seconds
Response status: �[1;32m200 - OK�[0m
Response headers: {'Content-Length': '918', 'Content-Type': 'application/json', 'Date': 'Mon, 18 Nov 2024 10:36:33 GMT', 'Access-Control-Allow-Origin': '*', 'Cache-Control': 'no-cache, must-revalidate', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'apim-request-id': 'd7f0a8c9-486f-4878-aa18-c84353911c1a', 'X-Content-Type-Options': 'nosniff', 'x-ms-region': 'East US', 'x-ratelimit-remaining-requests': '0', 'x-ratelimit-remaining-tokens': '1968', 'x-accel-buffering': 'no', 'x-ms-rai-invoked': 'true', 'X-Request-ID': 'e0b4ed91-6a26-4b2c-b2f2-a90e85bc3a00', 'x-ms-client-request-id': 'Not-Set', 'azureml-model-session': 'turbo-0301-12b1b122'}
x-ms-region: �[1;31mEast US�[0m
Token usage: {'completion_tokens': 25, 'prompt_tokens': 32, 'total_tokens': 57} 

💬  Sure, let me just consult my sundial and hourglass collection. Just kidding, it's 12:45 PM. 

▶️ Run: 3 / 20
⌚ 1.59 seconds
Response status: �[1;32m200 - OK�[0m
Response headers: {'Content-Length': '1038', 'Content-Type': 'application/json', 'Date': 'Mon, 18 Nov 2024 10:36:35 GMT', 'Access-Control-Allow-Origin': '*', 'Cache-Control': 'no-cache, must-revalidate', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'apim-request-id': '7fbca465-6118-44cb-9130-053d7725d8b5', 'X-Content-Type-Options': 'nosniff', 'x-ms-region': 'West Europe', 'x-ratelimit-remaining-requests': '1', 'x-ratelimit-remaining-tokens': '1984', 'x-accel-buffering': 'no', 'x-ms-rai-invoked': 'true', 'X-Request-ID': '99c0f711-25e1-462d-bebe-2db2e200b9b4', 'x-ms-client-request-id': 'Not-Set', 'azureml-model-session': 'turbo-0301-c770d4e1'}
x-ms-region: �[1;31mWest Europe�[0m
Token usage: {'completion_tokens': 51, 'prompt_tokens': 32, 'total_tokens': 83} 

💬  Sure, let me consult my crystal ball that displays the time in invisible ink. Oh wait, no, I can just tell you the time. It's time for you to get a watch. Just kidding, it's currently [insert current time here]. 

▶️ Run: 4 / 20
⌚ 0.95 seconds
Response status: �[1;32m200 - OK�[0m
Response headers: {'Content-Length': '899', 'Content-Type': 'application/json', 'Date': 'Mon, 18 Nov 2024 10:36:36 GMT', 'Access-Control-Allow-Origin': '*', 'Cache-Control': 'no-cache, must-revalidate', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'apim-request-id': '71feee97-5c7a-4cc0-af8b-0757d9b6f08e', 'X-Content-Type-Options': 'nosniff', 'x-ms-region': 'West Europe', 'x-ratelimit-remaining-requests': '0', 'x-ratelimit-remaining-tokens': '1968', 'x-accel-buffering': 'no', 'x-ms-rai-invoked': 'true', 'X-Request-ID': '9cf22743-73bd-4669-a45f-32ac45081f39', 'x-ms-client-request-id': 'Not-Set', 'azureml-model-session': 'turbo-0301-c770d4e1'}
x-ms-region: �[1;31mWest Europe�[0m
Token usage: {'completion_tokens': 18, 'prompt_tokens': 32, 'total_tokens': 50} 

💬  No, I prefer to keep that information to myself. It's more fun that way. 

▶️ Run: 5 / 20
⌚ 0.73 seconds
Response status: �[1;31m503 - Service Unavailable�[0m
Response headers: {'Content-Length': '0', 'Date': 'Mon, 18 Nov 2024 10:36:36 GMT'}

▶️ Run: 6 / 20
⌚ 0.72 seconds
Response status: �[1;31m503 - Service Unavailable�[0m
Response headers: {'Content-Length': '0', 'Date': 'Mon, 18 Nov 2024 10:36:37 GMT'}

▶️ Run: 7 / 20
⌚ 0.71 seconds
Response status: �[1;31m503 - Service Unavailable�[0m
Response headers: {'Content-Length': '0', 'Date': 'Mon, 18 Nov 2024 10:36:38 GMT'}

▶️ Run: 8 / 20
⌚ 0.71 seconds
Response status: �[1;31m503 - Service Unavailable�[0m
Response headers: {'Content-Length': '0', 'Date': 'Mon, 18 Nov 2024 10:36:38 GMT'}

▶️ Run: 9 / 20
⌚ 0.85 seconds
Response status: �[1;31m503 - Service Unavailable�[0m
Response headers: {'Content-Length': '0', 'Date': 'Mon, 18 Nov 2024 10:36:40 GMT'}

▶️ Run: 10 / 20
⌚ 0.82 seconds
Response status: �[1;31m503 - Service Unavailable�[0m
Response headers: {'Content-Length': '0', 'Date': 'Mon, 18 Nov 2024 10:36:40 GMT'}

▶️ Run: 11 / 20
⌚ 0.84 seconds
Response status: �[1;31m503 - Service Unavailable�[0m
Response headers: {'Content-Length': '0', 'Date': 'Mon, 18 Nov 2024 10:36:41 GMT'}

▶️ Run: 12 / 20
⌚ 0.71 seconds
Response status: �[1;31m503 - Service Unavailable�[0m
Response headers: {'Content-Length': '0', 'Date': 'Mon, 18 Nov 2024 10:36:41 GMT'}

▶️ Run: 13 / 20
⌚ 1.25 seconds
Response status: �[1;32m200 - OK�[0m
Response headers: {'Content-Length': '1004', 'Content-Type': 'application/json', 'Date': 'Mon, 18 Nov 2024 10:36:44 GMT', 'Access-Control-Allow-Origin': '*', 'Cache-Control': 'no-cache, must-revalidate', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'apim-request-id': '455436d4-b40d-4c28-a44a-8194ddfb607c', 'X-Content-Type-Options': 'nosniff', 'x-ms-region': 'East US', 'x-ratelimit-remaining-requests': '0', 'x-ratelimit-remaining-tokens': '1952', 'x-accel-buffering': 'no', 'x-ms-rai-invoked': 'true', 'X-Request-ID': 'df43023e-edd0-4c02-b7f3-c92807df6de5', 'x-ms-client-request-id': 'Not-Set', 'azureml-model-session': 'turbo-0301-42727a61'}
x-ms-region: �[1;31mEast US�[0m
Token usage: {'completion_tokens': 41, 'prompt_tokens': 32, 'total_tokens': 73} 

💬  Oh sure, let me just bend the fabric of space-time to my will and create a custom-made timepiece just for you. Or you can just look at your watch or phone like a regular person. 

▶️ Run: 14 / 20
⌚ 1.09 seconds
Response status: �[1;32m200 - OK�[0m
Response headers: {'Content-Length': '965', 'Content-Type': 'application/json', 'Date': 'Mon, 18 Nov 2024 10:36:45 GMT', 'Access-Control-Allow-Origin': '*', 'Cache-Control': 'no-cache, must-revalidate', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'apim-request-id': '1b87f147-df9f-471e-ae41-8197bb024c28', 'X-Content-Type-Options': 'nosniff', 'x-ms-region': 'East US', 'x-ratelimit-remaining-requests': '0', 'x-ratelimit-remaining-tokens': '1936', 'x-accel-buffering': 'no', 'x-ms-rai-invoked': 'true', 'X-Request-ID': '6ab1617f-d625-47c1-a5fc-1411f2bdfba7', 'x-ms-client-request-id': 'Not-Set', 'azureml-model-session': 'turbo-0301-42727a61'}
x-ms-region: �[1;31mEast US�[0m
Token usage: {'completion_tokens': 36, 'prompt_tokens': 32, 'total_tokens': 68} 

💬  Oh, sure, I would love to do that. Let me just consult my crystal ball... oh wait, I don't have one. Maybe try looking at a clock instead? 

▶️ Run: 15 / 20
⌚ 1.26 seconds
Response status: �[1;32m200 - OK�[0m
Response headers: {'Content-Length': '1085', 'Content-Type': 'application/json', 'Date': 'Mon, 18 Nov 2024 10:36:46 GMT', 'Access-Control-Allow-Origin': '*', 'Cache-Control': 'no-cache, must-revalidate', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'apim-request-id': 'fd466dc6-4a14-4cd5-8c15-15c26ade8e3c', 'X-Content-Type-Options': 'nosniff', 'x-ms-region': 'West Europe', 'x-ratelimit-remaining-requests': '0', 'x-ratelimit-remaining-tokens': '1952', 'x-accel-buffering': 'no', 'x-ms-rai-invoked': 'true', 'X-Request-ID': '941c2d76-1ebd-47d0-830d-8451a1e4ee26', 'x-ms-client-request-id': 'Not-Set', 'azureml-model-session': 'turbo-0301-7890f3a8'}
x-ms-region: �[1;31mWest Europe�[0m
Token usage: {'completion_tokens': 54, 'prompt_tokens': 32, 'total_tokens': 86} 

💬  Oh, absolutely not. The concept of time is simply a human construct, a tool for managing our lives that ultimately leads to our own insignificance in the vastness of the universe. But if you must know, it's probably somewhere between the past and the future. 

▶️ Run: 16 / 20
⌚ 1.06 seconds
Response status: �[1;32m200 - OK�[0m
Response headers: {'Content-Length': '965', 'Content-Type': 'application/json', 'Date': 'Mon, 18 Nov 2024 10:36:47 GMT', 'Access-Control-Allow-Origin': '*', 'Cache-Control': 'no-cache, must-revalidate', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'apim-request-id': '5a96356e-0750-4c2a-9d2b-85a92cdf7a52', 'X-Content-Type-Options': 'nosniff', 'x-ms-region': 'West Europe', 'x-ratelimit-remaining-requests': '0', 'x-ratelimit-remaining-tokens': '1936', 'x-accel-buffering': 'no', 'x-ms-rai-invoked': 'true', 'X-Request-ID': 'e708305a-5eff-4023-8bb5-5247878125eb', 'x-ms-client-request-id': 'Not-Set', 'azureml-model-session': 'turbo-0301-cd4aa9c8'}
x-ms-region: �[1;31mWest Europe�[0m
Token usage: {'completion_tokens': 35, 'prompt_tokens': 32, 'total_tokens': 67} 

💬  No, I can't. I'm too busy being sarcastic and unhelpful. Maybe you should try asking a clock or looking at your phone like a normal person. 

▶️ Run: 17 / 20
⌚ 0.70 seconds
Response status: �[1;31m503 - Service Unavailable�[0m
Response headers: {'Content-Length': '0', 'Date': 'Mon, 18 Nov 2024 10:36:47 GMT'}

▶️ Run: 18 / 20
⌚ 0.68 seconds
Response status: �[1;31m503 - Service Unavailable�[0m
Response headers: {'Content-Length': '0', 'Date': 'Mon, 18 Nov 2024 10:36:48 GMT'}

▶️ Run: 19 / 20
⌚ 0.68 seconds
Response status: �[1;31m503 - Service Unavailable�[0m
Response headers: {'Content-Length': '0', 'Date': 'Mon, 18 Nov 2024 10:36:48 GMT'}

▶️ Run: 20 / 20
⌚ 0.69 seconds
Response status: �[1;31m503 - Service Unavailable�[0m
Response headers: {'Content-Length': '0', 'Date': 'Mon, 18 Nov 2024 10:36:50 GMT'}

The question that I have is that why am I receiving ** 503 Service Unable **.
I thought the whole point of using load balancing is so that the request is routed to the service which is available and it seems weird that even after deploying 2 open-ai resources in 2 different locations I am still receiving this error.

My configuration for the deployment are as follows:

resource_group_name = f"lab-{deployment_name}" # change the name to match your naming style
resource_group_location = "westeurope"
apim_resource_name = "apim"
apim_resource_location = "westeurope"
apim_resource_sku = "Basicv2"
openai_resources = [ {"name": "openai1", "location": "eastus", "priority": 1, "weight": 80}, {"name": "openai2", "location": "westeurope", "priority": 1, "weight": 10}]#, {"name": "openai3", "location": "francecentral", "priority": 1, "weight": 10} ] # list of OpenAI resources to deploy. Clear this list to use only the mock resources
openai_resources_sku = "S0"
openai_model_name = "gpt-35-turbo"
openai_model_version = "0301"
openai_deployment_name = "gpt-35-turbo"
openai_api_version = "2024-02-01"
openai_specification_url='https://raw.githubusercontent.com/Azure/azure-rest-api-specs/main/specification/cognitiveservices/data-plane/AzureOpenAI/inference/stable/' + openai_api_version + '/inference.json'
openai_backend_pool = "openai-backend-pool"
mock_backend_pool = "mock-backend-pool"
mock_webapps = [ {"name": "openaimock1", "endpoint": "https://openaimock1.azurewebsites.net", "priority": 1, "weight": 80}, {"name": "openaimock2", "endpoint": "https://openaimock2.azurewebsites.net", "priority": 1, "weight": 20} ]

Any feedback or help would be greatly appreciated. Thank You :)

[vieiraae]

Thanks for reaching out. The service returns the 503 error when none of the backends is available. Try to increase the TPM allowed by the Azure OpenAI model by changing the openAIModelCapacity parameter in the bicep file. You can also use the tracing tool to see the tracing information and debug the backend pool behavior. Let me know if it helps.

[yash-husqvarna]

Thanks for the reply. I can confirm that the solution suggested by you works. My bad for not noticing it even though it was written in the notebook.