labs/prompt-flow/prompt-flow.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# APIM ❤️ OpenAI\n",
    "\n",
    "## Prompt flow lab\n",
    "![flow](../../images/prompt-flow.gif)\n",
    "\n",
    "Playground to try the [Azure AI Studio Prompt Flow](https://learn.microsoft.com/azure/ai-studio/how-to/prompt-flow) with Azure API Management. The Prompt Flow OpenAI connection will be facilitated by APIM, enabling load balancing, token counting, and other features. The Prompt Flow will run in an Azure Container App and will be accessed through APIM.\n",
    "\n",
    "Prompt flow is a development tool designed to streamline the entire development cycle of AI applications powered by Large Language Models (LLMs). Prompt flow provides a comprehensive solution that simplifies the process of prototyping, experimenting, iterating, and deploying your AI applications.  \n",
    "Prompt flow is available independently as an open-source project on [GitHub](https://github.com/microsoft/promptflow).\n",
    "\n",
    "### Result\n",
    "![result](result.png)\n",
    "\n",
    "### TOC\n",
    "- [0️⃣ Initialize notebook variables](#0)\n",
    "- [1️⃣ Create the Azure Resource Group](#1)\n",
    "- [2️⃣ Create deployment using 🦾 Bicep](#2)\n",
    "- [3️⃣ Get the deployment outputs](#3)\n",
    "- [4️⃣ Enable Prompt Flow tracing (optional)](#4)\n",
    "- [5️⃣ Update the connection with APIM Gateway URL and APIM Subscription Key](#5)\n",
    "- [🧪 Test the Flow locally](#localtest)\n",
    "- [6️⃣ Create the Flow in the AI Studio Project (optional)](#6)\n",
    "- [7️⃣ Build the Flow in the Docker format](#7)\n",
    "- [8️⃣ Deploy the Flow to an Azure Container App](#8)\n",
    "- [🧪 Test the Flow through APIM](#requests)\n",
    "- [🔍 Analyze Application Insights requests](#portal)\n",
    "- [🗑️ Clean up resources](#clean)\n",
    "\n",
    "### Prerequisites\n",
    "- [Python 3.9 or later version](https://www.python.org/) installed\n",
    "- [Prompt flow](https://microsoft.github.io/promptflow/how-to-guides/installation/index.html#install-prompt-flow) installed\n",
    "- [VS Code](https://code.visualstudio.com/) installed with the [Jupyter notebook extension](https://marketplace.visualstudio.com/items?itemName=ms-toolsai.jupyter) enabled\n",
    "- [Azure CLI](https://learn.microsoft.com/cli/azure/install-azure-cli) installed\n",
    "- [An Azure Subscription](https://azure.microsoft.com/free/) with Contributor permissions\n",
    "- [Access granted to Azure OpenAI](https://aka.ms/oai/access)\n",
    "- [Sign in to Azure with Azure CLI](https://learn.microsoft.com/cli/azure/authenticate-azure-cli-interactively)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='0'></a>\n",
    "### 0️⃣ Initialize notebook variables\n",
    "\n",
    "- Resources will be suffixed by a unique string based on your subscription id.\n",
    "- Adjust the location parameters according your preferences and on the [product availability by Azure region.](https://azure.microsoft.com/explore/global-infrastructure/products-by-region/?cdn=disable&products=cognitive-services,api-management) \n",
    "- Adjust the OpenAI model and version according the [availability by region.](https://learn.microsoft.com/azure/ai-services/openai/concepts/models) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "metadata": {}
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import json\n",
    "import datetime\n",
    "import requests\n",
    "\n",
    "deployment_name = os.path.basename(os.path.dirname(globals()['__vsc_ipynb_file__']))\n",
    "resource_group_name = f\"lab-{deployment_name}\" # change the name to match your naming style\n",
    "resource_group_location = \"westeurope\"\n",
    "apim_resource_name = \"apim\"\n",
    "apim_resource_location = \"westeurope\"\n",
    "apim_resource_sku = \"Basicv2\"\n",
    "openai_resources = [ {\"name\": \"openai1\", \"location\": \"swedencentral\"}, {\"name\": \"openai2\", \"location\": \"francecentral\"} ] # list of OpenAI resources to deploy. Clear this list to use only the mock resources\n",
    "openai_resources_sku = \"S0\"\n",
    "openai_model_name = \"gpt-35-turbo\"\n",
    "openai_model_version = \"0613\"\n",
    "openai_deployment_name = \"gpt-35-turbo\"\n",
    "openai_api_version = \"2024-02-01\"\n",
    "openai_specification_url='https://raw.githubusercontent.com/Azure/azure-rest-api-specs/main/specification/cognitiveservices/data-plane/AzureOpenAI/inference/stable/' + openai_api_version + '/inference.json'\n",
    "openai_backend_pool = \"openai-backend-pool\"\n",
    "mock_backend_pool = \"mock-backend-pool\"\n",
    "mock_webapps = [ ]\n",
    "\n",
    "log_analytics_name = \"workspace\"\n",
    "app_insights_name = 'insights'\n",
    "\n",
    "ai_studio_hub_name = 'hub'\n",
    "ai_studio_project_name = 'project'\n",
    "storage_account_name = 'storage'\n",
    "keyvault_name = 'keyvault'\n",
    "container_registry_name = 'registry'\n",
    "\n",
    "containerapp_env_name = 'acaenv'\n",
    "containerapp_name = 'aca'\n",
    "\n",
    "flow_name = 'basic-chat' # the local folder must have the same name\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='1'></a>\n",
    "### 1️⃣ Create the Azure Resource Group\n",
    "All resources deployed in this lab will be created in the specified resource group. Skip this step if you want to use an existing resource group."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "resource_group_stdout = ! az group create --name {resource_group_name} --location {resource_group_location}\n",
    "if resource_group_stdout.n.startswith(\"ERROR\"):\n",
    "    print(resource_group_stdout)\n",
    "else:\n",
    "    print(\"✅ Azure Resource Group \", resource_group_name, \" created ⌚ \", datetime.datetime.now().time())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='2'></a>\n",
    "### 2️⃣ Create deployment using 🦾 Bicep\n",
    "\n",
    "This lab uses [Bicep](https://learn.microsoft.com/azure/azure-resource-manager/bicep/overview?tabs=bicep) to declarative define all the resources that will be deployed. Change the parameters or the [main.bicep](main.bicep) directly to try different configurations. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if len(openai_resources) > 0:\n",
    "    backend_id = openai_backend_pool if len(openai_resources) > 1 else openai_resources[0].get(\"name\")\n",
    "\n",
    "with open(\"policy.xml\", 'r') as policy_xml_file:\n",
    "    policy_template_xml = policy_xml_file.read()\n",
    "    policy_xml = policy_template_xml.replace(\"{backend-id}\", backend_id)\n",
    "    policy_xml_file.close()\n",
    "open(\"policy.xml\", 'w').write(policy_xml)\n",
    "\n",
    "bicep_parameters = {\n",
    "  \"$schema\": \"https://schema.management.azure.com/schemas/2019-04-01/deploymentParameters.json#\",\n",
    "  \"contentVersion\": \"1.0.0.0\",\n",
    "  \"parameters\": {\n",
    "    \"mockWebApps\": { \"value\": mock_webapps },\n",
    "    \"mockBackendPoolName\": { \"value\": mock_backend_pool },\n",
    "    \"openAIBackendPoolName\": { \"value\": openai_backend_pool },\n",
    "    \"openAIConfig\": { \"value\": openai_resources },\n",
    "    \"openAIDeploymentName\": { \"value\": openai_deployment_name },\n",
    "    \"openAISku\": { \"value\": openai_resources_sku },\n",
    "    \"openAIModelName\": { \"value\": openai_model_name },\n",
    "    \"openAIModelVersion\": { \"value\": openai_model_version },\n",
    "    \"openAIAPISpecURL\": { \"value\": openai_specification_url },\n",
    "    \"apimResourceName\": { \"value\": apim_resource_name},\n",
    "    \"apimResourceLocation\": { \"value\": apim_resource_location},\n",
    "    \"apimSku\": { \"value\": apim_resource_sku},\n",
    "    \"logAnalyticsName\": { \"value\": log_analytics_name },\n",
    "    \"applicationInsightsName\": { \"value\": app_insights_name },\n",
    "    \"aiStudioHubName\": { \"value\": ai_studio_hub_name },\n",
    "    \"aiStudioProjectName\": { \"value\": ai_studio_project_name },\n",
    "    \"storageAccountName\": { \"value\": storage_account_name }, \n",
    "    \"keyVaultName\": { \"value\": keyvault_name },\n",
    "    \"containerRegistryName\": { \"value\": container_registry_name },\n",
    "    \"containerAppEnvName\": { \"value\": containerapp_env_name },\n",
    "    \"containerAppName\": { \"value\": containerapp_name },\n",
    "    \"flowName\": { \"value\": flow_name }\n",
    "  }\n",
    "}\n",
    "with open('params.json', 'w') as bicep_parameters_file:\n",
    "    bicep_parameters_file.write(json.dumps(bicep_parameters))\n",
    "\n",
    "! az deployment group create --name {deployment_name} --resource-group {resource_group_name} --template-file \"main.bicep\" --parameters \"params.json\"\n",
    "\n",
    "open(\"policy.xml\", 'w').write(policy_template_xml)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='3'></a>\n",
    "### 3️⃣ Get the deployment outputs\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "deployment_stdout = ! az deployment group show --name {deployment_name} -g {resource_group_name} --query properties.outputs.apimSubscriptionKey.value -o tsv\n",
    "apim_subscription_key = deployment_stdout.n\n",
    "deployment_stdout = ! az deployment group show --name {deployment_name} -g {resource_group_name} --query properties.outputs.apimResourceGatewayURL.value -o tsv\n",
    "apim_resource_gateway_url = deployment_stdout.n\n",
    "print(\"👉🏻 API Gateway URL: \", apim_resource_gateway_url)\n",
    "\n",
    "deployment_stdout = ! az deployment group show --name {deployment_name} -g {resource_group_name} --query properties.outputs.logAnalyticsWorkspaceId.value -o tsv\n",
    "workspace_id = deployment_stdout.n\n",
    "print(\"👉🏻 Workspace ID: \", workspace_id)\n",
    "\n",
    "deployment_stdout = ! az deployment group show --name {deployment_name} -g {resource_group_name} --query properties.outputs.applicationInsightsAppId.value -o tsv\n",
    "app_id = deployment_stdout.n\n",
    "print(\"👉🏻 App ID: \", app_id)\n",
    "\n",
    "deployment_stdout = ! az deployment group show --name {deployment_name} -g {resource_group_name} --query properties.outputs.projectName.value -o tsv\n",
    "ml_project_name = deployment_stdout.n\n",
    "print(\"👉🏻 Project Name: \", ml_project_name)\n",
    "\n",
    "deployment_stdout = ! az deployment group show --name {deployment_name} -g {resource_group_name} --query properties.outputs.projectId.value -o tsv\n",
    "ml_project_id = deployment_stdout.n\n",
    "print(\"👉🏻 Project Id: \", ml_project_id)\n",
    "\n",
    "deployment_stdout = ! az deployment group show --name {deployment_name} -g {resource_group_name} --query properties.outputs.containerAppResourceName.value -o tsv\n",
    "containerapp_resource_name = deployment_stdout.n\n",
    "print(\"👉🏻 Container App Name: \", containerapp_resource_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='4'></a>\n",
    "### 4️⃣ Enable Prompt Flow tracing (optional)\n",
    "\n",
    "Prompt flow [tracing feature](https://microsoft.github.io/promptflow/how-to-guides/tracing/index.html) enables users to trace LLM calls, functions and even LLM frameworks. Besides, with promptflow[azure] installed, prompt flow can also log traces to an Azure ML workspace or Azure AI project, which makes it possible to share traces with your team members."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! pf config set trace.destination=azureml:/{ml_project_id}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='5'></a>\n",
    "### 5️⃣ Update the connection with APIM Gateway URL and APIM Subscription Key\n",
    "\n",
    "Notes:\n",
    "- In this lab we will use the local basic-chat flow. Explore other sample flows from this [repo](https://github.com/microsoft/promptflow/tree/main/examples/flows/chat).\n",
    "- Use the [VS Code Extension](https://marketplace.visualstudio.com/items?itemName=prompt-flow.prompt-flow) to build your own flows. Follow this [guide](https://microsoft.github.io/promptflow/how-to-guides/develop-a-dag-flow/index.html) to get started."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! pf connection create --file ./{flow_name}/azure_openai.yaml --set api_key={apim_subscription_key} api_base={apim_resource_gateway_url} --name open_ai_connection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='localtest'></a>\n",
    "### 🧪 Test the Flow locally\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! pf flow test --flow {flow_name} --inputs question=\"What's OpenAI?\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='6'></a>\n",
    "### 6️⃣ Create the Flow in the AI Studio Project (optional)\n",
    "\n",
    "This feature is useful for sharing the flow with other users and leveraging the capabilities of Azure AI Studio. Open the [docs](https://microsoft.github.io/promptflow/cloud/azureai/manage-flows.html#create-a-flow) to learn more."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! pfazure flow create --flow {flow_name} --set display_name={flow_name} type=chat description=\"Basic Chat Flow\" -g {resource_group_name} -w {ml_project_name}\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='7'></a>\n",
    "### 7️⃣ Build the Flow in the Docker format\n",
    "\n",
    "This step will generate a build folder with a Dockerfile and prompt flow artifacts. Open the [docs](https://microsoft.github.io/promptflow/how-to-guides/deploy-a-flow/deploy-using-docker.html) to learn more.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! pf flow build --source {flow_name} --output build --format docker"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='8'></a>\n",
    "### 8️⃣ Deploy the Flow to an Azure Container App\n",
    "\n",
    "Note ⚠️: If the command fails inside this notebook, please run it in a separate terminal.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! echo az containerapp up --name {containerapp_resource_name} --source build --browse\n",
    "! az containerapp up --name {containerapp_resource_name} --source build"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='requests'></a>\n",
    "### 🧪 Test the Flow through APIM\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import uuid\n",
    "UUID = str(uuid.uuid4())\n",
    "print(f\"Request-Id: {UUID} - use this ID to trace the requests in Azure Application Insights.\")\n",
    "\n",
    "response = requests.post(apim_resource_gateway_url + \"/\" + flow_name + \"/score\", headers = {'api-key':apim_subscription_key, \"Request-Id\": UUID}, json = {\"question\": \"Which is the biggest football club in Portugal?\"})\n",
    "if (response.status_code == 200):\n",
    "    data = json.loads(response.text)\n",
    "    print(\"💬 \", data.get(\"answer\"))\n",
    "else:\n",
    "    print(response.text)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='portal'></a>\n",
    "### 🔍 Analyze Application Insights transactions\n",
    "\n",
    "Open the Application Insights resource in the Azure Portal and search for the Request-Id generated in the previous step.  \n",
    "You can also search for the prompt flow execution, although end-to-end correlation is not yet available.\n",
    "\n",
    "![result](result.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='clean'></a>\n",
    "### 🗑️ Clean up resources\n",
    "\n",
    "When you're finished with the lab, you should remove all your deployed resources from Azure to avoid extra charges and keep your Azure subscription uncluttered.\n",
    "Use the [clean-up-resources notebook](clean-up-resources.ipynb) for that."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}