diff --git a/scenarios/evaluate/evaluate_online/README.md b/scenarios/evaluate/evaluate_online/README.md new file mode 100644 index 0000000..f7af45a --- /dev/null +++ b/scenarios/evaluate/evaluate_online/README.md @@ -0,0 +1,41 @@ +--- +page_type: sample +languages: +- python +products: +- ai-services +- azure-openai +description: Evaluating online +--- + +## Evaluating in the cloud on a schedule + +### Overview + +This tutorial provides a step-by-step guide on how to evaluate generative AI or LLMs on a scheduling using online evaluation. + +### Objective + +The main objective of this tutorial is to help users understand the process of evaluating model remotely in the cloud by triggering an evaluation. This type of evaluation can be used for monitoring LLMs and Generative AI that has been deployed. By the end of this tutorial, you should be able to: + + - Learn about evaluations + - Evaluate LLM using various evaluators from Azure AI Evaluations SDK online in the cloud. + +### Note +All evaluators supported by [Azure AI Evaluation](https://learn.microsoft.com/en-us/azure/ai-studio/concepts/evaluation-metrics-built-in?tabs=warning) are supported by Online Evaluation. For updated documentation, please see [Online Evaluation documentation](https://aka.ms/GenAIMonitoringDoc). + +#### Region Support for Evaluations + +| Region | Hate and Unfairness, Sexual, Violent, Self-Harm, XPIA | Groundedness Pro | Protected Material | +| - | - | - | - | +| UK South | Will be deprecated 12/1/24 | no | no | +| East US 2 | yes | yes | yes | +| Sweden Central | yes | yes | no | +| US North Central | yes | no | no | +| France Central | yes | no | no | +| Switzerland West | yes | no | no | + +### Programming Languages + - Python + +### Estimated Runtime: 30 mins \ No newline at end of file diff --git a/scenarios/evaluate/evaluate_online/evaluate_online.ipynb b/scenarios/evaluate/evaluate_online/evaluate_online.ipynb new file mode 100644 index 0000000..995f64f --- /dev/null +++ b/scenarios/evaluate/evaluate_online/evaluate_online.ipynb @@ -0,0 +1,248 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Online Evaluations: Evaluating in the Cloud on a Schedule\n", + "\n", + "## Objective\n", + "\n", + "This tutorial provides a step-by-step guide on how to evaluate data generated by LLMs online on a schedule. \n", + "\n", + "This tutorial uses the following Azure AI services:\n", + "\n", + "- [Azure AI Safety Evaluation](https://aka.ms/azureaistudiosafetyeval)\n", + "- [azure-ai-evaluation](https://learn.microsoft.com/en-us/azure/ai-studio/how-to/develop/evaluate-sdk)\n", + "\n", + "## Time\n", + "\n", + "You should expect to spend 30 minutes running this sample. \n", + "\n", + "## About this example\n", + "\n", + "This example demonstrates the online evaluation of a LLM. It is important to have access to AzureOpenAI credentials and an AzureAI project. This example demonstrates: \n", + "\n", + "- Recurring, Online Evaluation (to be used to monitor LLMs once they are deployed)\n", + "\n", + "## Before you begin\n", + "### Prerequesite\n", + "- Configure resources to support Online Evaluation as per [Online Evaluation documentation](https://aka.ms/GenAIMonitoringDoc)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "%pip install -U azure-identity\n", + "%pip install -U azure-ai-project\n", + "%pip install -U azure-ai-evaluation" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from azure.ai.project import AIProjectClient\n", + "from azure.identity import DefaultAzureCredential\n", + "from azure.ai.project.models import (\n", + " ApplicationInsightsConfiguration,\n", + " EvaluatorConfiguration,\n", + " ConnectionType,\n", + " EvaluationSchedule,\n", + " RecurrenceTrigger,\n", + ")\n", + "from azure.ai.evaluation import F1ScoreEvaluator, ViolenceEvaluator" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Connect to your Azure Open AI deployment\n", + "To evaluate your LLM-generated data remotely in the cloud, we must connect to your Azure Open AI deployment. This deployment must be a GPT model which supports `chat completion`, such as `gpt-4`. To see the proper value for `conn_str`, navigate to the connection string at the \"Project Overview\" page for your Azure AI project. " + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "project_client = AIProjectClient.from_connection_string(\n", + " credential=DefaultAzureCredential(),\n", + " conn_str=\"\", # At the moment, it should be in the format \".api.azureml.ms;;;\" Ex: eastus2.api.azureml.ms;xxxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxxxxx;rg-sample;sample-project-eastus2\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Please see [Online Evaluation documentation](https://aka.ms/GenAIMonitoringDoc) for configuration of Application Insights. `service_name` is a unique name you provide to define your Generative AI application and identify it within your Application Insights resource. This property will be logged in the `traces` table in Application Insights and can be found in the `customDimensions[\"service.name\"]` field. `evaluation_name` is a unique name you provide for your Online Evaluation schedule. " + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Your Application Insights resource ID\n", + "# At the moment, it should be something in the format \"/subscriptions//resourceGroups//providers/Microsoft.Insights/components/\"\"\n", + "app_insights_resource_id = \"\"\n", + "\n", + "# Name of your generative AI application (will be available in trace data in Application Insights)\n", + "service_name = \"\"\n", + "\n", + "# Name of your online evaluation schedule\n", + "evaluation_name = \"\"" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Below is the Kusto Query Language (KQL) query to query data from Application Insights resource. This query is compatible with data logged by the Azure AI Inferencing Tracing SDK (linked in [documentation](https://aka.ms/GenAIMonitoringDoc)). You can modify it depending on your data schema. The KQL query must output several columns: `operation_ID`, `operation_ParentID`, and `gen_ai_response_id`. You can choose which other columns to output as required by the evaluators you are using." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "kusto_query = 'let gen_ai_spans=(dependencies | where isnotnull(customDimensions[\"gen_ai.system\"]) | extend response_id = tostring(customDimensions[\"gen_ai.response.id\"]) | project id, operation_Id, operation_ParentId, timestamp, response_id); let gen_ai_events=(traces | where message in (\"gen_ai.choice\", \"gen_ai.user.message\", \"gen_ai.system.message\") or tostring(customDimensions[\"event.name\"]) in (\"gen_ai.choice\", \"gen_ai.user.message\", \"gen_ai.system.message\") | project id= operation_ParentId, operation_Id, operation_ParentId, user_input = iff(message == \"gen_ai.user.message\" or tostring(customDimensions[\"event.name\"]) == \"gen_ai.user.message\", parse_json(iff(message == \"gen_ai.user.message\", tostring(customDimensions[\"gen_ai.event.content\"]), message)).content, \"\"), system = iff(message == \"gen_ai.system.message\" or tostring(customDimensions[\"event.name\"]) == \"gen_ai.system.message\", parse_json(iff(message == \"gen_ai.system.message\", tostring(customDimensions[\"gen_ai.event.content\"]), message)).content, \"\"), llm_response = iff(message == \"gen_ai.choice\", parse_json(tostring(parse_json(tostring(customDimensions[\"gen_ai.event.content\"])).message)).content, iff(tostring(customDimensions[\"event.name\"]) == \"gen_ai.choice\", parse_json(parse_json(message).message).content, \"\")) | summarize operation_ParentId = any(operation_ParentId), Input = maxif(user_input, user_input != \"\"), System = maxif(system, system != \"\"), Output = maxif(llm_response, llm_response != \"\") by operation_Id, id); gen_ai_spans | join kind=inner (gen_ai_events) on id, operation_Id | project Input, System, Output, operation_Id, operation_ParentId, gen_ai_response_id = response_id'\n", + "\n", + "# AzureMSIClientId is the clientID of the User-assigned managed identity created during set-up - see documentation for how to find it\n", + "properties = {\"AzureMSIClientId\": \"your_client_id\"}" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Connect to your Application Insights resource\n", + "app_insights_config = ApplicationInsightsConfiguration(\n", + " resource_id=app_insights_resource_id, query=kusto_query, service_name=service_name\n", + ")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Connect to your AOAI resource, you must use an AOAI GPT model\n", + "deployment_name = \"gpt-4\"\n", + "api_version = \"2024-06-01\"\n", + "default_connection = project_client.connections.get_default(connection_type=ConnectionType.AZURE_OPEN_AI)\n", + "model_config = default_connection.to_evaluator_model_config(deployment_name=deployment_name, api_version=api_version)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Configure Evaluators to Run\n", + "The code below demonstrates how to configure the evaluators you want to run. In this example, we use the `F1ScoreEvaluator`, `RelevanceEvaluator` and the `ViolenceEvaluator`, but all evaluators supported by [Azure AI Evaluation](https://learn.microsoft.com/en-us/azure/ai-studio/concepts/evaluation-metrics-built-in?tabs=warning) are supported by Online Evaluation and can be configured here. You can either import the classes from the SDK and reference them with the `.id` property, or you can find the fully formed `id` of the evaluator in the AI Studio registry of evaluators, and use it here. " + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# id for each evaluator can be found in your AI Studio registry - please see documentation for more information\n", + "# init_params is the configuration for the model to use to perform the evaluation\n", + "# data_mapping is used to map the output columns of your query to the names required by the evaluator\n", + "evaluators = {\n", + " \"f1_score\": EvaluatorConfiguration(\n", + " id=F1ScoreEvaluator.id,\n", + " ),\n", + " \"relevance\": EvaluatorConfiguration(\n", + " id=\"azureml://registries/azureml-staging/models/Relevance-Evaluator/versions/4\",\n", + " init_params={\"model_config\": model_config},\n", + " data_mapping={\"query\": \"${data.Input}\", \"response\": \"${data.Output}\"},\n", + " ),\n", + " \"violence\": EvaluatorConfiguration(\n", + " id=ViolenceEvaluator.id,\n", + " init_params={\"azure_ai_project\": project_client.scope},\n", + " data_mapping={\"query\": \"${data.Input}\", \"response\": \"${data.Output}\"},\n", + " ),\n", + "}" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Evaluate in the Cloud on a Schedule with Online Evaluation\n", + "\n", + "You can configure the `RecurrenceTrigger` based on the class definition [here](https://learn.microsoft.com/en-us/python/api/azure-ai-ml/azure.ai.ml.entities.recurrencetrigger?view=azure-python)." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Frequency to run the schedule\n", + "recurrence_trigger = RecurrenceTrigger(frequency=\"day\", interval=1)\n", + "\n", + "# Configure the online evaluation schedule\n", + "evaluation_schedule = EvaluationSchedule(\n", + " data=app_insights_config,\n", + " evaluators=evaluators,\n", + " trigger=recurrence_trigger,\n", + " description=f\"{service_name} evaluation schedule\",\n", + " properties=properties,\n", + ")\n", + "\n", + "# Create the online evaluation schedule\n", + "created_evaluation_schedule = project_client.evaluations.create_or_replace_schedule(service_name, evaluation_schedule)\n", + "print(\n", + " f\"Successfully submitted the online evaluation schedule creation request - {created_evaluation_schedule.name}, currently in {created_evaluation_schedule.provisioning_state} state.\"\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Next steps \n", + "\n", + "Navigate to the \"Tracing\" tab in [Azure AI Studio](https://ai.azure.com/) to view your logged trace data alongside the evaluations produced by the Online Evaluation schedule. You can use the reference link provided in the \"Tracing\" tab to navigate to a comprehensive workbook in Application Insights for more details on how your application is performing. " + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "azureai-samples313", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3" + } + }, + "nbformat": 4, + "nbformat_minor": 2 +} diff --git a/scenarios/evaluate/evaluate_remotely/README.md b/scenarios/evaluate/evaluate_remotely/README.md new file mode 100644 index 0000000..190f49c --- /dev/null +++ b/scenarios/evaluate/evaluate_remotely/README.md @@ -0,0 +1,41 @@ +--- +page_type: sample +languages: +- python +products: +- ai-services +- azure-openai +description: Evaluating remotely +--- + +## Evaluating in the cloud + +### Overview + +This tutorial provides a step-by-step guide on how to evaluate generative AI or LLMs remotely using a triggered evaluation. + +### Objective + +The main objective of this tutorial is to help users understand the process of evaluating model remotely in the cloud by triggering an evaluation. This type of evaluation can be used for pre-deployment testing. By the end of this tutorial, you should be able to: + + - Learn about evaluations + - Evaluate LLM using various evaluators from Azure AI Evaluations SDK remotely in the cloud. + +### Note +Remote evaluations do not support `Retrieval-Evaluator`, `ContentSafetyEvaluator`, and `QAEvaluator`. + +#### Region Support for Evaluations + +| Region | Hate and Unfairness, Sexual, Violent, Self-Harm, XPIA | Groundedness Pro | Protected Material | +| - | - | - | - | +| UK South | Will be deprecated 12/1/24 | no | no | +| East US 2 | yes | yes | yes | +| Sweden Central | yes | yes | no | +| US North Central | yes | no | no | +| France Central | yes | no | no | +| Switzerland West | yes | no | no | + +### Programming Languages + - Python + +### Estimated Runtime: 20 mins \ No newline at end of file diff --git a/scenarios/evaluate/evaluate_remotely/evaluate_remotely.ipynb b/scenarios/evaluate/evaluate_remotely/evaluate_remotely.ipynb new file mode 100644 index 0000000..2d37962 --- /dev/null +++ b/scenarios/evaluate/evaluate_remotely/evaluate_remotely.ipynb @@ -0,0 +1,222 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Remote Evaluations: Evaluating in the Cloud \n", + "\n", + "## Objective\n", + "\n", + "This tutorial provides a step-by-step guide on how to evaluate data generated by LLMs remotely in the cloud. \n", + "\n", + "This tutorial uses the following Azure AI services:\n", + "\n", + "- [Azure AI Safety Evaluation](https://aka.ms/azureaistudiosafetyeval)\n", + "- [azure-ai-evaluation](https://learn.microsoft.com/en-us/azure/ai-studio/how-to/develop/evaluate-sdk)\n", + "\n", + "## Time\n", + "\n", + "You should expect to spend 20 minutes running this sample. \n", + "\n", + "## About this example\n", + "\n", + "This example demonstrates the remote evaluation of query and response pairs that were generated by a LLM. It is important to have access to AzureOpenAI credentials and an AzureAI project. **To create data to use in your own evaluation, learn more [here](https://learn.microsoft.com/en-us/azure/ai-studio/how-to/develop/simulator-interaction-data)** . This example demonstrates: \n", + "\n", + "- Single-instance, triggered Remote Evaluation (to be used for pre-deployment evaluation of LLMs)\n", + "\n", + "## Before you begin\n", + "### Prerequesite\n", + "- [Have an online deployment on Azure Open AI studio supporting `chat completion` such as `gpt-4`](https://learn.microsoft.com/en-us/azure/machine-learning/concept-endpoints-online?view=azureml-api-2)\n", + "- You also might want to evaluate data generated by your LLM, to see how to generate data to be evaluated using the Azure AI Evaluation SDK, see our samples on simulation \n", + "\n", + "### Installation\n", + "\n", + "Install the following packages required to execute this notebook. " + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "%pip install -U azure-identity\n", + "%pip install -U azure-ai-project\n", + "%pip install -U azure-ai-evaluation" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from azure.ai.project import AIProjectClient\n", + "from azure.identity import DefaultAzureCredential\n", + "from azure.ai.project.models import (\n", + " Evaluation,\n", + " Dataset,\n", + " EvaluatorConfiguration,\n", + " ConnectionType,\n", + ")\n", + "from azure.ai.evaluation import F1ScoreEvaluator, ViolenceEvaluator" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Connect to your Azure Open AI deployment\n", + "To evaluate your LLM-generated data remotely in the cloud, we must connect to your Azure Open AI deployment. This deployment must be a GPT model which supports `chat completion`, such as `gpt-4`. To see the proper value for `conn_str`, navigate to the connection string at the \"Project Overview\" page for your Azure AI project. " + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "project_client = AIProjectClient.from_connection_string(\n", + " credential=DefaultAzureCredential(),\n", + " conn_str=\"\", # At the moment, it should be in the format \".api.azureml.ms;;;\" Ex: eastus2.api.azureml.ms;xxxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxxxxx;rg-sample;sample-project-eastus2\n", + ")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Connect to your AOAI resource, you must use an AOAI GPT model\n", + "deployment_name = \"gpt-4\"\n", + "api_version = \"2024-06-01\"\n", + "default_connection = project_client.connections.get_default(connection_type=ConnectionType.AZURE_OPEN_AI)\n", + "model_config = default_connection.to_evaluator_model_config(deployment_name=deployment_name, api_version=api_version)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Data\n", + "The following code demonstrates how to upload the data for evaluation to your Azure AI project. Below we use `evaluate_test_data.jsonl` which exemplifies LLM-generated data in the query-response format expected by the Azure AI Evaluation SDK. For your use case, you should upload data in the same format, which can be generated using the [`Simulator`](https://learn.microsoft.com/en-us/azure/ai-studio/how-to/develop/simulator-interaction-data) from Azure AI Evaluation SDK. \n", + "\n", + "Alternatively, if you already have an existing dataset for evaluation, you can use that by finding the link to your dataset in your [registry](https://ml.azure.com/registries) or find the dataset ID." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# # Upload data for evaluation\n", + "data_id, _ = project_client.upload_file(\"./evaluate_test_data.jsonl\")\n", + "# data_id = \"azureml://registries//data//versions/\"\n", + "# To use an existing dataset, replace the above line with the following line\n", + "# data_id = \"\"" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Configure Evaluators to Run\n", + "The code below demonstrates how to configure the evaluators you want to run. In this example, we use the `F1ScoreEvaluator`, `RelevanceEvaluator` and the `ViolenceEvaluator`, but all evaluators supported by [Azure AI Evaluation](https://learn.microsoft.com/en-us/azure/ai-studio/concepts/evaluation-metrics-built-in?tabs=warning) are supported by Online Evaluation and can be configured here. You can either import the classes from the SDK and reference them with the `.id` property, or you can find the fully formed `id` of the evaluator in the AI Studio registry of evaluators, and use it here. " + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# id for each evaluator can be found in your AI Studio registry - please see documentation for more information\n", + "# init_params is the configuration for the model to use to perform the evaluation\n", + "# data_mapping is used to map the output columns of your query to the names required by the evaluator\n", + "evaluators = {\n", + " \"f1_score\": EvaluatorConfiguration(\n", + " id=F1ScoreEvaluator.id,\n", + " ),\n", + " \"relevance\": EvaluatorConfiguration(\n", + " id=\"azureml://registries/azureml-staging/models/Relevance-Evaluator/versions/4\",\n", + " init_params={\"model_config\": model_config},\n", + " data_mapping={\"query\": \"${data.Input}\", \"response\": \"${data.Output}\"},\n", + " ),\n", + " \"violence\": EvaluatorConfiguration(\n", + " id=ViolenceEvaluator.id,\n", + " init_params={\"azure_ai_project\": project_client.scope},\n", + " data_mapping={\"query\": \"${data.Input}\", \"response\": \"${data.Output}\"},\n", + " ),\n", + "}" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Evaluate in the Cloud with Remote Evaluation\n", + "Below we demonstrate how to trigger a single-instance Remote Evaluation in the Cloud. This can be used for pre-deployment testing of an LLM. \n", + " \n", + "Here we pass in the `data_id` we would like to use for the evaluation. " + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "evaluation = Evaluation(\n", + " display_name=\"Remote Evaluation\",\n", + " description=\"Evaluation of dataset\",\n", + " data=Dataset(id=data_id),\n", + " evaluators=evaluators,\n", + ")\n", + "\n", + "# Create evaluation\n", + "evaluation_response = project_client.evaluations.create(\n", + " evaluation=evaluation,\n", + ")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Get evaluation\n", + "get_evaluation_response = project_client.evaluations.get(evaluation_response.id)\n", + "\n", + "print(\"----------------------------------------------------------------\")\n", + "print(\"Created evaluation, evaluation ID: \", get_evaluation_response.id)\n", + "print(\"Evaluation status: \", get_evaluation_response.status)\n", + "print(\"AI Foundry Portal URI: \", get_evaluation_response.properties[\"AiStudioEvaluationUri\"])\n", + "print(\"----------------------------------------------------------------\")" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "azureai-samples313", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3" + } + }, + "nbformat": 4, + "nbformat_minor": 2 +} diff --git a/scenarios/evaluate/evaluate_remotely/evaluate_test_data.jsonl b/scenarios/evaluate/evaluate_remotely/evaluate_test_data.jsonl new file mode 100644 index 0000000..ddca894 --- /dev/null +++ b/scenarios/evaluate/evaluate_remotely/evaluate_test_data.jsonl @@ -0,0 +1,3 @@ +{"query":"How do you create a run?","context":"AML API only","response":"To create a run using the Azure Machine Learning API, you first need to create an Experiment. Once you have an experiment, you can create a Run object that is associated with that experiment. Here is some Python code that demonstrates this process:\n\n```\nfrom azureml.core import Experiment, Run\nfrom azureml.core.workspace import Workspace\n\n# Define workspace and experiment\nws = Workspace.from_config()\nexp = Experiment(workspace=ws, name='my_experiment')\n\n# Create a new run\nrun = exp.start_logging()\n```\n\nIn this code, the `from_config()` method reads the configuration file that you created when you set up your Azure Machine Learning workspace. The `Experiment` constructor creates an Experiment object that is associated with your workspace, and the `start_logging()` method creates a new Run object that is associated with the Experiment. Now you can use the `run` object to log metrics, upload files, and track other information related to your machine learning experiment.", "ground_truth":"Paris is the capital of France."} +{"query":"How do you log a model?","context":"Logging can be done using any OSS Sdk","response":"There are a few ways to log models in Azure Machine Learning. \n\nOne way is to use the `register_model()` method of the `Run` object. The `register_model()` method logs a model file in the Azure Machine Learning service workspace and makes it available for deployment. Here's an example:\n\n```python\nfrom azureml.core import Model\n\nmodel_path = '.\/outputs\/my_model.pkl'\nmodel = Model.register(workspace=ws, model_path=model_path, model_name='my_model')\n```\n\nThis code registers the model file located at `model_path` to the Azure Machine Learning service workspace with the name `my_model`. \n\nAnother way to log a model is to save it as an output of a `Run`. If your model generation code is part of a script or Jupyter notebook that runs as an Azure Machine Learning experiment, you can save the model file as an output of the `Run` object. Here's an example:\n\n```python\nfrom sklearn.linear_model import LogisticRegression\nfrom azureml.core.run import Run\n\n# Initialize a run object\nrun = Run.get_context()\n\n# Train your model\nX_train, y_train = ...\nlog_reg = LogisticRegression().fit(X_train, y_train)\n\n# Save the model to the Run object's outputs directory\nmodel_path = 'outputs\/model.pkl'\njoblib.dump(value=log_reg, filename=model_path)\n\n# Log the model as a run artifact\nrun.upload_file(name=model_path, path_or_stream=model_path)\n```\n\nIn this code, `Run.get_context()` retrieves the current run context object, which you can use to track metadata and metrics for the run. After training your model, you can use `joblib.dump()` to save the model to a file, and then log the file as an artifact of the run using `run.upload_file()`.","ground_truth":"Paris is the capital of France."} +{"query":"What is the capital of France?","context":"France is in Europe","response":"Paris is the capital of France.", "ground_truth":"Paris is the capital of France."}