docs: anthropic tools tracing tutorial and fixture

src/phoenix/trace/fixtures.py

@@ -266,6 +266,13 @@ class TracesFixture:
     file_name="vision_fixture_trace_datasets.parquet",
 )
 
+anthropic_tools_fixture = TracesFixture(
+    name="anthropic_tools",
+    project_name="anthropic_tools",
+    description="Anthropic tools traces",
+    file_name="anthropic_tools.parquet",
+)
+
 random_fixture = TracesFixture(
     name="random",
     project_name="demo_random",
@@ -288,6 +295,7 @@ class TracesFixture:
     langchain_qa_with_sources_fixture,
     llama_index_calculator_agent_fixture,
     vision_fixture,
+    anthropic_tools_fixture,
 ]
 
 NAME_TO_TRACES_FIXTURE: Dict[str, TracesFixture] = {

tutorials/tracing/anthropic_tracing_tutorial.ipynb

@@ -0,0 +1,340 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "<center>\n",
+    "    <p style=\"text-align:center\">\n",
+    "        <img alt=\"phoenix logo\" src=\"https://storage.googleapis.com/arize-phoenix-assets/assets/phoenix-logo-light.svg\" width=\"200\"/>\n",
+    "        <br>\n",
+    "        <a href=\"https://docs.arize.com/phoenix/\">Docs</a>\n",
+    "        |\n",
+    "        <a href=\"https://github.com/Arize-ai/phoenix\">GitHub</a>\n",
+    "        |\n",
+    "        <a href=\"https://join.slack.com/t/arize-ai/shared_invite/zt-1px8dcmlf-fmThhDFD_V_48oU7ALan4Q\">Community</a>\n",
+    "    </p>\n",
+    "</center>\n",
+    "<h1 align=\"center\">Tracing and Evaluating a Structured Data Extraction Service</h1>\n",
+    "\n",
+    "In this tutorial, you will:\n",
+    "\n",
+    "- Use Anthropic's [tool calling](https://docs.anthropic.com/en/docs/build-with-claude/tool-use) to perform structured data extraction: the task of transforming unstructured input (e.g., user requests in natural language) into structured format (e.g., tabular format),\n",
+    "- Instrument your Anthropic client to record trace data in [OpenInference tracing](https://github.com/Arize-ai/open-inference-spec/blob/main/trace/spec/traces.md) format,\n",
+    "- Inspect the traces and spans of your application to visualize your trace data,\n",
+    "- Export your trace data to run an evaluation on the quality of your structured extractions.\n",
+    "\n",
+    "## Background\n",
+    "\n",
+    "One powerful feature of the Anthropic API is tool use (function calling), wherein a user describes the signature and arguments of one or more functions to the Anthropic API via a JSON Schema and natural language descriptions, and the LLM decides when to call each function and provides argument values depending on the context of the conversation. In addition to its primary purpose of integrating function inputs and outputs into a sequence of chat messages, function calling is also useful for structured data extraction, since you can specify a \"function\" that describes the desired format of your structured output. Structured data extraction is useful for a variety of purposes, including ETL or as input to another machine learning model such as a recommender system.\n",
+    "\n",
+    "While it's possible to produce structured output without using function calling via careful prompting, function calling is more reliable at producing output that conforms to a particular format. For more details on Anthropic's function calling API, see the [Anthropic documentation](https://docs.anthropic.com/en/home).\n",
+    "\n",
+    "Let's get started!\n",
+    "\n",
+    "ℹ️ This notebook requires an Anthropic API key."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 1. Install Dependencies and Import Libraries\n",
+    "\n",
+    "Install dependencies."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "!pip install anthropic arize-phoenix jsonschema openinference-instrumentation-anthropic"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Import libraries."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "from getpass import getpass\n",
+    "from typing import Any, Dict\n",
+    "\n",
+    "import pandas as pd\n",
+    "\n",
+    "import phoenix as px\n",
+    "\n",
+    "pd.set_option(\"display.max_colwidth\", None)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 2. Configure Your Anthropic API Key and Instantiate Your Anthropic Client\n",
+    "\n",
+    "Set your Anthropic API key if it is not already set as an environment variable."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import anthropic\n",
+    "\n",
+    "if not (anthropic_api_key := os.getenv(\"ANTHROPIC_API_KEY\")):\n",
+    "    anthropic_api_key = getpass(\"🔑 Enter your Anthropic API key: \")\n",
+    "client = anthropic.Anthropic(api_key=anthropic_api_key)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 3. Instrument Your Anthropic Client\n",
+    "\n",
+    "Instrument your Anthropic client with a tracer that emits telemetry data in OpenInference format. [OpenInference](https://arize-ai.github.io/open-inference-spec/trace/) is an open standard for capturing and storing LLM application traces that enables LLM applications to seamlessly integrate with LLM observability solutions such as Phoenix."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from openinference.instrumentation.anthropic import AnthropicInstrumentor\n",
+    "\n",
+    "from phoenix.otel import register\n",
+    "\n",
+    "tracer_provider = register(project_name=\"anthropic-tools\")\n",
+    "AnthropicInstrumentor().instrument(tracer_provider=tracer_provider)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 4. Run Phoenix in the Background\n",
+    "\n",
+    "Launch Phoenix as a background session to collect the trace data emitted by your instrumented OpenAI client."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "(session := px.launch_app()).view()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 5. Extract Your Structured Data\n",
+    "\n",
+    "We'll extract structured data from the following list of ten travel requests."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "travel_requests = [\n",
+    "    \"Can you recommend a luxury hotel in Tokyo with a view of Mount Fuji for a romantic honeymoon?\",\n",
+    "    \"I'm looking for a mid-range hotel in London with easy access to public transportation for a solo backpacking trip. Any suggestions?\",\n",
+    "    \"I need a budget-friendly hotel in San Francisco close to the Golden Gate Bridge for a family vacation. What do you recommend?\",\n",
+    "    \"Can you help me find a boutique hotel in New York City with a rooftop bar for a cultural exploration trip?\",\n",
+    "    \"I'm planning a business trip to Tokyo and I need a hotel near the financial district. What options are available?\",\n",
+    "    \"I'm traveling to London for a solo vacation and I want to stay in a trendy neighborhood with great shopping and dining options. Any recommendations for hotels?\",\n",
+    "    \"I'm searching for a luxury beachfront resort in San Francisco for a relaxing family vacation. Can you suggest any options?\",\n",
+    "    \"I need a mid-range hotel in New York City with a fitness center and conference facilities for a business trip. Any suggestions?\",\n",
+    "    \"I'm looking for a budget-friendly hotel in Tokyo with easy access to public transportation for a backpacking trip. What do you recommend?\",\n",
+    "    \"I'm planning a honeymoon in London and I want a luxurious hotel with a spa and romantic atmosphere. Can you suggest some options?\",\n",
+    "]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The Anthropic API uses [JSON Schema](https://json-schema.org/) and natural language descriptions to specify the signature of a function to call. In this case, we'll describe a function to record the following attributes of the unstructured text input:\n",
+    "\n",
+    "- **location:** The desired destination,\n",
+    "- **budget_level:** A categorical budget preference,\n",
+    "- **purpose:** The purpose of the trip.\n",
+    "\n",
+    "The use of JSON Schema enables us to define the type of each field in the output and even enumerate valid values in the case of categorical outputs. Anthropic function calling can thus be used for tasks that might previously have been performed by named-entity recognition (NER) and/ or classification models."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "tool_schema = {\n",
+    "    \"name\": \"record_travel_request_attributes\",\n",
+    "    \"description\": \"Records the attributes of a travel request\",\n",
+    "    \"input_schema\": {\n",
+    "        \"type\": \"object\",\n",
+    "        \"properties\": {\n",
+    "            \"location\": {\n",
+    "                \"type\": \"string\",\n",
+    "                \"description\": 'The desired destination location. Use city, state, and country format when possible. If no destination is provided, return \"not_stated\".',\n",
+    "            },\n",
+    "            \"budget_level\": {\n",
+    "                \"type\": \"string\",\n",
+    "                \"enum\": [\"low\", \"medium\", \"high\", \"not_stated\"],\n",
+    "                \"description\": 'The desired budget level. If no budget level is provided, return \"not_stated\".',\n",
+    "            },\n",
+    "            \"purpose\": {\n",
+    "                \"type\": \"string\",\n",
+    "                \"enum\": [\"business\", \"pleasure\", \"other\", \"not_stated\"],\n",
+    "                \"description\": 'The purpose of the trip. If no purpose is provided, return \"not_stated\".',\n",
+    "            },\n",
+    "        },\n",
+    "        \"required\": [\"location\", \"budget_level\", \"purpose\"],\n",
+    "    },\n",
+    "}\n",
+    "\n",
+    "system_message = (\n",
+    "    \"You are an assistant that parses and records the attributes of a user's travel request.\"\n",
+    ")\n",
+    "\n",
+    "\n",
+    "def extract_raw_travel_request_attributes_string(\n",
+    "    travel_request: str,\n",
+    "    tool_schema: Dict[str, Any],\n",
+    "    system_message: str,\n",
+    "    client: client,\n",
+    "    model: str = \"claude-3-5-sonnet-20240620\",\n",
+    ") -> str:\n",
+    "    response = client.messages.create(\n",
+    "        model=model,\n",
+    "        max_tokens=1024,\n",
+    "        messages=[\n",
+    "            {\"role\": \"user\", \"content\": travel_request},\n",
+    "        ],\n",
+    "        system=system_message,\n",
+    "        tools=[tool_schema],\n",
+    "        # By default, the LLM will choose whether or not to call a function given the conversation context.\n",
+    "        # The line below forces the LLM to call the function so that the output conforms to the schema.\n",
+    "        tool_choice={\"type\": \"tool\", \"name\": \"record_travel_request_attributes\"},\n",
+    "    )\n",
+    "    return response.content[0].input"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Run the extractions."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "raw_travel_attributes_column = []\n",
+    "for travel_request in travel_requests:\n",
+    "    print(\"Travel request:\")\n",
+    "    print(\"==============\")\n",
+    "    print(travel_request)\n",
+    "    print()\n",
+    "    raw_travel_attributes = extract_raw_travel_request_attributes_string(\n",
+    "        travel_request, tool_schema, system_message, client\n",
+    "    )\n",
+    "    raw_travel_attributes_column.append(raw_travel_attributes)\n",
+    "    print(\"Raw Travel Attributes:\")\n",
+    "    print(\"=====================\")\n",
+    "    print(raw_travel_attributes)\n",
+    "    print()\n",
+    "    print()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Your trace data should appear in real-time in the Phoenix UI."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "print(f\"🔥🐦 Open the Phoenix UI if you haven't already: {session.url}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# 6. Export and Evaluate Your Trace Data\n",
+    "\n",
+    "Your OpenInference trace data is collected by Phoenix and can be exported to a pandas dataframe for further analysis and evaluation."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "trace_ds = px.Client().get_trace_dataset(project_name=\"anthropic-tools\")\n",
+    "trace_ds.get_spans_dataframe().head()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "trace_ds.save()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 7. Recap\n",
+    "\n",
+    "Congrats! In this tutorial, you:\n",
+    "\n",
+    "- Built a service to perform structured data extraction on unstructured text using Anthropic function calling\n",
+    "- Instrumented your service with an OpenInference tracer\n",
+    "- Examined your telemetry data in Phoenix\n",
+    "\n",
+    "Check back soon for tips on evaluating the performance of your service using LLM evals."
+   ]
+  }
+ ],
+ "metadata": {
+  "language_info": {
+   "name": "python"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

tutorials/tracing/llama_index_openai_agent_tracing_tutorial.ipynb

@@ -315,22 +315,8 @@
   }
  ],
  "metadata": {
-  "kernelspec": {
-   "display_name": "envName=phoenix",
-   "language": "python",
-   "name": "python3"
-  },
   "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.10.14"
+   "name": "python"
   }
  },
  "nbformat": 4,