2-langchain-structured-output-evaluation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluating structured outputs in LangChain\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_core.prompts import ChatPromptTemplate\n",
    "from langchain_anthropic import ChatAnthropic\n",
    "from langchain_ollama import ChatOllama\n",
    "from pydantic import BaseModel, Field"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Experiment parameters\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "claude_api_key = \"<API KEY>\"\n",
    "experiment_date = \"9-12-24\"\n",
    "n_iter = 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prompt and problem setup\n",
    "\n",
    "For this test I’m going to start with a substitute task to write an article for a magazine and provide the response for different questions in a specific format.\n",
    "\n",
    "Here we specify the prompt and any inputs to use to vary the problem (the list of questions).0\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_science_prompt_txt = \"\"\"\n",
    "You are a professional science writer tasked with responding to members of\n",
    "the general public who write in asking questions about science.\n",
    "Write an article responding to a writer's question for publication in a\n",
    "science magazine intended for a general readership with a high-school education.\n",
    "You should write clearly and compellingly, include all relavent context,\n",
    "and provide motivating stories where applicable.\n",
    "\n",
    "Your response must be less than 200 words.\n",
    "\n",
    "The question given to you is the following:\n",
    "{question}\n",
    "\"\"\"\n",
    "\n",
    "questions = [\n",
    "    \"What is the oldest recorded fossil?\",\n",
    "    \"What is a black hole?\",\n",
    "    \"How far away is the sun?\",\n",
    "    \"Which other planet in the Solar System has a surface gravity closest to that of the Earth?\",\n",
    "    \"Eris, Haumea, Makemake and Ceres are all examples of what?\",\n",
    "    \"Why does earth have seasons? Do other planets exhibit seasons too?\",\n",
    "    \"What causes the aurora borealis?\",\n",
    "    \"Why is the sky blue?\",\n",
    "    \"How do bees communicate?\",\n",
    "    \"What is the smallest unit of life?\",\n",
    "    \"How do plants make their own food?\",\n",
    "    \"Why do we dream?\",\n",
    "    \"What is the theory of relativity?\",\n",
    "    \"How do volcanoes erupt?\",\n",
    "    \"What is the speed of light?\",\n",
    "    \"How do magnets work?\",\n",
    "    \"What is the purpose of DNA?\",\n",
    "    \"What are the different types of galaxies?\",\n",
    "    \"Why do some animals hibernate?\",\n",
    "    \"How do vaccines work?\",\n",
    "]\n",
    "\n",
    "prompt_direct = ChatPromptTemplate.from_template(test_science_prompt_txt)\n",
    "\n",
    "prompt_system_format = ChatPromptTemplate.from_messages(\n",
    "    [\n",
    "        (\n",
    "            \"system\",\n",
    "            \"Answer the user query.\\n{format_instructions}\",\n",
    "        ),\n",
    "        (\"human\", test_science_prompt_txt),\n",
    "    ]\n",
    ")\n",
    "\n",
    "prompt_user_format = ChatPromptTemplate.from_template(\n",
    "    test_science_prompt_txt + \"\\n{format_instructions}\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### JSON output format specs\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Pydantic structures\n",
    "\n",
    "To answer the question of how these models and output methods differ with different complexities of schema I’m defining four example schema in increasing order of complexity\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Simple types\n",
    "class ArticleResponse1(BaseModel):\n",
    "    \"\"\"Structured article for publication answering a reader's question.\"\"\"\n",
    "\n",
    "    title: str = Field(description=\"Title of the article\")\n",
    "    answer: str = Field(\n",
    "        description=\"Provide a detailed description of historical events to answer the question.\"\n",
    "    )\n",
    "    number: int = Field(\n",
    "        description=\"An arbitraty number that is most relevant to the question.\"\n",
    "    )\n",
    "\n",
    "\n",
    "# Nested types\n",
    "class HistoricalEvent(BaseModel):\n",
    "    \"\"\"The year and explanation of a historical event.\"\"\"\n",
    "\n",
    "    year: int = Field(description=\"The year of the historical event\")\n",
    "    description: str = Field(\n",
    "        description=\"A clear description of what happened in this event\"\n",
    "    )\n",
    "\n",
    "\n",
    "class ArticleResponse2(BaseModel):\n",
    "    \"\"\"Structured article for publication answering a reader's question.\"\"\"\n",
    "\n",
    "    title: str = Field(description=\"Title of the article\")\n",
    "    historical_event_1: HistoricalEvent = Field(\n",
    "        description=\"Provide a detailed description of one historical events to answer the question.\"\n",
    "    )\n",
    "    historical_event_2: HistoricalEvent = Field(\n",
    "        description=\"Provide a detailed description of one historical events to answer the question.\"\n",
    "    )\n",
    "\n",
    "\n",
    "# Lists of simple types\n",
    "class ArticleResponse3(BaseModel):\n",
    "    \"\"\"Structured article for publication answering a reader's question.\"\"\"\n",
    "\n",
    "    title: str = Field(description=\"Title of the article\")\n",
    "    further_questions: list[str] = Field(\n",
    "        description=\"A list of related questions that may be of interest to the readers.\"\n",
    "    )\n",
    "\n",
    "\n",
    "# Lists of custom types\n",
    "class ArticleResponse4(BaseModel):\n",
    "    \"\"\"Structured article for publication answering a reader's question.\"\"\"\n",
    "\n",
    "    title: str = Field(description=\"Title of the article\")\n",
    "    historical_timeline: list[HistoricalEvent] = Field(\n",
    "        description=\"Provide a compelling account of the historical context of the question\"\n",
    "    )\n",
    "\n",
    "\n",
    "structured_formats = [\n",
    "    dict(pydantic=ArticleResponse1),\n",
    "    dict(pydantic=ArticleResponse2),\n",
    "    dict(pydantic=ArticleResponse3),\n",
    "    dict(pydantic=ArticleResponse4),\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Models to evaluate\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Default temperature\n",
    "temperature = 0.8"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "llm_models = {\n",
    "    # \"Anthropic_Sonnet\": ChatAnthropic(\n",
    "    #     model=\"claude-3-5-sonnet-20241022\", api_key=claude_api_key\n",
    "    # ),\n",
    "    # \"Anthropic_Haiku\": ChatAnthropic(model=\"claude-3-5-haiku-20241022\", api_key=claude_api_key),\n",
    "    # \"Anthropic_Haiku\": ChatAnthropic(\n",
    "    #     model=\"claude-3-haiku-20240307\", api_key=claude_api_key\n",
    "    # ),\n",
    "    \"Ollama_llama32\": ChatOllama(model=\"llama3.2\", temperature=temperature),\n",
    "    \"nemotron-mini\": ChatOllama(model=\"nemotron-mini\", temperature=temperature),\n",
    "    \"Ollama_gemma2\": ChatOllama(model=\"gemma2\", temperature=temperature),\n",
    "    \"Ollama_phi3\": ChatOllama(model=\"phi3\", temperature=temperature),\n",
    "}\n",
    "\n",
    "llm_models_jsonmode = {\n",
    "    \"Ollama_llama32\": ChatOllama(\n",
    "        model=\"llama3.2\", format=\"json\", temperature=temperature\n",
    "    ),\n",
    "    \"nemotron-mini\": ChatOllama(\n",
    "        model=\"nemotron-mini\", format=\"json\", temperature=temperature\n",
    "    ),\n",
    "    \"Ollama_gemma2\": ChatOllama(model=\"gemma2\", format=\"json\", temperature=temperature),\n",
    "    \"Ollama_phi3\": ChatOllama(model=\"phi3\", format=\"json\", temperature=temperature),\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluation\n",
    "\n",
    "Let's loop over different structured outputs and check the adherence using the tool-calling API (structured output mode)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluate Tool Calling API with Pydantic objects\n",
    "\n",
    "Question - of the models that have tool calling, what complexity of structure can they support?\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Method 1 : Tool-calling API\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "structure_support_by_model = {}\n",
    "n_questions = len(questions)\n",
    "\n",
    "for model_name, llm_model in llm_models.items():\n",
    "    structure_support_by_model[model_name] = {}\n",
    "    for structure in structured_formats:\n",
    "        pydantic_obj = structure[\"pydantic\"]\n",
    "        print(f\"Model: {model_name}  Output: {pydantic_obj.__name__}\")\n",
    "\n",
    "        # Iterate over questions\n",
    "        output_valid = 0\n",
    "        tool_use = 0\n",
    "        error_messages = []\n",
    "        outputs = []\n",
    "        for kk in range(n_iter):\n",
    "            for ii in range(n_questions):\n",
    "                test_chain = prompt_direct | llm_model.with_structured_output(\n",
    "                    pydantic_obj, include_raw=True\n",
    "                )\n",
    "                try:\n",
    "                    output = test_chain.invoke(dict(question=questions[ii]))\n",
    "                    tool_use += 1\n",
    "                    if output[\"parsing_error\"] is None:\n",
    "                        output_valid += 1\n",
    "                    else:\n",
    "                        print(output[\"parsing_error\"])\n",
    "                        error_messages.append(output[\"parsing_error\"])\n",
    "                    outputs.append(output)\n",
    "                except Exception as e:\n",
    "                    print(f\"  Tool use error \\n{type(e)}.__name__: {e}\")\n",
    "\n",
    "        structure_support_by_model[model_name][pydantic_obj.__name__] = dict(\n",
    "            valid=output_valid / (n_iter * n_questions),\n",
    "            tool_use=tool_use / (n_iter * n_questions),\n",
    "            errors=error_messages,\n",
    "            outputs=outputs,\n",
    "        )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Method 2 : Output parser\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's do the same for the output parser formatting. Note that as a lot of models seem to ignore this, it takes a lot of time.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_core.output_parsers import PydanticOutputParser\n",
    "\n",
    "\n",
    "def run_experiment_with_op(prompt_format, llm_models, n_iter):\n",
    "    ss_results = {}\n",
    "    n_questions = len(questions)\n",
    "\n",
    "    for model_name, llm_model in llm_models.items():\n",
    "        ss_results[model_name] = {}\n",
    "        for structure in structured_formats:\n",
    "            pydantic_obj = structure[\"pydantic\"]\n",
    "            print(f\"Model: {model_name}  Output: {pydantic_obj.__name__}\")\n",
    "\n",
    "            # Iterate over questions\n",
    "            output_valid = 0\n",
    "            tool_use = 0\n",
    "            error_messages = []\n",
    "            outputs = []\n",
    "\n",
    "            for kk in range(n_iter):\n",
    "                for ii in range(n_questions):\n",
    "                    parser = PydanticOutputParser(pydantic_object=pydantic_obj)\n",
    "                    prompt = prompt_format.partial(\n",
    "                        format_instructions=parser.get_format_instructions()\n",
    "                    )\n",
    "                    test_chain = prompt | llm_model | parser\n",
    "\n",
    "                    try:\n",
    "                        output = test_chain.invoke(dict(question=questions[ii]))\n",
    "                        assert isinstance(output, pydantic_obj)\n",
    "                        output_valid += 1\n",
    "                        outputs.append(output)\n",
    "\n",
    "                    except Exception as e:\n",
    "                        print(f\"  Invalid ouput ({type(e)})\")\n",
    "                        error_messages.append(f\"{type(e).__name__}, {e}\")\n",
    "\n",
    "            ss_results[model_name][pydantic_obj.__name__] = dict(\n",
    "                valid=output_valid / (n_iter * n_questions),\n",
    "                tool_use=tool_use / (n_iter * n_questions),\n",
    "                errors=error_messages,\n",
    "                outputs=outputs,\n",
    "            )\n",
    "    return ss_results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "structure_support_by_model_op = run_experiment_with_op(\n",
    "    prompt_user_format, llm_models, n_iter\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "structure_support_by_model_op_jsonmode = run_experiment_with_op(\n",
    "    prompt_user_format, llm_models_jsonmode, n_iter\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "structure_support_by_model_op_system = run_experiment_with_op(\n",
    "    prompt_system_format, llm_models, n_iter\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Error analysis\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def results_to_df(ss_results, key=\"valid\"):\n",
    "    df = pd.DataFrame.from_dict(\n",
    "        {\n",
    "            mname: {\n",
    "                tname: ss_results[mname][tname][key] * 100 / n_questions\n",
    "                for tname in ss_results[mname].keys()\n",
    "            }\n",
    "            for mname in ss_results.keys()\n",
    "        },\n",
    "        orient=\"index\",\n",
    "    )\n",
    "    return df\n",
    "\n",
    "\n",
    "def analyse_errors_from_results(ss_results, key=\"errors\"):\n",
    "    error_counts = {}\n",
    "    for mname in ss_results.keys():\n",
    "        error_counts[mname] = {}\n",
    "        for tname in ss_results[mname].keys():\n",
    "            validation_error = 0\n",
    "            json_error = 0\n",
    "            unknown_error = 0\n",
    "            errors = ss_results[mname][tname][key]\n",
    "            for error in errors:\n",
    "\n",
    "                error_str = str(error)\n",
    "                if error_str.lower().find(\"invalid json output\") >= 0:\n",
    "                    json_error += 1\n",
    "                elif error_str.lower().find(\"validation error\") >= 0:\n",
    "                    validation_error += 1\n",
    "                else:\n",
    "                    unknown_error += 1\n",
    "            error_counts[mname][(tname, \"invalid_json\")] = json_error\n",
    "            error_counts[mname][(tname, \"validation\")] = validation_error\n",
    "            error_counts[mname][(tname, \"unknown\")] = unknown_error\n",
    "    return pd.DataFrame.from_dict(error_counts, orient=\"index\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "errors_df = analyse_errors_from_results(structure_support_by_model_op, \"errors\")\n",
    "errors_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "errors_df = analyse_errors_from_results(structure_support_by_model_op_system, \"errors\")\n",
    "errors_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "errors_df = analyse_errors_from_results(\n",
    "    structure_support_by_model_op_jsonmode, \"errors\"\n",
    ")\n",
    "errors_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "structure_support_by_model_op_jsonmode[\"Ollama_llama32_json\"][\"ArticleResponse2\"][\n",
    "    \"errors\"\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for ii in range(10):\n",
    "    try:\n",
    "        print(\n",
    "            structure_support_by_model[\"Ollama_llama32\"][\"ArticleResponse2\"][\"outputs\"][\n",
    "                ii\n",
    "            ][\"raw\"].response_metadata[\"message\"][\"tool_calls\"][0][\"function\"][\n",
    "                \"arguments\"\n",
    "            ]\n",
    "        )\n",
    "        print()\n",
    "    except:\n",
    "        print(\"OK\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Errors in tool usem\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(\n",
    "    pd.Series(\n",
    "        [\n",
    "            type(e)\n",
    "            for exp in structure_support_by_model[\"Ollama_llama32\"].values()\n",
    "            for e in exp[\"errors\"]\n",
    "        ]\n",
    "    )\n",
    ").value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(\n",
    "    pd.Series(\n",
    "        [\n",
    "            e.split(\",\")[0]\n",
    "            for exp in structure_support_by_model_op[\"Ollama_llama32\"].values()\n",
    "            for e in exp[\"errors\"]\n",
    "        ]\n",
    "    )\n",
    ").value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Results\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "results_list = {\n",
    "    \"Tool-calling API\": structure_support_by_model,\n",
    "    \"Output Parser User\": structure_support_by_model_op,\n",
    "    \"Output Parser JSONMode\": structure_support_by_model_op_jsonmode,\n",
    "    \"Output Parser System\": structure_support_by_model_op_system,\n",
    "}\n",
    "\n",
    "df_results = {}\n",
    "for name, ss_results in results_list.items():\n",
    "    df_results[name] = pd.DataFrame.from_dict(\n",
    "        {\n",
    "            mname: {\n",
    "                tname: ss_results[mname][tname][\"valid\"] * 100\n",
    "                for tname in ss_results[mname].keys()\n",
    "            }\n",
    "            for mname in ss_results.keys()\n",
    "        },\n",
    "        orient=\"index\",\n",
    "    )\n",
    "    display(name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.concat(df_results)\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.concat(df_results)\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tabulate\n",
    "\n",
    "print(\n",
    "    tabulate.tabulate(\n",
    "        df.reset_index(), headers=\"keys\", tablefmt=\"pipe\", showindex=False\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Save results\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pickle\n",
    "\n",
    "with open(file=f\"exp4_summary_df_{experiment_date}.json\", mode=\"wb\") as f:\n",
    "    df.to_json(f)\n",
    "\n",
    "with open(file=f\"exp4_all_models_{experiment_date}.pkl\", mode=\"wb\") as f:\n",
    "    pickle.dump(\n",
    "        dict(\n",
    "            structure_support_by_model=structure_support_by_model,\n",
    "            structure_support_by_model_op=structure_support_by_model_op,\n",
    "            structure_support_by_model_op_system=structure_support_by_model_op_system,\n",
    "            structure_support_by_model_op_jsonmode=structure_support_by_model_op_jsonmode,\n",
    "        ),\n",
    "        f,\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load results\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pickle\n",
    "import pandas as pd\n",
    "\n",
    "with open(file=f\"exp4_summary_df_{experiment_date}.json\", mode=\"rb\") as f:\n",
    "    df = pd.read_json(f)\n",
    "\n",
    "with open(file=f\"exp4_all_models_{experiment_date}.pkl\", mode=\"rb\") as f:\n",
    "    data = pickle.load(f)\n",
    "\n",
    "# Inject into toplevel namespace\n",
    "namespace = locals()\n",
    "for key, value in data.items():\n",
    "    if key not in namespace:\n",
    "        print(f\"Loaded {key}\")\n",
    "        namespace[key] = value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}