4-langchain-structured-output-evaluation-updated.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluating structured outputs in LangChain\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_core.prompts import ChatPromptTemplate\n",
    "from langchain_anthropic import ChatAnthropic\n",
    "from langchain_ollama import ChatOllama\n",
    "from langchain_fireworks import ChatFireworks\n",
    "from langchain_core.output_parsers import JsonOutputParser, PydanticOutputParser\n",
    "\n",
    "from pydantic import BaseModel, Field\n",
    "from time import sleep\n",
    "\n",
    "import pandas as pd\n",
    "\n",
    "import streamlit as st"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Experiment parameters\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "ANTHROPIC_API_KEY = \"<API KEY>\"\n",
    "FIREWORKS_API_KEY = \"<API KEY>\"\n",
    "experiment_date = \"09-02-25\"\n",
    "n_iter = 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prompt and problem setup\n",
    "\n",
    "For this test I’m going to start with a substitute task to write an article for a magazine and provide the response for different questions in a specific format.\n",
    "\n",
    "Here we specify the prompt and any inputs to use to vary the problem (the list of questions).0\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_science_prompt_txt = \"\"\"\n",
    "You are a professional science writer tasked with responding to members of\n",
    "the general public who write in asking questions about science.\n",
    "Write an article responding to a writer's question for publication in a\n",
    "science magazine intended for a general readership with a high-school education.\n",
    "You should write clearly and compellingly, include all relavent context,\n",
    "and provide motivating stories where applicable.\n",
    "\n",
    "Your response must be less than 200 words.\n",
    "\n",
    "The question given to you is the following:\n",
    "{question}\n",
    "\"\"\"\n",
    "\n",
    "questions = [\n",
    "    \"What is the oldest recorded fossil?\",\n",
    "    \"What is a black hole?\",\n",
    "    \"How far away is the sun?\",\n",
    "    \"Which other planet in the Solar System has a surface gravity closest to that of the Earth?\",\n",
    "    \"Eris, Haumea, Makemake and Ceres are all examples of what?\",\n",
    "    \"Why does earth have seasons? Do other planets exhibit seasons too?\",\n",
    "    \"What causes the aurora borealis?\",\n",
    "    \"Why is the sky blue?\",\n",
    "    \"How do bees communicate?\",\n",
    "    \"What is the smallest unit of life?\",\n",
    "    \"How do plants make their own food?\",\n",
    "    \"Why do we dream?\",\n",
    "    \"What is the theory of relativity?\",\n",
    "    \"How do volcanoes erupt?\",\n",
    "    \"What is the speed of light?\",\n",
    "    \"How do magnets work?\",\n",
    "    \"What is the purpose of DNA?\",\n",
    "    \"What are the different types of galaxies?\",\n",
    "    \"Why do some animals hibernate?\",\n",
    "    \"How do vaccines work?\",\n",
    "]\n",
    "\n",
    "prompt_direct = ChatPromptTemplate.from_template(test_science_prompt_txt)\n",
    "\n",
    "prompt_system_format = ChatPromptTemplate.from_messages(\n",
    "    [\n",
    "        (\n",
    "            \"system\",\n",
    "            \"Answer the user query.\\n{format_instructions}\",\n",
    "        ),\n",
    "        (\"human\", test_science_prompt_txt),\n",
    "    ]\n",
    ")\n",
    "\n",
    "prompt_user_format = ChatPromptTemplate.from_template(\n",
    "    test_science_prompt_txt + \"\\n{format_instructions}\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### JSON output format specs\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Pydantic structures\n",
    "\n",
    "To answer the question of how these models and output methods differ with different complexities of schema I’m defining four example schema in increasing order of complexity\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Simple types\n",
    "class ArticleResponse1(BaseModel):\n",
    "    \"\"\"Structured article for publication answering a reader's question.\"\"\"\n",
    "\n",
    "    title: str = Field(description=\"Title of the article\")\n",
    "    answer: str = Field(\n",
    "        description=\"Provide a detailed description of historical events to answer the question.\"\n",
    "    )\n",
    "    number: int = Field(\n",
    "        description=\"An arbitraty number that is most relevant to the question.\"\n",
    "    )\n",
    "\n",
    "\n",
    "# Lists of simple types\n",
    "class ArticleResponse2(BaseModel):\n",
    "    \"\"\"Structured article for publication answering a reader's question.\"\"\"\n",
    "\n",
    "    title: str = Field(description=\"Title of the article\")\n",
    "    further_questions: list[str] = Field(\n",
    "        description=\"A list of related questions that may be of interest to the readers.\"\n",
    "    )\n",
    "\n",
    "\n",
    "# Nested types\n",
    "class HistoricalEvent(BaseModel):\n",
    "    \"\"\"The year and explanation of a historical event.\"\"\"\n",
    "\n",
    "    year: int = Field(description=\"The year of the historical event\")\n",
    "    description: str = Field(\n",
    "        description=\"A clear description of what happened in this event\"\n",
    "    )\n",
    "\n",
    "\n",
    "class ArticleResponse3(BaseModel):\n",
    "    \"\"\"Structured article for publication answering a reader's question.\"\"\"\n",
    "\n",
    "    title: str = Field(description=\"Title of the article\")\n",
    "    historical_event_1: HistoricalEvent = Field(\n",
    "        description=\"Provide a detailed description of one historical events to answer the question.\"\n",
    "    )\n",
    "    historical_event_2: HistoricalEvent = Field(\n",
    "        description=\"Provide a detailed description of one historical events to answer the question.\"\n",
    "    )\n",
    "\n",
    "\n",
    "# Lists of custom types\n",
    "class ArticleResponse4(BaseModel):\n",
    "    \"\"\"Structured article for publication answering a reader's question.\"\"\"\n",
    "\n",
    "    title: str = Field(description=\"Title of the article\")\n",
    "    historical_timeline: list[HistoricalEvent] = Field(\n",
    "        description=\"Provide a compelling account of the historical context of the question\"\n",
    "    )\n",
    "\n",
    "\n",
    "# Nested types\n",
    "class CriicalAnalysis(BaseModel):\n",
    "    \"\"\"A critique of interpretations of historical events\"\"\"\n",
    "\n",
    "    historical_event: HistoricalEvent = Field(\n",
    "        description=\"Provide an overview of the facts of a historical event\"\n",
    "    )\n",
    "    common_understanding: str = Field(description=\"Agreed interpretation of event\")\n",
    "    analysis: str = Field(\n",
    "        description=\"Critical analysis of the event and opposing interpretations\"\n",
    "    )\n",
    "\n",
    "\n",
    "# Multiple nested custom types\n",
    "class ArticleResponse5(BaseModel):\n",
    "    \"\"\"Structured article for publication answering a reader's question.\"\"\"\n",
    "\n",
    "    title: str = Field(description=\"Title of the article\")\n",
    "    historical_timeline: list[HistoricalEvent] = Field(\n",
    "        description=\"Provide a compelling account of the historical context of the question\"\n",
    "    )\n",
    "    critique: list[CriicalAnalysis] = Field(\n",
    "        description=\"A list of key historical events and historical analysis of them\"\n",
    "    )\n",
    "\n",
    "\n",
    "structured_formats = [\n",
    "    dict(pydantic=ArticleResponse1),\n",
    "    dict(pydantic=ArticleResponse2),\n",
    "    dict(pydantic=ArticleResponse3),\n",
    "    dict(pydantic=ArticleResponse4),\n",
    "    dict(pydantic=ArticleResponse5),\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Models to evaluate\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Default temperature\n",
    "temperature = 0.0\n",
    "timeout = 30\n",
    "num_ctx = 8192\n",
    "num_predict = 4096"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "llm_models = {\n",
    "    \"Ollama_llama32\": ChatOllama(\n",
    "        model=\"llama3.2\",\n",
    "        temperature=temperature,\n",
    "        num_ctx=num_ctx,\n",
    "        num_thread=1,\n",
    "        num_predict=num_predict,\n",
    "    ),\n",
    "    \"Ollama_nemotron\": ChatOllama(\n",
    "        model=\"nemotron-mini\",\n",
    "        temperature=temperature,\n",
    "        num_ctx=num_ctx,\n",
    "        num_thread=1,\n",
    "        num_predict=num_predict,\n",
    "    ),\n",
    "    \"Ollama_phi3\": ChatOllama(\n",
    "        model=\"phi3\",\n",
    "        temperature=temperature,\n",
    "        num_ctx=num_ctx,\n",
    "        num_thread=1,\n",
    "        num_predict=num_predict,\n",
    "    ),\n",
    "    \"Ollama_phi4\": ChatOllama(\n",
    "        model=\"phi4\",\n",
    "        temperature=temperature,\n",
    "        num_ctx=num_ctx,\n",
    "        num_thread=1,\n",
    "        num_predict=num_predict,\n",
    "    ),\n",
    "    \"Ollama_deepseekr1\": ChatOllama(\n",
    "        model=\"deepseek-r1\",\n",
    "        temperature=temperature,\n",
    "        num_ctx=num_ctx,\n",
    "        num_thread=1,\n",
    "        num_predict=num_predict,\n",
    "    ),\n",
    "    \"fireworks_llama31\": ChatFireworks(\n",
    "        model_name=\"accounts/fireworks/models/llama-v3p1-70b-instruct\",\n",
    "        api_key=FIREWORKS_API_KEY,\n",
    "        temperature=temperature,\n",
    "        timeout=timeout,\n",
    "    ),\n",
    "    \"fireworks_llama32\": ChatFireworks(\n",
    "        model_name=\"accounts/fireworks/models/llama-v3p2-3b-instruct\",\n",
    "        api_key=FIREWORKS_API_KEY,\n",
    "        temperature=temperature,\n",
    "        timeout=timeout,\n",
    "    ),\n",
    "    \"fireworks_llama33\": ChatFireworks(\n",
    "        model_name=\"accounts/fireworks/models/llama-v3p3-70b-instruct\",\n",
    "        api_key=FIREWORKS_API_KEY,\n",
    "        temperature=temperature,\n",
    "        timeout=timeout,\n",
    "    ),\n",
    "    \"fireworks_deepseekr1_70b\": ChatFireworks(\n",
    "        model_name=\"accounts/fireworks/models/deepseek-r1\",\n",
    "        api_key=FIREWORKS_API_KEY,\n",
    "        temperature=temperature,\n",
    "        timeout=timeout,\n",
    "    ),\n",
    "}\n",
    "llm_models_with_anthropic = {\n",
    "    **llm_models,\n",
    "    \"Anthropic_Sonnet_35\": ChatAnthropic(\n",
    "        model=\"claude-3-5-sonnet-20241022\",\n",
    "        api_key=ANTHROPIC_API_KEY,\n",
    "        timeout=timeout,\n",
    "    ),\n",
    "    \"Anthropic_Haiku_35\": ChatAnthropic(\n",
    "        model=\"claude-3-5-haiku-20241022\",\n",
    "        api_key=ANTHROPIC_API_KEY,\n",
    "        timeout=timeout,\n",
    "    ),\n",
    "    \"Anthropic_Haiku_3\": ChatAnthropic(\n",
    "        model=\"claude-3-haiku-20240307\",\n",
    "        api_key=ANTHROPIC_API_KEY,\n",
    "        timeout=timeout,\n",
    "    ),\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluation\n",
    "\n",
    "Let's loop over different structured outputs and check the adherence using the tool-calling API (structured output mode)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluate Structured Ouputs accross providers & models\n",
    "\n",
    "Question - of the models that have tool calling, what complexity of structure can they support?\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pickle\n",
    "\n",
    "\n",
    "def run_experiment(\n",
    "    prompt_format,\n",
    "    questions,\n",
    "    llm_models,\n",
    "    method,\n",
    "    n_iter=1,\n",
    "    resume=0,\n",
    "    results_out=None,\n",
    "    save_file_name=None,\n",
    "):\n",
    "\n",
    "    if results_out is None:\n",
    "        structure_support_by_model = {}\n",
    "    else:\n",
    "        structure_support_by_model = results_out\n",
    "    n_questions = len(questions)\n",
    "\n",
    "    position = 0\n",
    "\n",
    "    # Iterate over models\n",
    "    for model_name, llm_model in llm_models.items():\n",
    "        structure_support_by_model[model_name] = {}\n",
    "\n",
    "        # Iterate over schemas\n",
    "        for structure in structured_formats:\n",
    "            pydantic_obj = structure[\"pydantic\"]\n",
    "            print(\n",
    "                f\"Model: {model_name}  Output: {pydantic_obj.__name__}   Pos: {position}\"\n",
    "            )\n",
    "\n",
    "            position += 1\n",
    "            if position < resume:\n",
    "                continue\n",
    "\n",
    "            # Format instructions if required\n",
    "            parser = PydanticOutputParser(pydantic_object=pydantic_obj)\n",
    "            prompt = prompt_format.partial(\n",
    "                format_instructions=parser.get_format_instructions()\n",
    "            )\n",
    "\n",
    "            # Iterate over questions\n",
    "            error_types = []\n",
    "            error_messages = []\n",
    "            outputs = []\n",
    "            output_valid = 0\n",
    "            for _ in range(n_iter):\n",
    "                for ii in range(n_questions):\n",
    "                    try:\n",
    "                        test_chain = prompt | llm_model.with_structured_output(\n",
    "                            pydantic_obj, method=method, include_raw=True\n",
    "                        )\n",
    "                        output = test_chain.invoke(dict(question=questions[ii]))\n",
    "                        outputs.append(output)\n",
    "\n",
    "                        # Typically Pydantic validation failure\n",
    "                        if output[\"parsing_error\"] is not None:\n",
    "                            error_types.append(\"parse_error\")\n",
    "                            error_messages.append(output[\"parsing_error\"])\n",
    "                            print(\"Error: Parse error\")\n",
    "\n",
    "                        # Typically function-calling failure\n",
    "                        elif output[\"parsed\"] is None:\n",
    "                            error_types.append(\"no_output\")\n",
    "                            print(\"Error: No output\")\n",
    "\n",
    "                        # This is not expected to happen\n",
    "                        elif not isinstance(output[\"parsed\"], pydantic_obj):\n",
    "                            error_types.append(\"unexpected_error\")\n",
    "                            raise RuntimeError(\"Unexpected error\")\n",
    "\n",
    "                        else:\n",
    "                            error_types.append(\"ok\")\n",
    "                            output_valid += 1\n",
    "\n",
    "                    # Other failures (typically function-calling not supported)\n",
    "                    except Exception as e:\n",
    "                        error_types.append(\"other_error\")\n",
    "                        print(f\"Error: Other error {type(e).__name__}\")\n",
    "                        error_messages.append(f\"{type(e).__name__}, {e}\")\n",
    "\n",
    "                    # Pause to avoid timeouts\n",
    "                    print(\".\", end=\"\")\n",
    "                    sleep(1)\n",
    "                print()\n",
    "\n",
    "            structure_support_by_model[model_name][pydantic_obj.__name__] = dict(\n",
    "                valid=output_valid / (n_iter * n_questions),\n",
    "                error_types=error_types,\n",
    "                errors=error_messages,\n",
    "                outputs=outputs,\n",
    "            )\n",
    "    if save_file_name:\n",
    "        with open(file=save_file_name, mode=\"wb\") as f:\n",
    "            pickle.dump(\n",
    "                dict(\n",
    "                    method=method,\n",
    "                    prompt=prompt,\n",
    "                    questions=questions,\n",
    "                    structure_support_by_model=structure_support_by_model,\n",
    "                ),\n",
    "                f,\n",
    "            )\n",
    "    return structure_support_by_model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Function Calling: Include Anthropic models\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if \"structure_support_by_model_fc\" not in locals():\n",
    "    structure_support_by_model_fc = {}\n",
    "run_experiment(\n",
    "    prompt_direct,\n",
    "    questions,\n",
    "    llm_models_with_anthropic,\n",
    "    method=\"function_calling\",\n",
    "    n_iter=1,\n",
    "    results_out=structure_support_by_model_fc,\n",
    "    save_file_name=f\"exp5_function_calling_{experiment_date}.pkl\",\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if \"structure_support_by_model_js\" not in locals():\n",
    "    structure_support_by_model_js = {}\n",
    "run_experiment(\n",
    "    prompt_direct,\n",
    "    questions,\n",
    "    llm_models,\n",
    "    method=\"json_schema\",\n",
    "    n_iter=1,\n",
    "    results_out=structure_support_by_model_js,\n",
    "    save_file_name=f\"exp5_json_schema_{experiment_date}.pkl\",\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if \"structure_support_by_model_jm\" not in locals():\n",
    "    structure_support_by_model_jm = {}\n",
    "run_experiment(\n",
    "    prompt_user_format,\n",
    "    questions,\n",
    "    llm_models,\n",
    "    method=\"json_mode\",\n",
    "    n_iter=1,\n",
    "    results_out=structure_support_by_model_jm,\n",
    "    save_file_name=f\"exp5_json_mode_{experiment_date}.pkl\",\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "def results_to_df(ss_results, key=\"valid\"):\n",
    "    df = pd.DataFrame.from_dict(\n",
    "        {\n",
    "            mname: {\n",
    "                tname: ss_results[mname][tname][key] * 100 / n_questions\n",
    "                for tname in ss_results[mname].keys()\n",
    "            }\n",
    "            for mname in ss_results.keys()\n",
    "        },\n",
    "        orient=\"index\",\n",
    "    )\n",
    "    return df\n",
    "\n",
    "\n",
    "def analyse_errors_from_results(ss_results, method=\"code\"):\n",
    "    error_counts = {}\n",
    "    for mname in ss_results.keys():\n",
    "        error_counts[mname] = {}\n",
    "        for tname in ss_results[mname].keys():\n",
    "            validation_error = 0\n",
    "            json_error = 0\n",
    "            unknown_error = 0\n",
    "\n",
    "            # Count errors by failure code above\n",
    "            if method == \"code\":\n",
    "                error_types = pd.Series(ss_results[mname][tname][\"error_types\"])\n",
    "                error_codes = error_types.value_counts()\n",
    "\n",
    "                for e_name, e_count in error_codes.items():\n",
    "                    error_counts[mname][(tname, e_name)] = e_count\n",
    "\n",
    "            elif method == \"parse\":\n",
    "                # Count errors by parsing error message\n",
    "                errors = ss_results[mname][tname][\"errors\"]\n",
    "                for error in errors:\n",
    "                    error_str = str(error)\n",
    "                    if error_str.lower().find(\"invalid json output\") >= 0:\n",
    "                        json_error += 1\n",
    "                    elif error_str.lower().find(\"validation error\") >= 0:\n",
    "                        validation_error += 1\n",
    "                    else:\n",
    "                        unknown_error += 1\n",
    "                error_counts[mname][(tname, \"invalid_json\")] = json_error\n",
    "                error_counts[mname][(tname, \"validation\")] = validation_error\n",
    "                error_counts[mname][(tname, \"unknown\")] = unknown_error\n",
    "\n",
    "            else:\n",
    "                raise NameError(f\"Method {method} not supported\")\n",
    "\n",
    "    return pd.DataFrame.from_dict(error_counts, orient=\"index\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "def results_to_df(ss_results, key=\"valid\"):\n",
    "    df = pd.DataFrame.from_dict(\n",
    "        {\n",
    "            mname: {\n",
    "                tname: ss_results[mname][tname][key] * 100 / n_questions\n",
    "                for tname in ss_results[mname].keys()\n",
    "            }\n",
    "            for mname in ss_results.keys()\n",
    "        },\n",
    "        orient=\"index\",\n",
    "    )\n",
    "    return df\n",
    "\n",
    "\n",
    "def analyse_errors_from_results(ss_results, method=\"code\"):\n",
    "    error_counts = {}\n",
    "    for mname in ss_results.keys():\n",
    "        error_counts[mname] = {}\n",
    "        for tname in ss_results[mname].keys():\n",
    "            validation_error = 0\n",
    "            json_error = 0\n",
    "            unknown_error = 0\n",
    "\n",
    "            # Count errors by failure code above\n",
    "            if method == \"code\":\n",
    "                error_types = pd.Series(ss_results[mname][tname][\"error_types\"])\n",
    "                error_codes = error_types.value_counts()\n",
    "\n",
    "                for e_name, e_count in error_codes.items():\n",
    "                    error_counts[mname][(tname, e_name)] = e_count\n",
    "\n",
    "            elif method == \"parse\":\n",
    "                # Count errors by parsing error message\n",
    "                errors = ss_results[mname][tname][\"errors\"]\n",
    "                for error in errors:\n",
    "                    error_str = str(error)\n",
    "                    if error_str.lower().find(\"invalid json output\") >= 0:\n",
    "                        json_error += 1\n",
    "                    elif error_str.lower().find(\"validation error\") >= 0:\n",
    "                        validation_error += 1\n",
    "                    else:\n",
    "                        unknown_error += 1\n",
    "                error_counts[mname][(tname, \"invalid_json\")] = json_error\n",
    "                error_counts[mname][(tname, \"validation\")] = validation_error\n",
    "                error_counts[mname][(tname, \"unknown\")] = unknown_error\n",
    "\n",
    "            else:\n",
    "                raise NameError(f\"Method {method} not supported\")\n",
    "\n",
    "    return pd.DataFrame.from_dict(error_counts, orient=\"index\").fillna(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Function calling errors\n",
    "\n",
    "- Fireworks Llama3.2 doesn't support function calling\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "errors_df = analyse_errors_from_results(structure_support_by_model_fc)\n",
    "errors_df.sort_index(axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### JSON Mode\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "errors_df = analyse_errors_from_results(structure_support_by_model_jm)\n",
    "errors_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### JSON Schema\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "errors_df = analyse_errors_from_results(structure_support_by_model_js)\n",
    "errors_df.sort_index(axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at the errors in the JSON schema method\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "all_ollama_errors = [\n",
    "    output[\"parsing_error\"]\n",
    "    for model in structure_support_by_model_js\n",
    "    if model.startswith(\"Ollama\")\n",
    "    for struct, outputs in structure_support_by_model_js[model].items()\n",
    "    for output in outputs[\"outputs\"]\n",
    "    if output[\"parsing_error\"] is not None\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "all_ollama_errors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Results\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "results_list = {\n",
    "    \"Function-calling\": structure_support_by_model_fc,\n",
    "    \"JSON Schema\": structure_support_by_model_js,\n",
    "    \"JSON Mode\": structure_support_by_model_jm,\n",
    "}\n",
    "\n",
    "df_results = {}\n",
    "for name, ss_results in results_list.items():\n",
    "    df_results[name] = pd.DataFrame.from_dict(\n",
    "        {\n",
    "            mname: {\n",
    "                tname: ss_results[mname][tname][\"valid\"] * 100\n",
    "                for tname in ss_results[mname].keys()\n",
    "            }\n",
    "            for mname in ss_results.keys()\n",
    "        },\n",
    "        orient=\"index\",\n",
    "    )\n",
    "    display(name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.concat(df_results).swaplevel(axis=0).sort_index(axis=0)\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Save results\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(file=f\"exp5_summary_df_{experiment_date}.json\", mode=\"wb\") as f:\n",
    "    df.to_json(f)\n",
    "\n",
    "with open(file=f\"exp5_all_models_{experiment_date}.pkl\", mode=\"wb\") as f:\n",
    "    pickle.dump(\n",
    "        dict(\n",
    "            temperature=temperature,\n",
    "            num_ctx=num_ctx,\n",
    "            num_predict=num_predict,\n",
    "            questions=questions,\n",
    "            prompt=prompt_direct,\n",
    "            structure_support_by_model_fc=structure_support_by_model_fc,\n",
    "            structure_support_by_model_jm=structure_support_by_model_jm,\n",
    "            structure_support_by_model_js=structure_support_by_model_js,\n",
    "        ),\n",
    "        f,\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load results\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pickle\n",
    "import pandas as pd\n",
    "\n",
    "# with open(file=f\"exp5_summary_df_{experiment_date}.json\", mode=\"rb\") as f:\n",
    "#    df = pd.read_json(f)\n",
    "\n",
    "with open(file=f\"exp5_all_models_{experiment_date}.pkl\", mode=\"rb\") as f:\n",
    "    data = pickle.load(f)\n",
    "\n",
    "# Inject into toplevel namespace\n",
    "namespace = locals()\n",
    "for key, value in data.items():\n",
    "    if key not in namespace:\n",
    "        print(f\"Loaded {key}\")\n",
    "        namespace[key] = value"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}