notebooks/spring-ai/tutorials/10. Local model.ipynb

{
 "cells": [
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": [
    "# Local Model\n",
    "\n",
    "In this tutorial, we'll explore working with local LLMs on your own computer using Spring-AI and Kotlin.\n",
    "While we've focused on remote providers like OpenAI and Anthropic in previous tutorials,\n",
    "running models locally offers several distinct advantages.\n",
    "Let's explore how to set up and use local models with the same Spring-AI abstractions we've been learning throughout this series.\n",
    "\n",
    "Let's explore how to set up and use local models with the same Spring-AI abstractions we've been learning throughout this series."
   ]
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-03-29T08:53:34.685171Z",
     "start_time": "2025-03-29T08:53:34.276343Z"
    }
   },
   "cell_type": "code",
   "source": [
    "@file:DependsOn(\"org.springframework.ai:spring-ai-ollama-spring-boot-starter:1.0.0-M6\")\n",
    "@file:DependsOn(\"com.fasterxml.jackson.module:jackson-module-kotlin:2.18.2\")"
   ],
   "outputs": [],
   "execution_count": 1
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": [
    "## Why use local models?\n",
    "\n",
    "Local models may not match the power of cloud-based alternatives, but they offer compelling benefits:\n",
    "\n",
    "- **Free to use** — no API costs or rate limits\n",
    "- **Work offline** — no internet connection required\n",
    "- **Privacy** — process sensitive data without sending it to third parties\n",
    "- **Full control** — you determine model settings and deployment\n",
    "\n",
    "Spring-AI supports local models through Ollama,\n",
    "which makes running LLMs on your machine remarkably straightforward.\n",
    "\n",
    "## Setting up Ollama\n",
    "\n",
    "First,\n",
    "visit [ollama.com](https://ollama.com/)\n",
    "and install the application by following the simple instructions for your operating system.\n",
    "\n",
    "Next,\n",
    "explore the [Models page](https://ollama.com/search)\n",
    "to find a model that fits your needs.\n",
    "You'll see options ranging from small,\n",
    "fast models to larger, more capable ones.\n",
    "Consider your hardware specifications when choosing —\n",
    "larger models require more RAM and GPU resources.\n",
    "\n",
    "For this tutorial, I've selected `deepseek-r1`,\n",
    "which demonstrates _\"thinking\"_ capabilities similar to reasoning modes in commercial models.\n",
    "\n",
    "Let's download and start the model:\n",
    "\n",
    "```bash\n",
    "ollama run deepseek-r1\n",
    "```\n",
    "\n",
    "This command will download the `deepseek-r1:7b` model\n",
    "(if you don't already have it) and start it.\n",
    "The first run may take some time as it downloads the model files.\n",
    "\n",
    "## Creating a Spring-AI Client for Local Model\n",
    "\n",
    "Now that our local model is running, let's set up the Spring-AI client:"
   ]
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-03-29T08:53:35.906256Z",
     "start_time": "2025-03-29T08:53:35.400043Z"
    }
   },
   "cell_type": "code",
   "source": [
    "import org.springframework.ai.chat.client.ChatClient\n",
    "import org.springframework.ai.ollama.OllamaChatModel\n",
    "import org.springframework.ai.ollama.api.OllamaApi\n",
    "import org.springframework.ai.ollama.api.OllamaModel\n",
    "import org.springframework.ai.ollama.api.OllamaOptions\n",
    "\n",
    "val ollamaApi = OllamaApi()\n",
    "\n",
    "val ollamaOptions = OllamaOptions.builder()\n",
    "    .model(\"deepseek-r1\")\n",
    "    .temperature(0.7)\n",
    "    .build()\n",
    "\n",
    "val ollamaModel = OllamaChatModel.builder()\n",
    "    .ollamaApi(ollamaApi)\n",
    "    .defaultOptions(ollamaOptions)\n",
    "    .build()\n",
    "\n",
    "val chatClient = ChatClient.create(ollamaModel)"
   ],
   "outputs": [],
   "execution_count": 2
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": [
    "Notice that we don't need any API keys — the model is already running on our device.\n",
    "The process for creating a ChatModel is very similar to what we've done with OpenAI,\n",
    "Anthropic, and other providers.\n",
    "\n",
    "With everything set up, let's test our model with a simple query:"
   ]
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-03-29T08:54:03.831229Z",
     "start_time": "2025-03-29T08:53:37.921194Z"
    }
   },
   "cell_type": "code",
   "source": "chatClient.prompt(\"tell me a joke about Kotlin\").call().content()",
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<think>\n",
       "Okay, so I need to come up with a joke about Kotlin. Hmm, where do I start? Well, I know that Kotlin is a programming language, and it's often compared to Java because it runs on the Java Virtual Machine (JVM). So maybe I can play around with that.\n",
       "\n",
       "Let me think about the differences between Java and Kotlin. One of the main selling points of Kotlin is that it's more modern and concise, which makes coding easier and less error-prone. There are also some features like null safety, which was a big issue in Java. Oh, right, in Java you can have nullable types with 'null' being allowed, but that can lead to NullPointerExceptions if not handled properly.\n",
       "\n",
       "In Kotlin, they introduced the concept of 'nullable' and 'non-nullable' variables using the '?' symbol. So, for example, a variable like String? would be nullable, meaning it could hold a String or null. But if you try to call a method on a null value, it will throw an exception automatically, which is called a NullPointerException but in Kotlin, it's more descriptive.\n",
       "\n",
       "Wait, so maybe I can make a joke about how Java has these issues with nulls and exceptions, while Kotlin handles them better. That could be funny because it's highlighting a problem in Java and showing that Kotlin makes things easier.\n",
       "\n",
       "Let me try to structure the joke. Maybe something like why don't Java developers argue with each other? Because they have so much in common... but then switch to Kotlin for the punchline. How about: \"Why don’t Java Developers argue with each other? They’ve got too much in common—like NullPointerExceptions! But wait, if you use Kotlin, those are a thing of the past!\"\n",
       "\n",
       "That makes sense because it starts by talking about Java's problem and then introduces Kotlin as the solution. The humor comes from the contrast between Java's issues and Kotlin's improvements.\n",
       "\n",
       "I should make sure that the joke is clear and not too technical for someone who isn't familiar with both languages. It needs to be relatable, so using terms like NullPointerExceptions is okay because they are a common pain point for Java developers.\n",
       "\n",
       "Another angle could be the syntax differences or other features, but I think focusing on null safety and exception handling is a strong point because it directly addresses a known issue and shows how Kotlin improves upon it.\n",
       "\n",
       "Maybe add a bit more flair to the punchline. Instead of just saying \"those are a thing of the past,\" perhaps something like \"Kotlin makes them a thing of the past.\" That emphasizes the benefit more.\n",
       "\n",
       "So putting it all together, the joke would be: \"Why don’t Java Developers argue with each other? They’ve got too much in common—like NullPointerExceptions! But if you use Kotlin, those are a thing of the past!\"\n",
       "\n",
       "I think that works. It's concise, makes the point clearly, and has a humorous twist by highlighting a problem and then offering a solution.\n",
       "</think>\n",
       "\n",
       "**Joke:**\n",
       "\n",
       "Why don’t Java Developers argue with each other?  \n",
       "They’ve got too much in common—like NullPointerExceptions!  \n",
       "\n",
       "But if you use Kotlin, those are a thing of the past!"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "execution_count": 3
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": [
    "You'll notice the response includes a new section under the `<think>` tag.\n",
    "This is the reasoning block where you can see how the model formed its response to your request.\n",
    "\n",
    "Given the amount of information returned,\n",
    "the model processed quite quickly — another advantage of local models.\n",
    "\n",
    "## Streaming Responses with Local Models\n",
    "\n",
    "Let's see how streaming works with local models,\n",
    "allowing us to watch the reasoning process in real-time.\n",
    "\n",
    "First, we'll need to add dependencies for working with `Flow`:"
   ]
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-03-29T08:54:04.633710Z",
     "start_time": "2025-03-29T08:54:03.836358Z"
    }
   },
   "cell_type": "code",
   "source": [
    "%useLatestDescriptors\n",
    "%use coroutines\n",
    "@file:DependsOn(\"org.jetbrains.kotlinx:kotlinx-coroutines-reactive:1.10.1\")"
   ],
   "outputs": [],
   "execution_count": 4
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": "Now, let's create a more complex prompt and stream the response:"
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-03-29T08:57:28.888616Z",
     "start_time": "2025-03-29T08:56:31.198549Z"
    }
   },
   "cell_type": "code",
   "source": [
    "import kotlinx.coroutines.reactive.asFlow\n",
    "\n",
    "val streamResponse = chatClient\n",
    "    .prompt(\n",
    "        \"\"\"\n",
    "        Generate a response in the format of a fully valid HTML5 document, answering the following question:\n",
    "\n",
    "        <Question>\n",
    "        Create a colorful periodic table of Kotlin operators showing precedence levels.\n",
    "        Include descriptions with examples of each operator in action.\n",
    "        </Question>\n",
    "\n",
    "        <Instructions>\n",
    "        Your answer must include:\n",
    "        1. The complete structure of an HTML document with <html>, <head> (including <title>) and <body> tags\n",
    "        2. Semantic markup (h1, h2, p, ul/ol, etc.)\n",
    "        3. Use tables or lists where appropriate to structure the data.\n",
    "        4. Include minimalist CSS styles to improve readability.\n",
    "        5. The page must be fully valid HTML5.\n",
    "\n",
    "        Do not add explanations outside the HTML markup. The entire answer must be only in HTML format!\n",
    "        </Instructions>\n",
    "        \"\"\"\n",
    "    )\n",
    "    .stream()\n",
    "    .content()\n",
    "    .asFlow()\n",
    "\n",
    "val answer = StringBuilder()\n",
    "runBlocking {\n",
    "    streamResponse.collect {\n",
    "        answer.append(it)\n",
    "        print(it)\n",
    "    }\n",
    "}"
   ],
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<think>\n",
      "Okay, I need to create a response that's a fully valid HTML5 document answering how to generate a colorful periodic table of Kotlin operators with precedence levels and examples. \n",
      "\n",
      "First, I'll start by setting up the basic structure: html, head, title, and body tags. The head should include meta tags for charset and viewport, and a title like \"Kotlin Operators Periodic Table.\"\n",
      "\n",
      "Next, styling is important for readability. I'll add internal CSS styles. Maybe define classes for precedence levels as different colors—like yellow for highest, green for next, etc. Also, style the table with borders and alternating row colors to make it visually appealing.\n",
      "\n",
      "In the body, an h1 heading will introduce the periodic table. Then, a p to explain what's included—precedence levels and examples.\n",
      "\n",
      "The main content is a table. I'll structure it with operators as rows, columns for precedence level, symbol, description, and example. Each operator should have its own row. For precedence, assign color classes based on their level. \n",
      "\n",
      "I'll list all the operators I can remember: assignment (=), increment (++), decrement (--), etc., making sure to include about 15-20 operators. Each entry will have a brief description and an example.\n",
      "\n",
      "Finally, add notes below the table for operator associativity and precedence tiebreakers, using unordered lists for clarity.\n",
      "\n",
      "I need to ensure all tags are properly closed, use semantic HTML elements where possible, and keep the CSS minimal but effective. Test the HTML to make sure it's valid and looks good.\n",
      "</think>\n",
      "\n",
      "```html\n",
      "<!DOCTYPE html>\n",
      "<html lang=\"en\">\n",
      "<head>\n",
      "    <meta charset=\"UTF-8\">\n",
      "    <meta name=\"viewport\" content=\"width=device-width, initial-scale=1.0\">\n",
      "    <title>Kotlin Operators Periodic Table</title>\n",
      "    <style>\n",
      "        body {\n",
      "            font-family: Arial, sans-serif;\n",
      "            margin: 20px;\n",
      "            background-color: #f0f0f0;\n",
      "        }\n",
      "        \n",
      "        h1 {\n",
      "            color: #333;\n",
      "            text-align: center;\n",
      "        }\n",
      "\n",
      "        .periodic-table {\n",
      "            width: 100%;\n",
      "            border-collapse: collapse;\n",
      "            margin: 20px auto;\n",
      "            box-shadow: 0 0 10px rgba(0,0,0,0.1);\n",
      "        }\n",
      "\n",
      "        th, td {\n",
      "            padding: 15px;\n",
      "            text-align: left;\n",
      "            border-bottom: 1px solid #ddd;\n",
      "        }\n",
      "\n",
      "        .highest-precedence {\n",
      "            background-color: #ffeb3b;\n",
      "            color: #333;\n",
      "        }\n",
      "\n",
      "        .high-precedence {\n",
      "            background-color: #90EE90;\n",
      "            color: #333;\n",
      "        }\n",
      "\n",
      "        .medium-precedence {\n",
      "            background-color: #87CEEB;\n",
      "            color: #333;\n",
      "        }\n",
      "\n",
      "        .low-precedence {\n",
      "            background-color: #fff5e6;\n",
      "            color: #333;\n",
      "        }\n",
      "\n",
      "        .lowest-precedence {\n",
      "            background-color: #f0f0f0;\n",
      "            color: #333;\n",
      "        }\n",
      "\n",
      "        .example-box {\n",
      "            background-color: #f8f9fa;\n",
      "            padding: 10px;\n",
      "            border-radius: 4px;\n",
      "            margin-top: 5px;\n",
      "        }\n",
      "    </style>\n",
      "</head>\n",
      "<body>\n",
      "    <h1>Kotlin Operators Periodic Table</h1>\n",
      "    \n",
      "    <p>This table presents Kotlin operators in a periodic table format, organized by precedence levels. Each entry includes the operator symbol, description, and an example.</p>\n",
      "\n",
      "    <table class=\"periodic-table\">\n",
      "        <tr>\n",
      "            <th>Precedence Level</th>\n",
      "            <th>Operator Symbol</th>\n",
      "            <th>Description</th>\n",
      "            <th>Example</th>\n",
      "        </tr>\n",
      "        \n",
      "        <tr class=\"highest-precedence\">\n",
      "            <td>Level 1</td>\n",
      "            <td><strong>++</strong></td>\n",
      "            <td>Increment operator (prefix)</td>\n",
      "            <td class=\"example-box\">val a = 5<br>println(a++) // Outputs 5 then increments to 6</td>\n",
      "        </tr>\n",
      "\n",
      "        <tr class=\"high-precedence\">\n",
      "            <td>Level 2</td>\n",
      "            <td><strong>-</strong></td>\n",
      "            <td>Unary minus</td>\n",
      "            <td class=\"example-box\">val b = -5<br>// Represents negative five</td>\n",
      "        </tr>\n",
      "\n",
      "        <tr class=\"medium-precedence\">\n",
      "            <td>Level 3</td>\n",
      "            <td><strong>*</strong></td>\n",
      "            <td>Multiplication</td>\n",
      "            <td class=\"example-box\">val c = 2 * 3<br>// Outputs 6</td>\n",
      "        </tr>\n",
      "\n",
      "        <tr class=\"low-precedence\">\n",
      "            <td>Level 4</td>\n",
      "            <td><strong>+</strong></td>\n",
      "            <td>Addition</td>\n",
      "            <td class=\"example-box\">val d = 2 + 3<br>// Outputs 5</td>\n",
      "        </tr>\n",
      "\n",
      "        <tr class=\"lowest-precedence\">\n",
      "            <td>Level 5</td>\n",
      "            <td><strong>=</strong></td>\n",
      "            <td>Assignment</td>\n",
      "            <td class=\"example-box\">var x = 10<br>// Assigns value 10 to x</td>\n",
      "        </tr>\n",
      "\n",
      "        <!-- Add more rows for other operators -->\n",
      "    </table>\n",
      "\n",
      "    <p class=\"note\"><em>Notes:</em></p>\n",
      "    <ul>\n",
      "        <li>Operators with the same precedence are evaluated left-to-right</li>\n",
      "        <li>Parentheses can override default precedence rules</li>\n",
      "    </ul>\n",
      "</body>\n",
      "</html>\n",
      "```"
     ]
    }
   ],
   "execution_count": 7
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": [
    "The response consists of two parts: the reasoning and the final HTML answer.\n",
    "Let's extract the HTML part and render it in our notebook:"
   ]
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-03-29T08:57:28.932968Z",
     "start_time": "2025-03-29T08:57:28.893597Z"
    }
   },
   "cell_type": "code",
   "source": [
    "val html = answer.toString().substringAfterLast(\"```html\").substringBeforeLast(\"```\")\n",
    "\n",
    "HTML(html)"
   ],
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "<!DOCTYPE html>\n",
       "<html lang=\"en\">\n",
       "<head>\n",
       "    <meta charset=\"UTF-8\">\n",
       "    <meta name=\"viewport\" content=\"width=device-width, initial-scale=1.0\">\n",
       "    <title>Kotlin Operators Periodic Table</title>\n",
       "    <style>\n",
       "        body {\n",
       "            font-family: Arial, sans-serif;\n",
       "            margin: 20px;\n",
       "            background-color: #f0f0f0;\n",
       "        }\n",
       "        \n",
       "        h1 {\n",
       "            color: #333;\n",
       "            text-align: center;\n",
       "        }\n",
       "\n",
       "        .periodic-table {\n",
       "            width: 100%;\n",
       "            border-collapse: collapse;\n",
       "            margin: 20px auto;\n",
       "            box-shadow: 0 0 10px rgba(0,0,0,0.1);\n",
       "        }\n",
       "\n",
       "        th, td {\n",
       "            padding: 15px;\n",
       "            text-align: left;\n",
       "            border-bottom: 1px solid #ddd;\n",
       "        }\n",
       "\n",
       "        .highest-precedence {\n",
       "            background-color: #ffeb3b;\n",
       "            color: #333;\n",
       "        }\n",
       "\n",
       "        .high-precedence {\n",
       "            background-color: #90EE90;\n",
       "            color: #333;\n",
       "        }\n",
       "\n",
       "        .medium-precedence {\n",
       "            background-color: #87CEEB;\n",
       "            color: #333;\n",
       "        }\n",
       "\n",
       "        .low-precedence {\n",
       "            background-color: #fff5e6;\n",
       "            color: #333;\n",
       "        }\n",
       "\n",
       "        .lowest-precedence {\n",
       "            background-color: #f0f0f0;\n",
       "            color: #333;\n",
       "        }\n",
       "\n",
       "        .example-box {\n",
       "            background-color: #f8f9fa;\n",
       "            padding: 10px;\n",
       "            border-radius: 4px;\n",
       "            margin-top: 5px;\n",
       "        }\n",
       "    </style>\n",
       "</head>\n",
       "<body>\n",
       "    <h1>Kotlin Operators Periodic Table</h1>\n",
       "    \n",
       "    <p>This table presents Kotlin operators in a periodic table format, organized by precedence levels. Each entry includes the operator symbol, description, and an example.</p>\n",
       "\n",
       "    <table class=\"periodic-table\">\n",
       "        <tr>\n",
       "            <th>Precedence Level</th>\n",
       "            <th>Operator Symbol</th>\n",
       "            <th>Description</th>\n",
       "            <th>Example</th>\n",
       "        </tr>\n",
       "        \n",
       "        <tr class=\"highest-precedence\">\n",
       "            <td>Level 1</td>\n",
       "            <td><strong>++</strong></td>\n",
       "            <td>Increment operator (prefix)</td>\n",
       "            <td class=\"example-box\">val a = 5<br>println(a++) // Outputs 5 then increments to 6</td>\n",
       "        </tr>\n",
       "\n",
       "        <tr class=\"high-precedence\">\n",
       "            <td>Level 2</td>\n",
       "            <td><strong>-</strong></td>\n",
       "            <td>Unary minus</td>\n",
       "            <td class=\"example-box\">val b = -5<br>// Represents negative five</td>\n",
       "        </tr>\n",
       "\n",
       "        <tr class=\"medium-precedence\">\n",
       "            <td>Level 3</td>\n",
       "            <td><strong>*</strong></td>\n",
       "            <td>Multiplication</td>\n",
       "            <td class=\"example-box\">val c = 2 * 3<br>// Outputs 6</td>\n",
       "        </tr>\n",
       "\n",
       "        <tr class=\"low-precedence\">\n",
       "            <td>Level 4</td>\n",
       "            <td><strong>+</strong></td>\n",
       "            <td>Addition</td>\n",
       "            <td class=\"example-box\">val d = 2 + 3<br>// Outputs 5</td>\n",
       "        </tr>\n",
       "\n",
       "        <tr class=\"lowest-precedence\">\n",
       "            <td>Level 5</td>\n",
       "            <td><strong>=</strong></td>\n",
       "            <td>Assignment</td>\n",
       "            <td class=\"example-box\">var x = 10<br>// Assigns value 10 to x</td>\n",
       "        </tr>\n",
       "\n",
       "        <!-- Add more rows for other operators -->\n",
       "    </table>\n",
       "\n",
       "    <p class=\"note\"><em>Notes:</em></p>\n",
       "    <ul>\n",
       "        <li>Operators with the same precedence are evaluated left-to-right</li>\n",
       "        <li>Parentheses can override default precedence rules</li>\n",
       "    </ul>\n",
       "</body>\n",
       "</html>\n"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "execution_count": 8
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": [
    "Local models offer an excellent alternative when you need privacy, offline capability, or want to avoid API costs.\n",
    "With Spring-AI, the programming interface remains consistent whether you're using cloud-based or local models,\n",
    "making it easy to switch between them as your needs change.\n",
    "\n",
    "You can apply all the techniques from our previous tutorials (prompts, streaming, tools, structured outputs, advisors, RAG) with local models as well —\n",
    "the same Spring-AI abstractions work across providers."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Kotlin",
   "language": "kotlin",
   "name": "kotlin"
  },
  "language_info": {
   "name": "kotlin",
   "version": "1.9.23",
   "mimetype": "text/x-kotlin",
   "file_extension": ".kt",
   "pygments_lexer": "kotlin",
   "codemirror_mode": "text/x-kotlin",
   "nbconvert_exporter": ""
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}