inference/ollama_infer.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "e6679051-2243-4d2d-b52a-d49cf69942c3",
   "metadata": {},
   "source": [
    "## 安装ollama"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "31d0975c-54cb-4d46-8ef6-f116a8bd13d9",
   "metadata": {},
   "outputs": [],
   "source": [
    "curl -fsSL https://ollama.com/install.sh | sh"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7384e09e-a05e-4869-ab07-810b5a058f93",
   "metadata": {},
   "source": [
    "## 基本命令\n",
    "\n",
    "> 当在没有桌面应用程序上运行ollama的时候请先运行`ollama serve`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "13a58eb4-2461-435d-9450-1cef0d86fc08",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 拉取模型\n",
    "ollama pull qwen2.5\n",
    "\n",
    "# 修改模型名字\n",
    "ollama cp qwen2.5 gpt-4\n",
    "\n",
    "# 运行模型\n",
    "ollama run qwen2.5\n",
    "\n",
    "# 推理单个prompt\n",
    "ollama run qwen2.5 \"help me write a story.\"\n",
    "\n",
    "# 带指标运行\n",
    "ollama run qwen2.5 \"help me write a story.\" --verbose\n",
    "\n",
    "# 删除模型\n",
    "ollama rm qwen2.5\n",
    "\n",
    "# 展示模型信息\n",
    "ollama show qwen2.5\n",
    "\n",
    "# 列出电脑上存在的模型\n",
    "ollama list\n",
    "ollama ls\n",
    "\n",
    "# 查看正在运行的模型\n",
    "ollama ps\n",
    "\n",
    "# 停止运行模型\n",
    "ollama stop qwen2.5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "59d10377-4eb7-41ff-bd9d-c9a5abb1ad25",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 请求使用ollama api\n",
    "# api接口信息https://github.com/ollama/ollama/blob/main/docs/api.md\n",
    "curl http://localhost:11434/api/generate -d '{\n",
    "  \"model\": \"qwen2.5\",\n",
    "  \"prompt\":\"Why is the sky blue?\",\n",
    "  \"stream\": false\n",
    "}'\n",
    "\n",
    "curl http://localhost:11434/api/chat -d '{\n",
    "  \"model\": \"qwen2.5\",\n",
    "  \"messages\": [\n",
    "    { \"role\": \"user\", \"content\": \"why is the sky blue?\" }\n",
    "  ],\n",
    "  \"stream\": false\n",
    "}'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7f63aaeb-1fc6-40f3-8c31-28e4607ace86",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 保证多次请求结果尽可能是一样的\n",
    "curl http://localhost:11434/api/generate -d '{\n",
    "  \"model\": \"qwen2.5:0.5b\",\n",
    "  \"prompt\":\"天空为什么是蓝色\",\n",
    "  \"stream\": false,\n",
    "  \"options\": {\n",
    "      \"seed\": 123,\n",
    "      \"temperature\": 0\n",
    "  }\n",
    "}'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "cc972c34-9d98-425d-9d90-58a08d04df23",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The sky appears blue because it reflects sunlight and light from above. The Earth, as we see it from space, is located at the center of the solar system, with the Sun shining down on it during the day and being blocked off by other celestial bodies at night. When the sun disappears below the horizon, the darkness forms a shadow that casts shadows onto the ground. These shadows have their own colors, which are often referred to as \"fog.\" The blue color of the sky is due to the reflection of sunlight from these shadows.\n",
      "\n",
      "The air also plays an important role in reflecting and scattering light. When we look at the Earth's surface, objects that appear darker than usual are actually not really dark but are just too small or out of focus for our eyes to see clearly. Similarly, bright colors can be seen because they reflect more light back into our eyes, making them easier to notice.\n",
      "\n",
      "So in summary, it is correct to say that the sky appears blue because it reflects sunlight and light from above, causing us to perceive colors through our eye's ability to differentiate between different wavelengths of visible light.\n"
     ]
    }
   ],
   "source": [
    "# python接口对话\n",
    "import ollama\n",
    " \n",
    "response = ollama.chat(model='qwen2.5', messages=[\n",
    "  {\n",
    "    'role': 'user',\n",
    "    'content': 'Why is the sky blue?',\n",
    "  },\n",
    "])\n",
    "print(response['message']['content'])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "c34948e1-2413-4595-b2f1-bd7af14467d0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "天象学家们普遍认为，我们看到的颜色来自于光在大气中的散射。当太阳光线照射到地球表面时，它会穿透大气层进入我们的视线。在这个过程中，大部分颜色（如红色、绿色和黄色）会被散射到空气以外的地方，而蓝色的光由于其波长较短，在大气中被散射得最多，所以我们看到天空是蓝色的。\n",
      "\n",
      "此外，我们知道太阳看起来像一个红点，这是因为太阳光线通过稀薄的大气层时，光线会因为折射和反射而向各个方向分发。这些光线在我们看来就像一条带状的红色光束，这就是为什么我们看见太阳是红色的原因。\n",
      "\n",
      "请注意，对于某些特定情况下（如日食、月食或强光环境），天空的颜色可能会受到其他因素的影响。"
     ]
    }
   ],
   "source": [
    "# 流式python接口对话\n",
    "import ollama\n",
    "\n",
    "stream = ollama.chat(\n",
    "    model='qwen2.5',\n",
    "    messages=[{'role': 'user', 'content': '为什么天空是蓝色的？'}],\n",
    "    stream=True,\n",
    ")\n",
    "\n",
    "for chunk in stream:\n",
    "  print(chunk['message']['content'], end='', flush=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "89afc417-e960-4353-a3c4-e74e26ffdb65",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "天空之所以呈现蔚蓝，是因为太阳光照射到地球大气层上时，太阳光中的紫外线和红外线等成分被分散成各种颜色的光。这些颜色随着距离和温度的变化而变化，当太阳光照在大范围内时，会呈现出红、橙、黄、绿、青、蓝、紫七种颜色。\n",
      "\n",
      "蓝色是属于晴天的日落时间范围，此时太阳的光线到达地球大气层后，大部分阳光被折射进入高空，并且由于空气密度差异导致反射光线的强度减弱，因此天空看起来是蔚蓝的。而当太阳光照在地面时，它则会照亮周围的环境和建筑物，使得天空显得更加明亮。\n",
      "\n",
      "另外，我们观察到一些特定的“雾”现象，比如云朵的雾、浮云上的雾等，通常它们呈现出的是白色或淡灰色，并且会遮挡掉更多的蓝色光。所以，当太阳照射在这些特殊的天体上时，我们会看到更接近白色的天空。\n",
      "\n",
      "总之，天空呈现为蓝色是因为太阳光通过大气层进入地面后，被分散成七种颜色的光线，而这些光线又与云朵、雾等特殊天体的形态变化有关。\n"
     ]
    }
   ],
   "source": [
    "import ollama\n",
    " \n",
    "response = ollama.generate(model='qwen2.5', \n",
    "                           prompt='为什么天空是蓝色的？')\n",
    "\n",
    "print(response['response'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "718e550a-e0ae-4d22-a24e-d18cfd80de45",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'embedding': [0.27271202206611633, 0.5691584944725037, 0.40936174988746643, 0.05066325515508652, -0.08940620720386505, 0.4723708927631378, 1.0018465518951416, -0.26635169982910156, 0.21395941078662872, -0.4298495352268219, 0.5982128381729126, -0.4286032021045685, -0.026222366839647293, -0.4238537847995758, -0.05918879806995392, -0.33097073435783386, -0.021263379603624344, -0.05248946696519852, -0.35049498081207275, 0.029436204582452774, 0.07253152132034302, 0.10874532163143158, 0.023502811789512634, 0.49305805563926697, -0.2993105351924896, 0.027715805917978287, 0.20971010625362396, -0.0297342948615551, 0.056742772459983826, 0.0344640389084816, -0.1760476678609848, 0.6093783378601074, -0.17544004321098328, 0.020226823166012764, 0.14467453956604004, 0.29832616448402405, -0.02842816896736622, -0.10476450622081757, -0.22009208798408508, 0.0004442967474460602, -0.13644203543663025, -0.17567402124404907, 0.3449799716472626, -0.06442157924175262, -0.15037637948989868, -0.02132883109152317, -0.28770115971565247, 0.04319564253091812, 0.42920419573783875, -0.07597281038761139, -0.25178810954093933, 0.022366628050804138, -0.4321227967739105, -0.2505331039428711, 0.6278479099273682, 0.4611020088195801, -0.057756464928388596, -0.20766815543174744, -0.165151908993721, -0.08907727897167206, -0.40908437967300415, 0.30312108993530273, 0.4106566607952118, 0.02797730267047882, 0.20191985368728638, -0.10624654591083527, 0.362199068069458, -0.029218889772892, 0.2751997411251068, -0.09992596507072449, -0.2479369193315506, 0.05013201758265495, 0.0772266834974289, -0.26060450077056885, -0.486531138420105, -0.33852508664131165, 0.1001940369606018, -0.011304751969873905, -0.022771120071411133, -0.3130466043949127, 0.10737912356853485, -0.0338311642408371, -0.011382649652659893, 0.425618052482605, -0.2769486606121063, 0.16834744811058044, -0.12617000937461853, 0.058972813189029694, 0.007806415669620037, 0.14530375599861145, -0.033582963049411774, 0.005082598887383938, -0.41341322660446167, -0.07146604359149933, -0.5436571836471558, 0.281854510307312, -0.3547723889350891, -0.6648226380348206, -0.2206607311964035, -0.10946326702833176, 0.4719463586807251, 0.1509571075439453, 0.0206752922385931, 0.2508767545223236, -0.07067127525806427, -0.23003527522087097, -0.06070879101753235, -0.43332862854003906, 0.008957470767199993, 0.12741349637508392, -0.3462328016757965, -0.5650070905685425, 0.011444859206676483, -0.5046390891075134, -0.1278580278158188, 0.3538094162940979, -0.010490121319890022, -0.1418067216873169, -0.47473520040512085, 0.04121401160955429, -0.3674679398536682, 0.28709876537323, -0.1513494849205017, -0.16564644873142242, -0.32496118545532227, -0.1503523737192154, 0.5586214065551758, -3.260124114094194e-33, -0.022601954638957977, -0.2569267749786377, 0.1042589321732521, 0.058822568506002426, -0.3950172960758209, -0.06787247955799103, -0.07994364202022552, 0.16485700011253357, -0.14362838864326477, 0.13166263699531555, -0.18173012137413025, -0.3159809708595276, -0.2180246114730835, 0.10201624035835266, 0.16810911893844604, -0.08895545452833176, 0.0012072697281837463, -0.5218627452850342, 0.5766334533691406, -0.031646229326725006, -0.2581489384174347, -0.0574064776301384, -0.18573933839797974, -0.3857046663761139, 0.4028427302837372, -0.47218000888824463, -0.08640758693218231, 0.15438251197338104, -0.10863590985536575, 0.19219675660133362, -0.15767867863178253, 0.13134443759918213, -0.025839127600193024, 0.21651145815849304, -0.33881130814552307, 0.23750518262386322, 0.08255010843276978, 0.506920576095581, 0.06908285617828369, 0.17962853610515594, 0.012580156326293945, -0.11760948598384857, -0.46378761529922485, -0.39625340700149536, 0.04678143560886383, 0.2762201428413391, -0.0718059092760086, -0.2850745916366577, 0.20058128237724304, -0.25616002082824707, 0.05026509612798691, -0.0389181524515152, -0.20231619477272034, 0.12776853144168854, 0.39595311880111694, -0.10088621079921722, -0.1988855004310608, 0.008614546619355679, -0.4802107512950897, 0.020668117329478264, 0.17432615160942078, -0.5293053984642029, -0.3314419984817505, 0.2212177813053131, 0.12244445085525513, 0.2738058269023895, 0.03404765576124191, -0.24780741333961487, 0.1737576425075531, -0.23151540756225586, -0.45163631439208984, -0.23660022020339966, 0.47614428400993347, 0.1165703535079956, -0.28473713994026184, -0.037947241216897964, -0.3418933153152466, 0.18018250167369843, 0.2062923163175583, -0.0826176106929779, -0.365668386220932, -0.07351649552583694, 0.4798714518547058, 0.36678045988082886, 0.3794291317462921, 0.18607375025749207, 0.3817995488643646, 0.20063573122024536, -0.02943519316613674, -0.04108364135026932, -0.8194882273674011, -0.15730327367782593, 0.7511117458343506, -0.007069846615195274, -0.08908247947692871, -1.4630584725998208e-32, -0.11949238926172256, 0.3344869017601013, -0.38356518745422363, 0.019279390573501587, 0.016213255003094673, -0.08192373067140579, -0.14215883612632751, 0.5262813568115234, -0.25357571244239807, 0.30555665493011475, -0.6406610012054443, -0.16439563035964966, 0.18576669692993164, -0.21063417196273804, 0.01629011519253254, 0.18344739079475403, 0.235393226146698, 0.4179987907409668, -0.1526259332895279, 0.07119445502758026, -0.04164832830429077, -0.4782084822654724, -0.275020569562912, -0.14502932131290436, -0.2213897705078125, 0.21741952002048492, 0.612690806388855, -0.11729775369167328, -0.18218150734901428, -0.18962615728378296, 0.053631290793418884, -0.06970011442899704, -0.04473157599568367, 0.6185312867164612, -0.14504608511924744, -0.22439102828502655, 0.31229478120803833, -0.3794853091239929, -0.42768216133117676, -0.026680950075387955, -0.16278794407844543, 0.06646822392940521, 0.29114410281181335, -0.3349377512931824, -0.012453880161046982, 0.31975194811820984, -0.29713138937950134, -0.06382126361131668, -0.37785184383392334, 0.04346800595521927, 0.0861940085887909, -0.3020162880420685, 0.13058623671531677, -0.07740481197834015, -0.04624481499195099, 0.33176013827323914, -0.3027704060077667, -0.11895087361335754, 0.6012225151062012, -0.0888625755906105, 0.09052659571170807, -0.12722671031951904, -0.10407731682062149, 0.17715322971343994, 0.10407436639070511, 0.04463955760002136, 0.05162995681166649, 0.23339787125587463, 0.26999393105506897, 0.11762157827615738, 0.3828250765800476, -0.0834888368844986, -0.15479540824890137, 0.250304639339447, -0.558667778968811, 0.4056349992752075, -0.09705990552902222, 0.061676111072301865, 0.06925006955862045, 0.1884245127439499, 0.2433377355337143, -0.08983486145734787, 0.06083691865205765, 0.05205162614583969, -0.1036667674779892, -0.27932679653167725, 0.047667697072029114, 0.4351609945297241, 0.05494795739650726, -0.1053638607263565, 0.29186341166496277, 0.04867056384682655, -0.2764197885990143, -0.37713390588760376, 0.07557125389575958, -9.933628319913623e-08, 0.013717427849769592, -0.14381536841392517, -0.44643634557724, -0.10554074496030807, -0.05239609628915787, 0.03516552224755287, -0.06345576792955399, 0.3254408538341522, 0.29291149973869324, 0.029561961069703102, 0.27122926712036133, 0.15255625545978546, -0.3901978135108948, 0.6658909320831299, -0.3139336109161377, 0.04256432503461838, -0.18456445634365082, 0.013730719685554504, 0.0961199551820755, -0.3095851540565491, 0.22904258966445923, 0.019145160913467407, -0.0036747592967003584, -0.3687795400619507, -0.2943710386753082, 0.3238690197467804, -0.23328301310539246, 0.16944073140621185, -0.11051733791828156, 0.25354254245758057, -0.09970809519290924, 0.014375744387507439, -0.2204597145318985, 0.05124708265066147, -0.320590615272522, -0.08951099961996078, 0.5367712378501892, -0.055038146674633026, 0.04565365985035896, 0.053182199597358704, 0.45361268520355225, 0.19456537067890167, 0.3227885365486145, 0.009924978017807007, 0.5890240669250488, -0.3515245318412781, 0.6329891085624695, 0.11257694661617279, -0.3065791428089142, 0.46436405181884766, -0.24058575928211212, 0.5899039506912231, 0.20957231521606445, -0.289259135723114, -0.31943032145500183, 0.33084815740585327, -0.28958413004875183, 0.3923909664154053, -0.0507710799574852, -0.20188847184181213, -0.018883051350712776, 0.3719806671142578, -0.5049054026603699, -0.07154994457960129]}\n"
     ]
    }
   ],
   "source": [
    "import ollama\n",
    " \n",
    "response = ollama.embeddings(model='all-minilm', \n",
    "                           prompt='为什么天空是蓝色的？')\n",
    "\n",
    "print(response)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "7706c0fe-db1b-4850-9a09-e8ece5dfafd3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The sky appears to be blue because it reflects light from the sun and other celestial bodies bouncing off the Earth's surface.\n",
      "The Earth's atmosphere, which extends below the ground, bounces incoming light waves back toward us at angles between 5 degrees and 10 degrees. This phenomenon is known as absorption or scattering of sunlight. These rays are bent by air molecules around the world causing blue vision (which we perceive as sky-blue color).\n",
      "The angle that these rays strike on our retina makes their wavelengths longer, which causes shorter infrared light to bounce off them back to our eyes as heat which turns it red. Therefore, the sun's radiation cannot reach the earth without being absorbed by atmospheric layers.\n",
      "As a result of this reflection and scattering of solar radiations into our atmosphere, we perceive the sky blue color.\n"
     ]
    }
   ],
   "source": [
    "# OpenAI接口对话\n",
    "from openai import OpenAI\n",
    " \n",
    "client = OpenAI(\n",
    "    base_url = 'http://localhost:11434/v1',\n",
    "    api_key='ollama', # required, but unused\n",
    ")\n",
    " \n",
    "response = client.chat.completions.create(\n",
    "  model=\"qwen2.5:0.5b\",\n",
    "  messages=[\n",
    "    {\"role\": \"system\", \"content\": \"You are a helpful assistant.\"},\n",
    "    {\"role\": \"user\", \"content\": \"Why is the sky blue?\"}\n",
    "  ]\n",
    "  # 带有历史对话\n",
    "  # messages=[\n",
    "  #   {\"role\": \"system\", \"content\": \"You are a helpful assistant.\"},\n",
    "  #   {\"role\": \"user\", \"content\": \"Why is the sky blue?\"},\n",
    "  #   {\"role\": \"assistant\", \"content\": \"The LA Dodgers won in 2020.\"},\n",
    "  #   {\"role\": \"user\", \"content\": \"Where was it played?\"}\n",
    "  # ]\n",
    ")\n",
    "print(response.choices[0].message.content)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d152f6e8-bfff-4464-b29c-6683b556f435",
   "metadata": {},
   "outputs": [],
   "source": [
    "curl http://localhost:11434/api/embeddings -d '{\n",
    "  \"model\": \"all-minilm\",\n",
    "  \"prompt\": \"You are a helpful assistant.\"\n",
    "}'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "b5524d17-9410-4ea1-b19f-bc2cee5f4fea",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'model': 'qwen2.5:0.5b', 'created_at': '2024-10-10T08:53:44.299573916Z', 'message': {'role': 'assistant', 'content': '这是因为地球大气中的二氧化碳和其他气体分子散射太阳光。当阳光照射到云层上，光线穿过空气进入我们的视线时，不同颜色的光在到达我们的眼睛时会经历不同的折射和散射。蓝色光波长较短，在空气中更容易被散射，因此我们在天空中看到的是蓝色的。'}, 'done_reason': 'stop', 'done': True, 'total_duration': 509740784, 'load_duration': 20605047, 'prompt_eval_count': 35, 'prompt_eval_duration': 27575000, 'eval_count': 69, 'eval_duration': 454847000}\n"
     ]
    }
   ],
   "source": [
    "from ollama import Client\n",
    "client = Client(host='http://localhost:11434')\n",
    "\n",
    "response = client.chat(model='qwen2.5:0.5b', messages=[\n",
    "  {\n",
    "    'role': 'user',\n",
    "    'content': '为什么天空是蓝色的？',\n",
    "  },\n",
    "])\n",
    "\n",
    "print(response)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "82014058-69e2-48e0-b927-edd6e7531eb3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'model': 'qwen2.5:0.5b', 'created_at': '2024-10-10T08:52:19.954623531Z', 'message': {'role': 'assistant', 'content': '因为地球表面大部分被海洋覆盖，而海洋中的水分子能够吸收并反射太阳光中的蓝、绿和紫光。这些波长的光线在穿过大气层时会遇到各种反射、折射和散射现象，使得天边看起来呈现出蓝色的色调。此外，天空中还有许多云朵、浮尘和其他小颗粒，也会影响我们对蓝天的感知。'}, 'done_reason': 'stop', 'done': True, 'total_duration': 591470999, 'load_duration': 20614090, 'prompt_eval_count': 35, 'prompt_eval_duration': 25634000, 'eval_count': 81, 'eval_duration': 538285000}\n"
     ]
    }
   ],
   "source": [
    "# 异步调用\n",
    "import asyncio\n",
    "from ollama import AsyncClient\n",
    "import nest_asyncio\n",
    "\n",
    "nest_asyncio.apply()\n",
    "\n",
    "async def chat():\n",
    "    message = {'role': 'user', 'content': '为什么天空是蓝色的？'}\n",
    "    response = await AsyncClient().chat(model='qwen2.5:0.5b', messages=[message])\n",
    "    print(response)\n",
    "\n",
    "asyncio.run(chat())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "17f76dc5-bccf-4841-a92f-3286d197a5e4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "因为太阳和星星发出的光线照射到地球表面，使得地球上的物体看起来被照亮。地球大气层中的气体，如氮气、氧气和水蒸气等，在阳光的照射下分散成各种颜色的光谱，即红橙黄色、蓝绿青色、紫红外等，使天空呈现出蓝色。"
     ]
    }
   ],
   "source": [
    "# 流式异步调用\n",
    "import asyncio\n",
    "from ollama import AsyncClient\n",
    "import nest_asyncio\n",
    "\n",
    "nest_asyncio.apply()\n",
    "async def chat():\n",
    "  message = {'role': 'user', 'content': '为什么天空是蓝色的？'}\n",
    "  async for part in await AsyncClient().chat(model='qwen2.5:0.5b', messages=[message], stream=True):\n",
    "    print(part['message']['content'], end='', flush=True)\n",
    "\n",
    "asyncio.run(chat())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9167252f-cc52-4001-a85c-afe878eecdfb",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 更多参考\n",
    "from openai import OpenAI\n",
    "\n",
    "client = OpenAI(\n",
    "    base_url='http://localhost:11434/v1/',\n",
    "\n",
    "    # required but ignored\n",
    "    api_key='ollama',\n",
    ")\n",
    "\n",
    "chat_completion = client.chat.completions.create(\n",
    "    messages=[\n",
    "        {\n",
    "            'role': 'user',\n",
    "            'content': 'Say this is a test',\n",
    "        }\n",
    "    ],\n",
    "    model='llama3.2',\n",
    ")\n",
    "\n",
    "response = client.chat.completions.create(\n",
    "    model=\"llava\",\n",
    "    messages=[\n",
    "        {\n",
    "            \"role\": \"user\",\n",
    "            \"content\": [\n",
    "                {\"type\": \"text\", \"text\": \"What's in this image?\"},\n",
    "                {\n",
    "                    \"type\": \"image_url\",\n",
    "                    \"image_url\": \"iVBORw0KGgoAAAANSUhEUgAAAG0AAABmCAYAAADBPx+VAAAACXBIWXMAAAsTAAALEwEAmpwYAAAAAXNSR0IArs4c6QAAAARnQU1BAACxjwv8YQUAAA3VSURBVHgB7Z27r0zdG8fX743i1bi1ikMoFMQloXRpKFFIqI7LH4BEQ+NWIkjQuSWCRIEoULk0gsK1kCBI0IhrQVT7tz/7zZo888yz1r7MnDl7z5xvsjkzs2fP3uu71nNfa7lkAsm7d++Sffv2JbNmzUqcc8m0adOSzZs3Z+/XES4ZckAWJEGWPiCxjsQNLWmQsWjRIpMseaxcuTKpG/7HP27I8P79e7dq1ars/yL4/v27S0ejqwv+cUOGEGGpKHR37tzJCEpHV9tnT58+dXXCJDdECBE2Ojrqjh071hpNECjx4cMHVycM1Uhbv359B2F79+51586daxN/+pyRkRFXKyRDAqxEp4yMlDDzXG1NPnnyJKkThoK0VFd1ELZu3TrzXKxKfW7dMBQ6bcuWLW2v0VlHjx41z717927ba22U9APcw7Nnz1oGEPeL3m3p2mTAYYnFmMOMXybPPXv2bNIPpFZr1NHn4HMw0KRBjg9NuRw95s8PEcz/6DZELQd/09C9QGq5RsmSRybqkwHGjh07OsJSsYYm3ijPpyHzoiacg35MLdDSIS/O1yM778jOTwYUkKNHWUzUWaOsylE00MyI0fcnOwIdjvtNdW/HZwNLGg+sR1kMepSNJXmIwxBZiG8tDTpEZzKg0GItNsosY8USkxDhD0Rinuiko2gfL/RbiD2LZAjU9zKQJj8RDR0vJBR1/Phx9+PHj9Z7REF4nTZkxzX4LCXHrV271qXkBAPGfP/atWvu/PnzHe4C97F48eIsRLZ9+3a3f/9+87dwP1JxaF7/3r17ba+5l4EcaVo0lj3SBq5kGTJSQmLWMjgYNei2GPT1MuMqGTDEFHzeQSP2wi/jGnkmPJ/nhccs44jvDAxpVcxnq0F6eT8h4ni/iIWpR5lPyA6ETkNXoSukvpJAD3AsXLiwpZs49+fPn5ke4j10TqYvegSfn0OnafC+Tv9ooA/JPkgQysqQNBzagXY55nO/oa1F7qvIPWkRL12WRpMWUvpVDYmxAPehxWSe8ZEXL20sadYIozfmNch4QJPAfeJgW3rNsnzphBKNJM2KKODo1rVOMRYik5ETy3ix4qWNI81qAAirizgMIc+yhTytx0JWZuNI03qsrgWlGtwjoS9XwgUhWGyhUaRZZQNNIEwCiXD16tXcAHUs79co0vSD8rrJCIW98pzvxpAWyyo3HYwqS0+H0BjStClcZJT5coMm6D2LOF8TolGJtK9fvyZpyiC5ePFi9nc/oJU4eiEP0jVoAnHa9wyJycITMP78+eMeP37sXrx44d6+fdt6f82aNdkx1pg9e3Zb5W+RSRE+n+VjksQWifvVaTKFhn5O8my63K8Qabdv33b379/PiAP//vuvW7BggZszZ072/+TJk91YgkafPn166zXB1rQHFvouAWHq9z3SEevSUerqCn2/dDCeta2jxYbr69evk4MHDyY7d+7MjhMnTiTPnz9Pfv/+nfQT2ggpO2dMF8cghuoM7Ygj5iWCqRlGFml0QC/ftGmTmzt3rmsaKDsgBSPh0/8yPeLLBihLkOKJc0jp8H8vUzcxIA1k6QJ/c78tWEyj5P3o4u9+jywNPdJi5rAH9x0KHcl4Hg570eQp3+vHXGyrmEeigzQsQsjavXt38ujRo44LQuDDhw+TW7duRS1HGgMxhNXHgflaNTOsHyKvHK5Ijo2jbFjJBQK9YwFd6RVMzfgRBmEfP37suBBm/p49e1qjEP2mwTViNRo0VJWH1deMXcNK08uUjVUu7s/zRaL+oLNxz1bpANco4npUgX4G2eFbpDFyQoQxojBCpEGSytmOH8qrH5Q9vuzD6ofQylkCUmh8DBAr+q8JCyVNtWQIidKQE9wNtLSQnS4jDSsxNHogzFuQBw4cyM61UKVsjfr3ooBkPSqqQHesUPWVtzi9/vQi1T+rJj7WiTz4Pt/l3LxUkr5P2VYZaZ4URpsE+st/dujQoaBBYokbrz/8TJNQYLSonrPS9kUaSkPeZyj1AWSj+d+VBoy1pIWVNed8P0Ll/ee5HdGRhrHhR5GGN0r4LGZBaj8oFDJitBTJzIZgFcmU0Y8ytWMZMzJOaXUSrUs5RxKnrxmbb5YXO9VGUhtpXldhEUogFr3IzIsvlpmdosVcGVGXFWp2oU9kLFL3dEkSz6NHEY1sjSRdIuDFWEhd8KxFqsRi1uM/nz9/zpxnwlESONdg6dKlbsaMGS4EHFHtjFIDHwKOo46l4TxSuxgDzi+rE2jg+BaFruOX4HXa0Nnf1lwAPufZeF8/r6zD97WK2qFnGjBxTw5qNGPxT+5T/r7/7RawFC3j4vTp09koCxkeHjqbHJqArmH5UrFKKksnxrK7FuRIs8STfBZv+luugXZ2pR/pP9Ois4z+TiMzUUkUjD0iEi1fzX8GmXyuxUBRcaUfykV0YZnlJGKQpOiGB76x5GeWkWWJc3mOrK6S7xdND+W5N6XyaRgtWJFe13GkaZnKOsYqGdOVVVbGupsyA/l7emTLHi7vwTdirNEt0qxnzAvBFcnQF16xh/TMpUuXHDowhlA9vQVraQhkudRdzOnK+04ZSP3DUhVSP61YsaLtd/ks7ZgtPcXqPqEafHkdqa84X6aCeL7YWlv6edGFHb+ZFICPlljHhg0bKuk0CSvVznWsotRu433alNdFrqG45ejoaPCaUkWERpLXjzFL2Rpllp7PJU2a/v7Ab8N05/9t27Z16KUqoFGsxnI9EosS2niSYg9SpU6B4JgTrvVW1flt1sT+0ADIJU2maXzcUTraGCRaL1Wp9rUMk16PMom8QhruxzvZIegJjFU7LLCePfS8uaQdPny4jTTL0dbee5mYokQsXTIWNY46kuMbnt8Kmec+LGWtOVIl9cT1rCB0V8WqkjAsRwta93TbwNYoGKsUSChN44lgBNCoHLHzquYKrU6qZ8lolCIN0Rh6cP0Q3U6I6IXILYOQI513hJaSKAorFpuHXJNfVlpRtmYBk1Su1obZr5dnKAO+L10Hrj3WZW+E3qh6IszE37F6EB+68mGpvKm4eb9bFrlzrok7fvr0Kfv727dvWRmdVTJHw0qiiCUSZ6wCK+7XL/AcsgNyL74DQQ730sv78Su7+t/A36MdY0sW5o40ahslXr58aZ5HtZB8GH64m9EmMZ7FpYw4T6QnrZfgenrhFxaSiSGXtPnz57e9TkNZLvTjeqhr734CNtrK41L40sUQckmj1lGKQ0rC37x544r8eNXRpnVE3ZZY7zXo8NomiO0ZUCj2uHz58rbXoZ6gc0uA+F6ZeKS/jhRDUq8MKrTho9fEkihMmhxtBI1DxKFY9XLpVcSkfoi8JGnToZO5sU5aiDQIW716ddt7ZLYtMQlhECdBGXZZMWldY5BHm5xgAroWj4C0hbYkSc/jBmggIrXJWlZM6pSETsEPGqZOndr2uuuR5rF169a2HoHPdurUKZM4CO1WTPqaDaAd+GFGKdIQkxAn9RuEWcTRyN2KSUgiSgF5aWzPTeA/lN5rZubMmR2bE4SIC4nJoltgAV/dVefZm72AtctUCJU2CMJ327hxY9t7EHbkyJFseq+EJSY16RPo3Dkq1kkr7+q0bNmyDuLQcZBEPYmHVdOBiJyIlrRDq41YPWfXOxUysi5fvtyaj+2BpcnsUV/oSoEMOk2CQGlr4ckhBwaetBhjCwH0ZHtJROPJkyc7UjcYLDjmrH7ADTEBXFfOYmB0k9oYBOjJ8b4aOYSe7QkKcYhFlq3QYLQhSidNmtS2RATwy8YOM3EQJsUjKiaWZ+vZToUQgzhkHXudb/PW5YMHD9yZM2faPsMwoc7RciYJXbGuBqJ1UIGKKLv915jsvgtJxCZDubdXr165mzdvtr1Hz5LONA8jrUwKPqsmVesKa49S3Q4WxmRPUEYdTjgiUcfUwLx589ySJUva3oMkP6IYddq6HMS4o55xBJBUeRjzfa4Zdeg56QZ43LhxoyPo7Lf1kNt7oO8wWAbNwaYjIv5lhyS7kRf96dvm5Jah8vfvX3flyhX35cuX6HfzFHOToS1H4BenCaHvO8pr8iDuwoUL7tevX+b5ZdbBair0xkFIlFDlW4ZknEClsp/TzXyAKVOmmHWFVSbDNw1l1+4f90U6IY/q4V27dpnE9bJ+v87QEydjqx/UamVVPRG+mwkNTYN+9tjkwzEx+atCm/X9WvWtDtAb68Wy9LXa1UmvCDDIpPkyOQ5ZwSzJ4jMrvFcr0rSjOUh+GcT4LSg5ugkW1Io0/SCDQBojh0hPlaJdah+tkVYrnTZowP8iq1F1TgMBBauufyB33x1v+NWFYmT5KmppgHC+NkAgbmRkpD3yn9QIseXymoTQFGQmIOKTxiZIWpvAatenVqRVXf2nTrAWMsPnKrMZHz6bJq5jvce6QK8J1cQNgKxlJapMPdZSR64/UivS9NztpkVEdKcrs5alhhWP9NeqlfWopzhZScI6QxseegZRGeg5a8C3Re1Mfl1ScP36ddcUaMuv24iOJtz7sbUjTS4qBvKmstYJoUauiuD3k5qhyr7QdUHMeCgLa1Ear9NquemdXgmum4fvJ6w1lqsuDhNrg1qSpleJK7K3TF0Q2jSd94uSZ60kK1e3qyVpQK6PVWXp2/FC3mp6jBhKKOiY2h3gtUV64TWM6wDETRPLDfSakXmH3w8g9Jlug8ZtTt4kVF0kLUYYmCCtD/DrQ5YhMGbA9L3ucdjh0y8kOHW5gU/VEEmJTcL4Pz/f7mgoAbYkAAAAAElFTkSuQmCC\",\n",
    "                },\n",
    "            ],\n",
    "        }\n",
    "    ],\n",
    "    max_tokens=300,\n",
    ")\n",
    "\n",
    "completion = client.completions.create(\n",
    "    model=\"llama3.2\",\n",
    "    prompt=\"Say this is a test\",\n",
    ")\n",
    "\n",
    "list_completion = client.models.list()\n",
    "\n",
    "model = client.models.retrieve(\"llama3.2\")\n",
    "\n",
    "embeddings = client.embeddings.create(\n",
    "    model=\"all-minilm\",\n",
    "    input=[\"why is the sky blue?\", \"why is the grass green?\"],\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "967e37b5-fa23-474d-b72e-ae02a4adfef8",
   "metadata": {},
   "source": [
    "## 并发请求"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cb1bb41f-e926-4591-8d85-664d6075228b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 单个模型同时处理多个请求\n",
    "OLLAMA_NUM_PARALLEL=2 ollama serve\n",
    "\n",
    "# 同时加载多个模型\n",
    "OLLAMA_MAX_LOADED_MODELS=4 ollama serve\n",
    "\n",
    "# 同时使用\n",
    "OLLAMA_NUM_PARALLEL=4 OLLAMA_MAX_LOADED_MODELS=4 ollama serve\n",
    "\n",
    "# 注意\n",
    "# 以上不是必须的，随着ollama的不断升级，他在不断默认支持这些功能"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}