llama-cpp-rag - final.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Retrieval Augmentation Generation (RAG) with LLAMA.CPP Quantized Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Install llama.cpp llama-cpp-python, chromadb\n",
    "In my previous video, I have shown how to build a quantized model from llama.cpp\n",
    "\n",
    "In this notebook, you will see how to do RAG on a quantied model so that you can query your documents.\n",
    "\n",
    "CMAKE_ARGS=\"-DLLAMA_CUBLAS=on\" FORCE_CMAKE=1 pip install llama-cpp-python==0.1.64 --no-cache-dir\n",
    "\n",
    "pip install chromadb "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Step 1: Instantiate an embed model which later will be used for storing data in the vector DB"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch import cuda\n",
    "from langchain.embeddings.huggingface import HuggingFaceEmbeddings\n",
    "\n",
    "embed_model_id = 'sentence-transformers/all-MiniLM-L6-v2'\n",
    "\n",
    "device = f'cuda:{cuda.current_device()}' if cuda.is_available() else 'cpu'\n",
    "\n",
    "embed_model = HuggingFaceEmbeddings(\n",
    "    model_name=embed_model_id,\n",
    "    model_kwargs={'device': device},\n",
    "    encode_kwargs={'device': device, 'batch_size': 32}\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Step 2: Process Custom Content into Chunks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "from langchain.document_loaders import WebBaseLoader\n",
    "\n",
    "##loader = WebBaseLoader(\"https://lilianweng.github.io/posts/2023-06-23-agent/\")\n",
    "loader = WebBaseLoader(\"https://www.quadratics.com/MLOPSimplified.html\")\n",
    "\n",
    "data = loader.load()\n",
    "\n",
    "from langchain.text_splitter import RecursiveCharacterTextSplitter\n",
    "\n",
    "text_splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=0)\n",
    "all_splits = text_splitter.split_documents(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Step 3: Store the custom content into a Vector DB (Chroma)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.vectorstores import Chroma\n",
    "from langchain.embeddings import GPT4AllEmbeddings\n",
    "\n",
    "vectorstore = Chroma.from_documents(documents=all_splits, embedding=embed_model)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Step 4: Set bindings for LLAMA.CPP quantized model and instantiate the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.embeddings import LlamaCppEmbeddings\n",
    "from langchain.llms import LlamaCpp\n",
    "from langchain.callbacks.manager import CallbackManager\n",
    "from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler\n",
    "n_gpu_layers = 32  # Metal set to 1 is enough.\n",
    "n_batch = 512  # Should be between 1 and n_ctx, consider the amount of RAM of your Apple Silicon Chip.\n",
    "callback_manager = CallbackManager([StreamingStdOutCallbackHandler()])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "ggml_init_cublas: found 2 CUDA devices:\n",
      "  Device 0: NVIDIA GeForce GTX 1080\n",
      "  Device 1: NVIDIA GeForce GTX 1080\n",
      "llama.cpp: loading model from /data/llama.cpp/models/llama-2-7b-chat/ggml-model-q4_0.bin\n",
      "llama_model_load_internal: format     = ggjt v3 (latest)\n",
      "llama_model_load_internal: n_vocab    = 32000\n",
      "llama_model_load_internal: n_ctx      = 2048\n",
      "llama_model_load_internal: n_embd     = 4096\n",
      "llama_model_load_internal: n_mult     = 256\n",
      "llama_model_load_internal: n_head     = 32\n",
      "llama_model_load_internal: n_layer    = 32\n",
      "llama_model_load_internal: n_rot      = 128\n",
      "llama_model_load_internal: ftype      = 2 (mostly Q4_0)\n",
      "llama_model_load_internal: n_ff       = 11008\n",
      "llama_model_load_internal: n_parts    = 1\n",
      "llama_model_load_internal: model size = 7B\n",
      "llama_model_load_internal: ggml ctx size =    0.07 MB\n",
      "llama_model_load_internal: using CUDA for GPU acceleration\n",
      "ggml_cuda_set_main_device: using device 0 (NVIDIA GeForce GTX 1080) as main device\n",
      "llama_model_load_internal: mem required  = 1964.94 MB (+ 1026.00 MB per state)\n",
      "llama_model_load_internal: allocating batch_size x 1 MB = 512 MB VRAM for the scratch buffer\n",
      "llama_model_load_internal: offloading 32 repeating layers to GPU\n",
      "llama_model_load_internal: offloaded 32/35 layers to GPU\n",
      "llama_model_load_internal: total VRAM used: 3987 MB\n",
      "..................................................................................................\n",
      "llama_init_from_file: kv self size  = 1024.00 MB\n"
     ]
    }
   ],
   "source": [
    "#llama = LlamaCppEmbeddings(model_path=\"/data/llama.cpp/models/llama-2-7b-chat/ggml-model-q4_0.bin\")\n",
    "llm = LlamaCpp(\n",
    "    model_path=\"/data/llama.cpp/models/llama-2-7b-chat/ggml-model-q4_0.bin\",\n",
    "    n_gpu_layers=n_gpu_layers,\n",
    "    n_batch=n_batch,\n",
    "    n_ctx=2048,\n",
    "    f16_kv=True,  # MUST set to True, otherwise you will run into problem after a couple of calls\n",
    "    callback_manager=callback_manager,\n",
    "    verbose=False,\n",
    ")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Step 5: Do a similarity search on the Vectordb to retrieve data related to the query"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Document(page_content='Training models at scale   \\n Data acquisition for exploratory analysis \\n Consistent interface for training and serving\\nDeployment to production\\nMonitoring the model performance\\n\\nSolutions Delivered:\\n\\r\\n                        Quadratic has built a set of accelerators for enabling Ml/AI Model Lifecycle as a MLOPS suite.  \\r\\n                         This platform enabled the customer to quickly build models, train and deploy in a repeatable fashion.\\nOutcomes', metadata={'description': '', 'language': 'en', 'source': 'https://www.quadratics.com/MLOPSimplified.html', 'title': 'Quadratics'}),\n",
       " Document(page_content='Quadratics\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\nHome\\n\\n\\n\\n\\r\\n              Competencies \\n\\n\\nML/AL Platforms \\nData Engineering \\nCloud Adoption\\n\\n\\n\\nGlobal Workforce\\n\\n\\nPlatforml\\n\\n\\nContact us\\n\\n\\n\\n\\n\\r\\n                         Insights \\n\\n\\nCase Studies\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\nMLOPs Simplified', metadata={'description': '', 'language': 'en', 'source': 'https://www.quadratics.com/MLOPSimplified.html', 'title': 'Quadratics'}),\n",
       " Document(page_content='Reduced model development time by 80%  \\n Reduced deployment costs by 90% \\n Consistent model development methodology across the data science teams with collaboration.\\nOver 200% ROI in less than one year\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\nAbout Us', metadata={'description': '', 'language': 'en', 'source': 'https://www.quadratics.com/MLOPSimplified.html', 'title': 'Quadratics'}),\n",
       " Document(page_content='However, the firm was running into problems regularly in delivering the solutions to production on time due to the following issues:', metadata={'description': '', 'language': 'en', 'source': 'https://www.quadratics.com/MLOPSimplified.html', 'title': 'Quadratics'})]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "question = \"what accelerators did quadratic build\"\n",
    "docs = vectorstore.similarity_search(question)\n",
    "#result = llm_chain(docs)\n",
    "docs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Step 6: Create a RAG pipeline to contextualize with the custom data and Query"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.chains import RetrievalQA\n",
    "\n",
    "rag_pipeline = RetrievalQA.from_chain_type(\n",
    "    llm=llm, chain_type='stuff',\n",
    "    retriever=vectorstore.as_retriever()\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " Quadratic built a set of accelerators for enabling Ml/AI Model Lifecycle as an MLOPS suite."
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'query': 'what accelerators did quadratic build',\n",
       " 'result': ' Quadratic built a set of accelerators for enabling Ml/AI Model Lifecycle as an MLOPS suite.'}"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rag_pipeline(\"what accelerators did quadratic build\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  Accelerators created by Quadratic enable Ml/AI Model Lifecycle as a MLOPS suite, enabling the customer to quickly build models, train and deploy in a repeatable fashion."
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'query': 'how do the accelerators built by Quadratic help their customers',\n",
       " 'result': '  Accelerators created by Quadratic enable Ml/AI Model Lifecycle as a MLOPS suite, enabling the customer to quickly build models, train and deploy in a repeatable fashion.'}"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rag_pipeline(\"how do the accelerators built by Quadratic help their customers\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "?\n",
      " nobody knows when or if quadratic will launch. the company has not provided any updates on its launch plans, and its website is no longer active.\n",
      "Quadratic was a startup that aimed to build a decentralized exchange (DEX) for non-fungible tokens (NFTs). The platform was designed to provide a more secure and reliable way of trading NFTs compared to traditional centralized exchanges. However, the project appears to have been abandoned, and no further information is available on its launch plans or development progress.\n",
      "Quadratic's conceptual design involved using smart contracts to enable decentralized trading of NFTs without the need for intermediaries. The platform was expected to offer a range of features, including support for multiple blockchain networks, an intuitive user interface, and automated liquidity provision through quadratic funding.\n",
      "While Quadratic's idea was innovative, it faced significant challenges in terms of scalability, security, and regulatory compliance. The decentralized exchange space is highly competitive, with several established players already operating in the market. Moreover, NFTs are still a relatively new concept, and their long-term viability and potential"
     ]
    },
    {
     "data": {
      "text/plain": [
       "\"?\\n nobody knows when or if quadratic will launch. the company has not provided any updates on its launch plans, and its website is no longer active.\\nQuadratic was a startup that aimed to build a decentralized exchange (DEX) for non-fungible tokens (NFTs). The platform was designed to provide a more secure and reliable way of trading NFTs compared to traditional centralized exchanges. However, the project appears to have been abandoned, and no further information is available on its launch plans or development progress.\\nQuadratic's conceptual design involved using smart contracts to enable decentralized trading of NFTs without the need for intermediaries. The platform was expected to offer a range of features, including support for multiple blockchain networks, an intuitive user interface, and automated liquidity provision through quadratic funding.\\nWhile Quadratic's idea was innovative, it faced significant challenges in terms of scalability, security, and regulatory compliance. The decentralized exchange space is highly competitive, with several established players already operating in the market. Moreover, NFTs are still a relatively new concept, and their long-term viability and potential\""
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "llm(\"what accelerators did quadratic build\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}