02_using_transformers.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "02_using_transformers.ipynb",
      "provenance": [],
      "collapsed_sections": [
        "OEidjmFMuBh-",
        "6Xw5zHfnvrN0",
        "ZuF4q3pBLB2B",
        "0wWhshrwUct9",
        "e1nDzI1P1rkZ",
        "P9UT0NkVXqly",
        "uWJDc5a0AurP",
        "R1dgQS2rEd8h",
        "jKDVNtV1I_mr",
        "SZdox8iDh9s0",
        "MVCoyQOvio-v",
        "NnFdxvjSlaG5",
        "LPtt_OPRyw4w"
      ]
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1mUI8lDUxtENZFHlhkKMz1e4sp-NNUuKD?usp=sharing)"
      ],
      "metadata": {
        "id": "FtRNS344DBgi"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "R8M4jBBPGnQu"
      },
      "source": [
        "# Chapter 2. Using 🤗Transformers\n",
        "\n",
        "## What Happens Inside the pipeline Function?\n",
        "\n",
        "The ```pipeline()``` groups together three steps:\n",
        "- preprocessing (i.e., tokenizing and creating an e\n",
        "- passing the inputs to the model\n",
        "- postprocessing "
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install transformers"
      ],
      "metadata": {
        "id": "fJrMkmC5tLns",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "577ee63e-285c-4e53-95df-35d4bb8bf68c"
      },
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Collecting transformers\n",
            "  Downloading transformers-4.15.0-py3-none-any.whl (3.4 MB)\n",
            "\u001b[K     |████████████████████████████████| 3.4 MB 4.1 MB/s \n",
            "\u001b[?25hRequirement already satisfied: filelock in /usr/local/lib/python3.7/dist-packages (from transformers) (3.4.2)\n",
            "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.7/dist-packages (from transformers) (2019.12.20)\n",
            "Collecting tokenizers<0.11,>=0.10.1\n",
            "  Downloading tokenizers-0.10.3-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (3.3 MB)\n",
            "\u001b[K     |████████████████████████████████| 3.3 MB 36.4 MB/s \n",
            "\u001b[?25hRequirement already satisfied: importlib-metadata in /usr/local/lib/python3.7/dist-packages (from transformers) (4.10.0)\n",
            "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.7/dist-packages (from transformers) (21.3)\n",
            "Collecting sacremoses\n",
            "  Downloading sacremoses-0.0.47-py2.py3-none-any.whl (895 kB)\n",
            "\u001b[K     |████████████████████████████████| 895 kB 62.1 MB/s \n",
            "\u001b[?25hRequirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from transformers) (2.23.0)\n",
            "Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.7/dist-packages (from transformers) (1.19.5)\n",
            "Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.7/dist-packages (from transformers) (4.62.3)\n",
            "Collecting pyyaml>=5.1\n",
            "  Downloading PyYAML-6.0-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (596 kB)\n",
            "\u001b[K     |████████████████████████████████| 596 kB 60.5 MB/s \n",
            "\u001b[?25hCollecting huggingface-hub<1.0,>=0.1.0\n",
            "  Downloading huggingface_hub-0.4.0-py3-none-any.whl (67 kB)\n",
            "\u001b[K     |████████████████████████████████| 67 kB 4.6 MB/s \n",
            "\u001b[?25hRequirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.7/dist-packages (from huggingface-hub<1.0,>=0.1.0->transformers) (3.10.0.2)\n",
            "Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from packaging>=20.0->transformers) (3.0.6)\n",
            "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata->transformers) (3.7.0)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests->transformers) (2021.10.8)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests->transformers) (2.10)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests->transformers) (3.0.4)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests->transformers) (1.24.3)\n",
            "Requirement already satisfied: click in /usr/local/lib/python3.7/dist-packages (from sacremoses->transformers) (7.1.2)\n",
            "Requirement already satisfied: joblib in /usr/local/lib/python3.7/dist-packages (from sacremoses->transformers) (1.1.0)\n",
            "Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from sacremoses->transformers) (1.15.0)\n",
            "Installing collected packages: pyyaml, tokenizers, sacremoses, huggingface-hub, transformers\n",
            "  Attempting uninstall: pyyaml\n",
            "    Found existing installation: PyYAML 3.13\n",
            "    Uninstalling PyYAML-3.13:\n",
            "      Successfully uninstalled PyYAML-3.13\n",
            "Successfully installed huggingface-hub-0.4.0 pyyaml-6.0 sacremoses-0.0.47 tokenizers-0.10.3 transformers-4.15.0\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "from transformers import pipeline\n",
        "\n",
        "classifier = pipeline(\"sentiment-analysis\")\n",
        "\n",
        "sample_sentences = [\n",
        "                    \"I've been waiting for HuggingFace course my whole life.\",\n",
        "                    \"I hate this so much!\"\n",
        "]\n",
        "\n",
        "classifier(sample_sentences)\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "gcPDjvXNtbMe",
        "outputId": "66f4d475-9ad3-4ac6-aa4e-eb2818840871"
      },
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "No model was supplied, defaulted to distilbert-base-uncased-finetuned-sst-2-english (https://huggingface.co/distilbert-base-uncased-finetuned-sst-2-english)\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "bce4de8aad0a48e890fe7bef01358f80",
              "version_minor": 0,
              "version_major": 2
            },
            "text/plain": [
              "Downloading:   0%|          | 0.00/629 [00:00<?, ?B/s]"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "27b1a6c2221e4d20922da96a9b94cdc9",
              "version_minor": 0,
              "version_major": 2
            },
            "text/plain": [
              "Downloading:   0%|          | 0.00/255M [00:00<?, ?B/s]"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "7b036a5dc6aa4c9f94f630cf0842fed4",
              "version_minor": 0,
              "version_major": 2
            },
            "text/plain": [
              "Downloading:   0%|          | 0.00/48.0 [00:00<?, ?B/s]"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "2d072d9246004286a0d0c0da336eea0a",
              "version_minor": 0,
              "version_major": 2
            },
            "text/plain": [
              "Downloading:   0%|          | 0.00/226k [00:00<?, ?B/s]"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[{'label': 'POSITIVE', 'score': 0.985034167766571},\n",
              " {'label': 'NEGATIVE', 'score': 0.9994558691978455}]"
            ]
          },
          "metadata": {},
          "execution_count": 2
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now let's go through each of the three steps one at a time.\n",
        "\n",
        "## Tokenizer \n",
        "\n",
        "Tokenizers serve three purposoes:\n",
        "- split the input into tokens\n",
        "- map the tokens to an integer\n",
        "- add padding and/or special characters (i.e., begin/end of sequence tokens) to the input\n",
        "\n",
        "**Key Point**: 🤗Transformer models only accept tensors as inputs so we have to specify the type of tensor we want with the ```return_tensor``` argument.  "
      ],
      "metadata": {
        "id": "OEidjmFMuBh-"
      }
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yoGQ0AHZGt3Y"
      },
      "source": [
        "# Tokenizer \n",
        "\n",
        "from transformers import AutoTokenizer"
      ],
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "R3_UWR6iHEYK"
      },
      "source": [
        "checkpoint = 'distilbert-base-uncased-finetuned-sst-2-english'\n",
        "tokenizer = AutoTokenizer.from_pretrained(checkpoint)"
      ],
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FfLtwNkrHpmp"
      },
      "source": [
        "raw_inputs = [\n",
        "              \"I've been waiting for a HuggingFace course my whole life.\", \n",
        "              \"I hate this so much!\",\n",
        "              ]\n",
        "\n",
        "inputs = tokenizer(raw_inputs, \n",
        "                   padding=True,\n",
        "                   truncation=True,\n",
        "                   return_tensors='pt')"
      ],
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "T-IeLWwdHeFQ",
        "outputId": "ce23a322-cc1f-4f45-f3e6-15f2ba8bc0c7"
      },
      "source": [
        "print(inputs)"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "{'input_ids': tensor([[  101,  1045,  1005,  2310,  2042,  3403,  2005,  1037, 17662, 12172,\n",
            "          2607,  2026,  2878,  2166,  1012,   102],\n",
            "        [  101,  1045,  5223,  2023,  2061,  2172,   999,   102,     0,     0,\n",
            "             0,     0,     0,     0,     0,     0]]), 'attention_mask': tensor([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1],\n",
            "        [1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0]])}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "inputs['attention_mask']"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "577oXB5QtOQn",
        "outputId": "80c2b203-aa1f-4ea1-8b85-63a03146c73f"
      },
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1],\n",
              "        [1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0]])"
            ]
          },
          "metadata": {},
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Key Point**: look at ```attention_mask``` above. See the zeros? Those are correlate to padding tokens so we are going to mask them so that model doesn't worry about them during training. \n",
        "\n",
        "## Going through the Model\n",
        "\n",
        "We downlaod our pretrained model the same way we did our tokenizer."
      ],
      "metadata": {
        "id": "6Xw5zHfnvrN0"
      }
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "UiAV2JCoITtC",
        "outputId": "f7690e6f-4bda-4781-b67b-65fe2dc689c5"
      },
      "source": [
        "from transformers import AutoModel\n",
        "\n",
        "checkpoint = 'distilbert-base-uncased-finetuned-sst-2-english'\n",
        "model = AutoModel.from_pretrained(checkpoint)\n",
        "outputs = model(**inputs)\n",
        "print(outputs.last_hidden_state.shape)"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Some weights of the model checkpoint at distilbert-base-uncased-finetuned-sst-2-english were not used when initializing DistilBertModel: ['classifier.weight', 'classifier.bias', 'pre_classifier.weight', 'pre_classifier.bias']\n",
            "- This IS expected if you are initializing DistilBertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing DistilBertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "torch.Size([2, 16, 768])\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dg_d8vmOJXgv"
      },
      "source": [
        "The code above outputs the *hidden states* aka *features* of the model.\n",
        "\n",
        "What does that mean? Basically, we're going to use those features as inputs for the head of the model. \n",
        "\n",
        "Now, while human heads can do many things, transformer heads can essentially do one thing REALLY well so, to actually solve our classification problem, we need a model with a **sequence classificaiton** head as opposed to . "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "znHY8AMrIvK5"
      },
      "source": [
        "from transformers import  AutoModelForSequenceClassification\n",
        "\n",
        "checkpoint = 'distilbert-base-uncased-finetuned-sst-2-english'\n",
        "model = AutoModelForSequenceClassification.from_pretrained(checkpoint)\n",
        "\n",
        "outputs = model(**inputs)"
      ],
      "execution_count": 9,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "kNh_su_aK7pj",
        "outputId": "55d7ab29-8493-469e-9e03-e191fcbaf30a"
      },
      "source": [
        "print(outputs.logits.shape)"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "torch.Size([2, 2])\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZuF4q3pBLB2B"
      },
      "source": [
        "Since we have two sentences with two labels, we get an output of two by two. \n",
        "\n",
        "## Postporcessing the output\n",
        "Unfortunatley, the output doesn't make any sense on their own because they are raw logits."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "R1XgPKNcK_bA",
        "outputId": "7e1621f0-b413-4ccd-9a09-780bafbb9ebb"
      },
      "source": [
        "outputs.logits"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[-1.5607,  1.6123],\n",
              "        [ 4.1692, -3.3464]], grad_fn=<AddmmBackward0>)"
            ]
          },
          "metadata": {},
          "execution_count": 11
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0VK1rHXRLgjJ"
      },
      "source": [
        "The question, of course, is what do thos logits actually mean? To answer that, we move on to Postprocessing. \n",
        "\n",
        "We add a SoftMax layer to the logits to get a probablity."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "_dXk0VwmL2dB",
        "outputId": "d5b53c58-6689-42c3-a7e2-6b1b301ece87"
      },
      "source": [
        "import torch\n",
        "\n",
        "predictions = torch.nn.functional.softmax(outputs.logits, dim=-1)\n",
        "\n",
        "print(predictions)"
      ],
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor([[4.0195e-02, 9.5980e-01],\n",
            "        [9.9946e-01, 5.4418e-04]], grad_fn=<SoftmaxBackward0>)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DYBUd3SdMYWI"
      },
      "source": [
        "The model predicted [.0402, 0.9598] and [0.9946, 0.0544] for the second. \n",
        "\n",
        "But what are the labels for those probablities?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "I3oVuGiwLeB1",
        "outputId": "98d58609-7d7b-4267-f4cb-516313a9e013"
      },
      "source": [
        "model.config.id2label"
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{0: 'NEGATIVE', 1: 'POSITIVE'}"
            ]
          },
          "metadata": {},
          "execution_count": 13
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Dj7ywxTiL0UX",
        "outputId": "9d33677a-d87f-4ba9-cf0c-f04382eafbd3"
      },
      "source": [
        "raw_inputs"
      ],
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[\"I've been waiting for a HuggingFace course my whole life.\",\n",
              " 'I hate this so much!']"
            ]
          },
          "metadata": {},
          "execution_count": 14
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XXcwAwB_Nr8N"
      },
      "source": [
        "So, for the first sentence, the model predicts with 96% confidence that it is positive while the second is predicted at nearly 100% as being negative."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0wWhshrwUct9"
      },
      "source": [
        "## Deep Dive on [Models](https://huggingface.co/course/chapter2/3?fw=pt)\n",
        "\n",
        "It's easy to load a model based on a checkpoint"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "R2DVcoIQNj8A"
      },
      "source": [
        "from transformers import AutoModel"
      ],
      "execution_count": 15,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6aRozDfFWC99"
      },
      "source": [
        "bert_checkpoint = 'bert-base-cased'\n",
        "gpt2_checkpoint = 'gpt2'\n",
        "bart_checkpoint = 'facebook/bart-base'"
      ],
      "execution_count": 16,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "vSvBT5qlUwCj",
        "outputId": "6b43ca02-313f-434e-fefb-4ddae960d316"
      },
      "source": [
        "bert_model = AutoModel.from_pretrained(bert_checkpoint)\n",
        "gpt_model = AutoModel.from_pretrained(gpt2_checkpoint)\n",
        "bart_model = AutoModel.from_pretrained(bart_checkpoint)\n",
        "\n",
        "print(type(bert_model))\n",
        "print(type(gpt_model))\n",
        "print(type(bart_model))"
      ],
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "21a8865437944d5595985a915cb8062a",
              "version_minor": 0,
              "version_major": 2
            },
            "text/plain": [
              "Downloading:   0%|          | 0.00/570 [00:00<?, ?B/s]"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "fb6c53f476eb4c4ead4d472a13fa5d57",
              "version_minor": 0,
              "version_major": 2
            },
            "text/plain": [
              "Downloading:   0%|          | 0.00/416M [00:00<?, ?B/s]"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Some weights of the model checkpoint at bert-base-cased were not used when initializing BertModel: ['cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.decoder.weight', 'cls.predictions.bias', 'cls.seq_relationship.bias', 'cls.seq_relationship.weight', 'cls.predictions.transform.dense.weight']\n",
            "- This IS expected if you are initializing BertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing BertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "45d1176f81d2411595cd6a8e3dd3e693",
              "version_minor": 0,
              "version_major": 2
            },
            "text/plain": [
              "Downloading:   0%|          | 0.00/665 [00:00<?, ?B/s]"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "6a3d97d83e674f91b97f74d60694221f",
              "version_minor": 0,
              "version_major": 2
            },
            "text/plain": [
              "Downloading:   0%|          | 0.00/523M [00:00<?, ?B/s]"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "1fa84f7562fa46588ee9a635005763fd",
              "version_minor": 0,
              "version_major": 2
            },
            "text/plain": [
              "Downloading:   0%|          | 0.00/1.65k [00:00<?, ?B/s]"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "bda3ceeb2719481aa34828e759a1d44a",
              "version_minor": 0,
              "version_major": 2
            },
            "text/plain": [
              "Downloading:   0%|          | 0.00/532M [00:00<?, ?B/s]"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "<class 'transformers.models.bert.modeling_bert.BertModel'>\n",
            "<class 'transformers.models.gpt2.modeling_gpt2.GPT2Model'>\n",
            "<class 'transformers.models.bart.modeling_bart.BartModel'>\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Zc_5KJsnVqYB"
      },
      "source": [
        "We need to load the correct config file or else our model will not run."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Kj6ySkIGVEvn"
      },
      "source": [
        "from transformers import  AutoConfig\n",
        "\n",
        "bert_config = AutoConfig.from_pretrained(bert_checkpoint)\n",
        "gpt_config = AutoConfig.from_pretrained(gpt2_checkpoint)\n",
        "bart_config = AutoConfig.from_pretrained(bart_checkpoint)"
      ],
      "execution_count": 18,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "J5Ep2LEfWquN",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "f71656e8-0cff-43fe-eb19-530476004303"
      },
      "source": [
        "print(type(bert_config))\n",
        "print(type(gpt_config))\n",
        "print(type(bart_config))"
      ],
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "<class 'transformers.models.bert.configuration_bert.BertConfig'>\n",
            "<class 'transformers.models.gpt2.configuration_gpt2.GPT2Config'>\n",
            "<class 'transformers.models.bart.configuration_bart.BartConfig'>\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Creating a Transformer\n",
        "\n",
        "We need to load the architecture and the config."
      ],
      "metadata": {
        "id": "e1nDzI1P1rkZ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from transformers import BertConfig, BertModel\n",
        "\n",
        "config = BertConfig()\n",
        "\n",
        "model = BertModel(config)\n",
        "\n",
        "print(config)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "CglPNFiH1tJ-",
        "outputId": "1f38f435-477f-483a-de0a-9a6e970ad6c6"
      },
      "execution_count": 20,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "BertConfig {\n",
            "  \"attention_probs_dropout_prob\": 0.1,\n",
            "  \"classifier_dropout\": null,\n",
            "  \"hidden_act\": \"gelu\",\n",
            "  \"hidden_dropout_prob\": 0.1,\n",
            "  \"hidden_size\": 768,\n",
            "  \"initializer_range\": 0.02,\n",
            "  \"intermediate_size\": 3072,\n",
            "  \"layer_norm_eps\": 1e-12,\n",
            "  \"max_position_embeddings\": 512,\n",
            "  \"model_type\": \"bert\",\n",
            "  \"num_attention_heads\": 12,\n",
            "  \"num_hidden_layers\": 12,\n",
            "  \"pad_token_id\": 0,\n",
            "  \"position_embedding_type\": \"absolute\",\n",
            "  \"transformers_version\": \"4.15.0\",\n",
            "  \"type_vocab_size\": 2,\n",
            "  \"use_cache\": true,\n",
            "  \"vocab_size\": 30522\n",
            "}\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GKAAGcVdXELy"
      },
      "source": [
        "It is also possible to just import the config for the desired snapshot like below."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cE1mJJW_W0q8",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "1e03debf-86d8-46cb-e6e8-553215dd4af4"
      },
      "source": [
        "from transformers import  BertConfig\n",
        "\n",
        "bert_config = BertConfig.from_pretrained(bert_checkpoint)\n",
        "print(type(bert_config))\n",
        "print(bert_config)"
      ],
      "execution_count": 21,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "<class 'transformers.models.bert.configuration_bert.BertConfig'>\n",
            "BertConfig {\n",
            "  \"architectures\": [\n",
            "    \"BertForMaskedLM\"\n",
            "  ],\n",
            "  \"attention_probs_dropout_prob\": 0.1,\n",
            "  \"classifier_dropout\": null,\n",
            "  \"gradient_checkpointing\": false,\n",
            "  \"hidden_act\": \"gelu\",\n",
            "  \"hidden_dropout_prob\": 0.1,\n",
            "  \"hidden_size\": 768,\n",
            "  \"initializer_range\": 0.02,\n",
            "  \"intermediate_size\": 3072,\n",
            "  \"layer_norm_eps\": 1e-12,\n",
            "  \"max_position_embeddings\": 512,\n",
            "  \"model_type\": \"bert\",\n",
            "  \"num_attention_heads\": 12,\n",
            "  \"num_hidden_layers\": 12,\n",
            "  \"pad_token_id\": 0,\n",
            "  \"position_embedding_type\": \"absolute\",\n",
            "  \"transformers_version\": \"4.15.0\",\n",
            "  \"type_vocab_size\": 2,\n",
            "  \"use_cache\": true,\n",
            "  \"vocab_size\": 28996\n",
            "}\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "P9UT0NkVXqly"
      },
      "source": [
        "__Key Point__: The config provides all the information necessary to load the model. Namely, it provides the information needed to create the archetecture.\n",
        "\n",
        "## Tokenizers\n",
        "Machine learning models can only work with numbers so we have to convert our raw text into numbers and that is exactly what our transformer tokenizers do: take raw text and convert them to useful pieces of information for our model to process. \n",
        "\n",
        "To get a better understanding of how tokenizers work, we'll go over three kinds: \n",
        "- word based\n",
        "- character based\n",
        "- subword based \n",
        "\n",
        "Which tokenizer you use will be based on your checkpoint which, of course, is based on the task you are performing. \n",
        "\n",
        "### Word Based\n",
        "Every word is treated as a unique token.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "tokenized_text = \"Jim Henson was a puppeteer\".split()\n",
        "print(tokenized_text)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ODXhAmHb8aMy",
        "outputId": "02876d9d-1f42-48aa-b811-436fcd49af05"
      },
      "execution_count": 22,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['Jim', 'Henson', 'was', 'a', 'puppeteer']\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Unfortunately, this means \"play\", \"played\", \"plays\", and \"playing\" are all considered unique words. Needless to say, this creates a HUGE model. \n",
        "\n",
        "To combat this issue, we can simply limit the size of our corpus by telling our model to only remember the top k number of words. The problem this practice gives rise to is that we now have several [unknown] words which all have the same meaning. \n",
        "\n",
        "One solution is to use, \n",
        "\n",
        "### Character-based Tokenizers\n",
        "\n",
        "Instead of mapping a number to every word, we'll map a number to every character. "
      ],
      "metadata": {
        "id": "uWJDc5a0AurP"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def split(word):\n",
        "  return[char for char in word if char.strip()]\n",
        "\n",
        "text = \"Jim Henson was a puppeteer\"\n",
        "\n",
        "print(split(text))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "tE144iovByVn",
        "outputId": "cba6634e-be5e-4934-bd5e-86b88ca08e0a"
      },
      "execution_count": 23,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['J', 'i', 'm', 'H', 'e', 'n', 's', 'o', 'n', 'w', 'a', 's', 'a', 'p', 'u', 'p', 'p', 'e', 't', 'e', 'e', 'r']\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "The new problem is quite obvious: indiviual characters in languages which use alphabets do not convey nearly as much meaning as words.\n",
        "\n",
        "Additionally, we now have to process an incredible number of tokens for our model. \n",
        "\n",
        "To that end, our last approach is a fusion to the two previously mentioned. \n",
        "\n",
        "### Subword tokenization\n",
        "\n",
        "**Key Point**: Frequently used words should not be split into subwords while less freqently words should. \n",
        "\n",
        "By following the above principle, we will avoid the pitfalls word-based tokenizers (i.e., having a huge number of unique tokens as well as unknown tokents) as well as the issue with character-based (i.e., having tokens which carry little meaning and having too many tokens for our model to process). \n",
        "\n",
        "Consequently, all state of the art (SOTA) architectures use a form of subword tokenization. \n",
        "\n",
        "## Loading and saving\n",
        "\n",
        "It's just like loading and saving models.\n",
        "\n",
        "We can load it directly: "
      ],
      "metadata": {
        "id": "R1dgQS2rEd8h"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from transformers import BertTokenizer\n",
        "\n",
        "tokenizer = BertTokenizer.from_pretrained('bert-base-cased')"
      ],
      "metadata": {
        "id": "I4iICv4FAhm7",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "6e45d69b-280a-41d0-c5a5-074e4b0d6baf"
      },
      "execution_count": 24,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "9df8f049f31a4c02976fcdb0066c9dca",
              "version_minor": 0,
              "version_major": 2
            },
            "text/plain": [
              "Downloading:   0%|          | 0.00/208k [00:00<?, ?B/s]"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "87ad12e34cee4b17a7ed7e47743f7627",
              "version_minor": 0,
              "version_major": 2
            },
            "text/plain": [
              "Downloading:   0%|          | 0.00/29.0 [00:00<?, ?B/s]"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "3a39aa37a8334d7e8bf731040cb8e8ac",
              "version_minor": 0,
              "version_major": 2
            },
            "text/plain": [
              "Downloading:   0%|          | 0.00/426k [00:00<?, ?B/s]"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Or we can use ```AutoTokenizer```"
      ],
      "metadata": {
        "id": "_0N1WMZOHljn"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from transformers import AutoTokenizer\n",
        "\n",
        "tokenizer2 = AutoTokenizer.from_pretrained(\"bert-base-cased\")"
      ],
      "metadata": {
        "id": "p1UdjGtAHcNt"
      },
      "execution_count": 25,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "sample_text = \"Using a Transformer network is simple.\""
      ],
      "metadata": {
        "id": "T7XE03JQH1Lp"
      },
      "execution_count": 26,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "tokenizer(sample_text)['input_ids'] == tokenizer2(sample_text)['input_ids']"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "xl9ns_V9Ice0",
        "outputId": "4a043882-bcb2-4e93-d3eb-c0e3bcff3e4a"
      },
      "execution_count": 27,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "metadata": {},
          "execution_count": 27
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Saving it is just as simple: "
      ],
      "metadata": {
        "id": "Wg6UytoxI2L9"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# tokenizer.save_pretrained(\"some_directory_somewhere\")"
      ],
      "metadata": {
        "id": "UPSyk-g4In14"
      },
      "execution_count": 28,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### [Encoding](https://huggingface.co/course/chapter2/4?fw=pt#encoding)\n",
        "\n",
        "Encoding is simply converting text to numbers. Since machines don't understand raw text, we have to map text to numbers and then pass those digits as inputs to our model.\n",
        "\n",
        "Therefore, we have to tokenize our raw text and then map these tokens to numbers which stored in our model's ```vocabulary```. \n",
        "\n",
        "### Tokenization "
      ],
      "metadata": {
        "id": "jKDVNtV1I_mr"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from transformers import AutoTokenizer\n",
        "\n",
        "tokenizer = AutoTokenizer.from_pretrained('bert-base-cased')\n",
        "\n",
        "sequence = \"Using a transformer network is piss easy.\"\n",
        "\n",
        "tokens = tokenizer.tokenize(sequence)\n",
        "\n",
        "print(tokens)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "x8A0A4lIgtYt",
        "outputId": "6987e56a-5438-43d0-efeb-1681bdd9e5ae"
      },
      "execution_count": 29,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['Using', 'a', 'transform', '##er', 'network', 'is', 'piss', 'easy', '.']\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Since we're using a Bert-based model tokenizer, our tokenizer splits on the subword in order to avoid ```UNK``` tokens.\n",
        "\n",
        "### From tokens to input ID's\n",
        "\n",
        "Now we can conver the tokens to numbers aka input IDs."
      ],
      "metadata": {
        "id": "SZdox8iDh9s0"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "ids = tokenizer.convert_tokens_to_ids(tokens)\n",
        "print(ids)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "WwD1U5emgtWM",
        "outputId": "63cebbc3-4a98-43f6-fd74-d871478ad737"
      },
      "execution_count": 30,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[7993, 170, 11303, 1200, 2443, 1110, 20693, 3123, 119]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Decoding \n",
        "Just like *decrypt* is the opposite of *incrypt*, *decode* is the opposite of *encode*. "
      ],
      "metadata": {
        "id": "MVCoyQOvio-v"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "decoded_string = tokenizer.decode(ids)\n",
        "print(decoded_string)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ukYKnSb4j9VI",
        "outputId": "fb6a15b7-0f03-49ad-ba37-55a433968abd"
      },
      "execution_count": 31,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Using a transformer network is piss easy.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Handling Multiple Sequnces\n",
        "\n",
        "### Modles expect a batch of inputs\n",
        "Of course, we're not going to be working with one sentence at a time so we need to work with batches of inputs. Luckily, transformers makes that dead simple and, infact, expect multiple sequences by default.\n"
      ],
      "metadata": {
        "id": "NnFdxvjSlaG5"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import torch \n",
        "from transformers import AutoTokenizer, AutoModelForSequenceClassification\n",
        "\n",
        "checkpoint = \"distilbert-base-uncased-finetuned-sst-2-english\"\n",
        "\n",
        "tokenizer = AutoTokenizer.from_pretrained(checkpoint)\n",
        "model = AutoModelForSequenceClassification.from_pretrained(checkpoint)\n",
        "\n",
        "sequence = \"I've been waiting for a HuggingFace course my whole life.\""
      ],
      "metadata": {
        "id": "j2AghdoykETY",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "5b3135aa-1f22-4631-e205-5a1867cd2846"
      },
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "bdb8a0f1298542119dc7b05ca17c7855",
              "version_minor": 0,
              "version_major": 2
            },
            "text/plain": [
              "Downloading:   0%|          | 0.00/48.0 [00:00<?, ?B/s]"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "8061d46a03ed45c2bffa8fa8763c6fd7",
              "version_minor": 0,
              "version_major": 2
            },
            "text/plain": [
              "Downloading:   0%|          | 0.00/629 [00:00<?, ?B/s]"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "2df35e334bf64cad99e68000a7513939",
              "version_minor": 0,
              "version_major": 2
            },
            "text/plain": [
              "Downloading:   0%|          | 0.00/226k [00:00<?, ?B/s]"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "b507431e07314f64ba8be1d2f8ff17e1",
              "version_minor": 0,
              "version_major": 2
            },
            "text/plain": [
              "Downloading:   0%|          | 0.00/255M [00:00<?, ?B/s]"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "tokens = tokenizer.tokenize(sequence)\n",
        "ids = tokenizer.convert_tokens_to_ids(tokens)\n",
        "input_ids = torch.tensor([ids])\n",
        "\n",
        "print(\"Input IDs:\", input_ids)\n",
        "\n",
        "output = model(input_ids)\n",
        "print(\"Logits:\", output.logits)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "iGjbirhKm2T8",
        "outputId": "b35361e9-5645-4456-ac79-8ef22db7eea9"
      },
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Input IDs: tensor([[ 1045,  1005,  2310,  2042,  3403,  2005,  1037, 17662, 12172,  2607,\n",
            "          2026,  2878,  2166,  1012]])\n",
            "Logits: tensor([[-2.7276,  2.8789]], grad_fn=<AddmmBackward0>)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Padding the inputs\n",
        "All sequences must be the same length meaning we either need to truncate or pad our sequences. \n",
        "\n",
        "To make this process super simple, we can just pass ```padding=True``` to our tokenizer. "
      ],
      "metadata": {
        "id": "LPtt_OPRyw4w"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from transformers import AutoTokenizer\n",
        "\n",
        "checkpoint = \"distilbert-base-uncased-finetuned-sst-2-english\"\n",
        "\n",
        "tokenizer = AutoTokenizer.from_pretrained(checkpoint)\n",
        "\n",
        "sentences = [\n",
        "             \"I've been waiting for this course for what seems like forever.\",\n",
        "             \"I can't stand this!\"\n",
        "]\n",
        "\n",
        "tokens = tokenizer(sentences, padding=True)"
      ],
      "metadata": {
        "id": "7Tw7wItQnEc7"
      },
      "execution_count": 12,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "for x in tokens['input_ids']:\n",
        "  print(x)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "yQla5Mad0Rmu",
        "outputId": "065b2877-eed7-48f0-8b91-38b597eb1c9d"
      },
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[101, 1045, 1005, 2310, 2042, 3403, 2005, 2023, 2607, 2005, 2054, 3849, 2066, 5091, 1012, 102]\n",
            "[101, 1045, 2064, 1005, 1056, 3233, 2023, 999, 102, 0, 0, 0, 0, 0, 0, 0]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "And how do we get the model to ignore the padding? We pass it an attention mask literally telling the model to pay attention to tokens marked 1 and ignore 0."
      ],
      "metadata": {
        "id": "yqlclrgL1vqN"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "for x in tokens['attention_mask']:\n",
        "  print(x)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "3lpnZP2O0dnP",
        "outputId": "14dc547a-c085-4534-8268-ba0541472119"
      },
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]\n",
            "[1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Putting it all together\n",
        "\n",
        "As hinted at above, we can trust the API to pad and/or truncate our inputs accordingly. Additionally, we can explicitly tell the API how it should be done like so: "
      ],
      "metadata": {
        "id": "K798pSuh3Vyg"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#pad to the length of the longest sentence\n",
        "model_inputs = tokenizer(sentences, padding='longest')\n",
        "\n",
        "#Max length of the model \n",
        "# model_inputs = tokenizer(sentences, padding='max_length', truncation=True)\n",
        "\n",
        "#Specified max lenth\n",
        "model_inputs = tokenizer(sentences, \n",
        "                         padding='max_length', \n",
        "                         max_length=8,\n",
        "                         truncation=True) "
      ],
      "metadata": {
        "id": "q8gh-pZl0upL"
      },
      "execution_count": 33,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "for x in model_inputs['input_ids']:\n",
        "  print(x)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "QmK_lsR04Lnv",
        "outputId": "68c8ba04-2c65-40ed-a95e-699581c73f1e"
      },
      "execution_count": 34,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[101, 1045, 1005, 2310, 2042, 3403, 2005, 102]\n",
            "[101, 1045, 2064, 1005, 1056, 3233, 2023, 102]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Additionally, we can specify the tensor type to return: NumPy (\"np\"), TensorFlow (\"tf\"), or PyTorch (\"pt\"). "
      ],
      "metadata": {
        "id": "IFcvRUVa5lBH"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "model_inputs = tokenizer(sentences, \n",
        "                         padding='max_length', \n",
        "                         max_length=8,\n",
        "                         truncation=True, \n",
        "                         return_tensors=\"pt\")"
      ],
      "metadata": {
        "id": "l7tSbrLh4hpH"
      },
      "execution_count": 35,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "for x in model_inputs['input_ids']:\n",
        "  print(x)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "nn-tPW4j6TDI",
        "outputId": "c93150f6-28ae-497f-dd10-ff33ef81a9e2"
      },
      "execution_count": 36,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor([ 101, 1045, 1005, 2310, 2042, 3403, 2005,  102])\n",
            "tensor([ 101, 1045, 2064, 1005, 1056, 3233, 2023,  102])\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "All together now! "
      ],
      "metadata": {
        "id": "vOwj-8L27_Aj"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import torch\n",
        "from transformers import AutoTokenizer, AutoModelForSequenceClassification\n",
        "\n",
        "checkpoint = \"distilbert-base-uncased-finetuned-sst-2-english\"\n",
        "tokenizer = AutoTokenizer.from_pretrained(checkpoint)\n",
        "model = AutoModelForSequenceClassification.from_pretrained(checkpoint)\n",
        "sequences = [\"I've been waiting for a HuggingFace course my whole life.\", \"So have I!\"]\n",
        "\n",
        "tokens = tokenizer(sequences, padding=True, truncation=True, return_tensors=\"pt\")\n",
        "\n",
        "output = model(**tokens)\n",
        "\n",
        "predictions = torch.nn.functional.softmax(output.logits, dim=-1)"
      ],
      "metadata": {
        "id": "Ixisdc2o6VZI"
      },
      "execution_count": 48,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "for prediction in predictions:\n",
        "  print(prediction)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "EUKmegby7Aiv",
        "outputId": "caffab9c-2eef-4e76-de48-714a062ec159"
      },
      "execution_count": 50,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor([0.0402, 0.9598], grad_fn=<UnbindBackward0>)\n",
            "tensor([5.3534e-04, 9.9946e-01], grad_fn=<UnbindBackward0>)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        ""
      ],
      "metadata": {
        "id": "ZmO3Ix_x7FXD"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}