04_(개정)케라스로_Q&A_구현하기_With_SQUAD.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.9"
    },
    "colab": {
      "name": "04_(개정)케라스로_Q&A_구현하기_With_SQUAD.ipynb",
      "provenance": [],
      "include_colab_link": true
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/kimwoonggon/publicservant_AI/blob/master/04_(%EA%B0%9C%EC%A0%95)%EC%BC%80%EB%9D%BC%EC%8A%A4%EB%A1%9C_Q%26A_%EA%B5%AC%ED%98%84%ED%95%98%EA%B8%B0_With_SQUAD.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Fn7pKyD8dBEq",
        "colab_type": "text"
      },
      "source": [
        "<h1><strong>케라스로 Q&A 구현하기 With SQUAD</strong></h1>\n",
        "<h1><strong><u>Youtube 공무원AI</u></strong></h1>\n",
        "\n",
        "![대체 텍스트](https://i.imgur.com/XSuDmcC.png)\n",
        "\n",
        "이번 튜토리얼에서는 케라스와 BERT를 활용하여 <strong>SQUAD(Standford Question and Answering Dataset)</strong>을 실습해보고자 합니다.\n",
        "\n",
        "SQUAD는 문장과 질문(Question)을 입력하면 그에 해당하는 답(ANSWER)를 알려주는 문제입니다.  \n",
        "즉 AI가 영어 독해 문제를 풀어주는 것입니다.   \n",
        "Tensorflow나 PyTorch로 SQUAD를 구현하는 코드들은 인터넷에 많지만 초보자 입장에서는 코드를 봐도 구현하기가 상당히 어렵습니다. \n",
        "막상 코드를 돌려봐도 어떤 원리로 돌아가는지 알기 어렵습니다.  \n",
        " \n",
        "그래서 KERAS를 활용하여 쉽게 SQUAD를 구현해보고자 합니다.  \n",
        "본 튜토리얼은 1)SQUAD 이해 2) BERT INPUT 만들기 3) SQUAD 구현 4) SQUAD 예측 총 4단계로 구성되어 있습니다.  \n",
        "각 단계마다 이해하기 쉬운 설명을 곁들이도록 하겠습니다. \n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_LeBONl3Rt6J",
        "colab_type": "text"
      },
      "source": [
        "<h2><strong> 1. SQUAD 개념 이해</strong></h2>  \n",
        "SQUAD 문제는 아래와 같습니다.\n",
        "Question을 하면 AI가 Context를 읽고 Answer를 출력값으로 알려주게 됩니다.  \n",
        "처음에 나온 ANSWER는  AI가 예측한 정답입니다. 그리고 원래 정답은 Von Miller인데 왜 von miller ##r로 예측이 되었냐면, 실제로 예측한 것은 bert에 들어간 sentencepiece 토큰이 예측 된 것이기 때문입니다.  \n",
        "  \n",
        "\n",
        "![Imgur](https://i.imgur.com/sjf7vyj.png)  \n",
        "\n",
        "사실 SQUAD는 ANSWER를 다 예측하는 것이 아니라, ANSWER 중에서도 시작단어와 끝 단어만을 예측합니다. 시작과 끝을 알면 자연스럽게 가운데 위치한 글자들도 예측이 되는 것이겠지요. 그리고 SQUAD 문제를 풀기 위해서 BERT 알고리즘을 사용합니다.  \n",
        "  \n",
        "\n",
        "위 그림에서 SQUAD는 ANSWER를 다 예측하는 것이 아니라, ANSWER 중에서도 시작단어와 끝 단어만을 예측합니다. 시작과 끝을 알면 자연스럽게 가운데 위치한 글자들도 예측이 되는 것이겠지요. 그리고 SQUAD 문제를 풀기 위해서 BERT 알고리즘을 사용합니다.  \n",
        "  \n",
        "  ![imgur](https://i.imgur.com/xhbWa13.png)  \n",
        "  위 그림에서 버트 모형에 들어가는 인풋은 Question(질문)과 Paragraph(지문)입니다.  \n",
        "  QUESTION과 PARAGRAPH가 버트의 인풋으로 들어가면 아웃풋 값으로 처음 단어와 큰 단어를 알려주게 됩니다.  \n",
        "\n",
        "  Train Data의 생김새입니다. 총 87599개의 문장이 있습니다. Question과 Context를 인풋으로 받아서, 아웃풋으로 Text(정답)을 맞추게 됩니다.\n",
        "  ![Imgur](https://i.imgur.com/rzTXJ1W.png)\n",
        "\n",
        "  Test Data의 생김새입니다. 총 10,570개의 문장으로 이루어져 있으며, question과 context를 인풋으로 넣으면, 파인튜닝된 버트 모형이 text를 아웃풋 값으로 출력하게 됩니다.\n",
        "  ![Imgur](https://i.imgur.com/s9zqAvv.png)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TwFvMz0AwGJ-",
        "colab_type": "text"
      },
      "source": [
        "<h2><strong>2. BERT 사전학습 모형 가져오기</strong></h2>\n",
        "\n",
        "https://github.com/google-research/bert 에 접속하셔서 BERT-Base, Multilingual Cased: 104 languages, 12-layer, 768-hidden, 12-heads, 110M parameters 파일을 다운 받으시길 바랍니다.\n",
        "\n",
        "![대체 텍스트](https://i.imgur.com/RlutYyW.png)\n",
        "\n",
        "<p>사전 학습 모형을 다운 받으셨다면, Colab에서 활용하기 위해 구글 GDRIVE에 업로드 하시길 바랍니다.</p>\n",
        "\n",
        "![대체 텍스트](https://i.imgur.com/uM97zAQ.png)\n",
        "\n",
        "\n",
        "\n",
        "<p> 누구나 쉽게 실습할 수 있도록 구글 Colaboratory를 활용하였습니다<br> <u>런타임->런타임 유형 변경에서 GPU를 꼭 선택하시기 바랍니다.</u><br>\n",
        "그리고 데이터는 구글 G드라이브에 넣어 두었습니다. 사정에 맞게 폴더 경로를 변경하시기 바랍니다.</p>\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SwK1tYV_uayz",
        "colab_type": "code",
        "outputId": "c6b73e93-0c87-44df-d95d-bf092a4f0736",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 210
        }
      },
      "source": [
        "# wget을 활용해서 bert 모델 다운로드 가능\n",
        "!wget https://storage.googleapis.com/bert_models/2018_11_23/multi_cased_L-12_H-768_A-12.zip"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "--2020-02-03 15:10:13--  https://storage.googleapis.com/bert_models/2018_11_23/multi_cased_L-12_H-768_A-12.zip\n",
            "Resolving storage.googleapis.com (storage.googleapis.com)... 172.217.218.128, 2a00:1450:4013:c01::80\n",
            "Connecting to storage.googleapis.com (storage.googleapis.com)|172.217.218.128|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 662903077 (632M) [application/zip]\n",
            "Saving to: ‘multi_cased_L-12_H-768_A-12.zip’\n",
            "\n",
            "multi_cased_L-12_H- 100%[===================>] 632.19M  58.5MB/s    in 9.8s    \n",
            "\n",
            "2020-02-03 15:10:29 (64.6 MB/s) - ‘multi_cased_L-12_H-768_A-12.zip’ saved [662903077/662903077]\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Sx2147Ndq3lS",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import os\n",
        "if \"bert\" not in os.listdir():\n",
        "  os.makedirs(\"bert\")\n",
        "else:\n",
        "  pass\n",
        "\n",
        "import zipfile\n",
        "import shutil\n",
        "         \n",
        "bert_zip = zipfile.ZipFile('multi_cased_L-12_H-768_A-12.zip')\n",
        "bert_zip.extractall('bert')\n",
        " \n",
        "bert_zip.close()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3yaskz4Aujj1",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def copytree(src, dst, symlinks=False, ignore=None):\n",
        "    for item in os.listdir(src):\n",
        "        s = os.path.join(src, item)\n",
        "        d = os.path.join(dst, item)\n",
        "        if os.path.isdir(s):\n",
        "            shutil.copytree(s, d, symlinks, ignore)\n",
        "        else:\n",
        "            shutil.copy2(s, d)\n",
        "\n",
        "copytree(\"bert/multi_cased_L-12_H-768_A-12\", \"bert\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "F4a7LicBl40e",
        "colab_type": "code",
        "outputId": "a4a4d31e-739a-4c88-f003-5d9637f375c1",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 404
        }
      },
      "source": [
        "!wget https://raw.githubusercontent.com/nate-parrott/squad/master/data/train-v1.1.json\n",
        "!wget https://raw.githubusercontent.com/nate-parrott/squad/master/data/dev-v1.1.json"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "--2020-02-03 15:11:00--  https://raw.githubusercontent.com/nate-parrott/squad/master/data/train-v1.1.json\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.0.133, 151.101.64.133, 151.101.128.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.0.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 30288272 (29M) [text/plain]\n",
            "Saving to: ‘train-v1.1.json’\n",
            "\n",
            "\rtrain-v1.1.json       0%[                    ]       0  --.-KB/s               \rtrain-v1.1.json      44%[=======>            ]  12.84M  64.2MB/s               \rtrain-v1.1.json     100%[===================>]  28.88M   106MB/s    in 0.3s    \n",
            "\n",
            "2020-02-03 15:11:02 (106 MB/s) - ‘train-v1.1.json’ saved [30288272/30288272]\n",
            "\n",
            "--2020-02-03 15:11:05--  https://raw.githubusercontent.com/nate-parrott/squad/master/data/dev-v1.1.json\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.0.133, 151.101.64.133, 151.101.128.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.0.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 4854279 (4.6M) [text/plain]\n",
            "Saving to: ‘dev-v1.1.json’\n",
            "\n",
            "dev-v1.1.json       100%[===================>]   4.63M  --.-KB/s    in 0.08s   \n",
            "\n",
            "2020-02-03 15:11:05 (57.8 MB/s) - ‘dev-v1.1.json’ saved [4854279/4854279]\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zXWE_XTqmta6",
        "colab_type": "code",
        "outputId": "59ce8319-267e-4a76-ae47-bfcec34138b9",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 122
        }
      },
      "source": [
        "os.listdir()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['.config',\n",
              " 'multi_cased_L-12_H-768_A-12.zip',\n",
              " 'train-v1.1.json',\n",
              " 'bert',\n",
              " 'dev-v1.1.json',\n",
              " 'sample_data']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_Tkah2Z4n_z6",
        "colab_type": "text"
      },
      "source": [
        "\n",
        "##**구글 드라이브와 Colab을 연동합니다**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hPs7cH6On_Qa",
        "colab_type": "code",
        "outputId": "b846db3a-3292-47fa-f40c-98997797ec6c",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 125
        }
      },
      "source": [
        "import os\n",
        "from google.colab import drive\n",
        "drive.mount('/content/gdrive/')"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Go to this URL in a browser: https://accounts.google.com/o/oauth2/auth?client_id=947318989803-6bn6qk8qdgf4n4g3pfee6491hc0brc4i.apps.googleusercontent.com&redirect_uri=urn%3aietf%3awg%3aoauth%3a2.0%3aoob&response_type=code&scope=email%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdocs.test%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdrive%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdrive.photos.readonly%20https%3a%2f%2fwww.googleapis.com%2fauth%2fpeopleapi.readonly\n",
            "\n",
            "Enter your authorization code:\n",
            "··········\n",
            "Mounted at /content/gdrive/\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HQ6glcvV1h73",
        "colab_type": "text"
      },
      "source": [
        "<h2><strong>본격적으로 케라스와 버트를 활용하여 SQUAD 예측 모델을 만들어 보겠습니다.</strong></h2>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QVGCHkKN0TZ3",
        "colab_type": "text"
      },
      "source": [
        "텐서플로우, 판다스, 넘파이, 케라스 등 필요한 모듈들을 임포트합니다"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ffa---Ov8WfF",
        "colab_type": "code",
        "outputId": "1b4f5b3e-cf66-4543-da2b-83f87f41eb30",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 81
        }
      },
      "source": [
        "import tensorflow as tf\n",
        "\n",
        "import pandas as pd\n",
        "import numpy as np  \n",
        "import re\n",
        "import pickle\n",
        "\n",
        "import keras as keras\n",
        "from keras.models import load_model\n",
        "from keras import backend as K\n",
        "from keras import Input, Model\n",
        "from keras import optimizers\n",
        "\n",
        "from keras import backend as K\n",
        "from keras.layers import Layer\n",
        "\n",
        "import codecs\n",
        "from tqdm import tqdm\n",
        "import shutil\n",
        "import json"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<p style=\"color: red;\">\n",
              "The default version of TensorFlow in Colab will soon switch to TensorFlow 2.x.<br>\n",
              "We recommend you <a href=\"https://www.tensorflow.org/guide/migrate\" target=\"_blank\">upgrade</a> now \n",
              "or ensure your notebook will continue to use TensorFlow 1.x via the <code>%tensorflow_version 1.x</code> magic:\n",
              "<a href=\"https://colab.research.google.com/notebooks/tensorflow_version.ipynb\" target=\"_blank\">more info</a>.</p>\n"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "Using TensorFlow backend.\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bniU2r9PouUO",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import warnings\n",
        "import tensorflow as tf\n",
        "warnings.filterwarnings(action='ignore')\n",
        "os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3' \n",
        "tf.logging.set_verbosity(tf.logging.ERROR)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "A3bWhmOP0mOv",
        "colab_type": "text"
      },
      "source": [
        "케라스에서 Bert 활용을 쉽게 만들어주는 모듈 keras-bert를 설치합니다<br>그리고 Adam optimizer의 수정판인 keras-radam 모듈을 임포트합니다."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "09QnmSZXGbbD",
        "colab_type": "code",
        "outputId": "9f3b3ea9-0a0e-458a-a977-9ede30f44c1e",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "!pip install keras-bert\n",
        "!pip install keras-radam"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting keras-bert\n",
            "  Downloading https://files.pythonhosted.org/packages/2c/0f/cdc886c1018943ea62d3209bc964413d5aa9d0eb7e493abd8545be679294/keras-bert-0.81.0.tar.gz\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from keras-bert) (1.17.5)\n",
            "Requirement already satisfied: Keras in /usr/local/lib/python3.6/dist-packages (from keras-bert) (2.2.5)\n",
            "Collecting keras-transformer>=0.30.0\n",
            "  Downloading https://files.pythonhosted.org/packages/54/0c/fede535ac576c03863c44bf2e0bf051fe21f5e10103631b6b6236ae446f3/keras-transformer-0.32.0.tar.gz\n",
            "Requirement already satisfied: keras-applications>=1.0.8 in /usr/local/lib/python3.6/dist-packages (from Keras->keras-bert) (1.0.8)\n",
            "Requirement already satisfied: six>=1.9.0 in /usr/local/lib/python3.6/dist-packages (from Keras->keras-bert) (1.12.0)\n",
            "Requirement already satisfied: h5py in /usr/local/lib/python3.6/dist-packages (from Keras->keras-bert) (2.8.0)\n",
            "Requirement already satisfied: scipy>=0.14 in /usr/local/lib/python3.6/dist-packages (from Keras->keras-bert) (1.4.1)\n",
            "Requirement already satisfied: keras-preprocessing>=1.1.0 in /usr/local/lib/python3.6/dist-packages (from Keras->keras-bert) (1.1.0)\n",
            "Requirement already satisfied: pyyaml in /usr/local/lib/python3.6/dist-packages (from Keras->keras-bert) (3.13)\n",
            "Collecting keras-pos-embd>=0.10.0\n",
            "  Downloading https://files.pythonhosted.org/packages/09/70/b63ed8fc660da2bb6ae29b9895401c628da5740c048c190b5d7107cadd02/keras-pos-embd-0.11.0.tar.gz\n",
            "Collecting keras-multi-head>=0.22.0\n",
            "  Downloading https://files.pythonhosted.org/packages/40/3e/d0a64bb2ac5217928effe4507c26bbd19b86145d16a1948bc2d4f4c6338a/keras-multi-head-0.22.0.tar.gz\n",
            "Collecting keras-layer-normalization>=0.12.0\n",
            "  Downloading https://files.pythonhosted.org/packages/a4/0e/d1078df0494bac9ce1a67954e5380b6e7569668f0f3b50a9531c62c1fc4a/keras-layer-normalization-0.14.0.tar.gz\n",
            "Collecting keras-position-wise-feed-forward>=0.5.0\n",
            "  Downloading https://files.pythonhosted.org/packages/e3/59/f0faa1037c033059e7e9e7758e6c23b4d1c0772cd48de14c4b6fd4033ad5/keras-position-wise-feed-forward-0.6.0.tar.gz\n",
            "Collecting keras-embed-sim>=0.7.0\n",
            "  Downloading https://files.pythonhosted.org/packages/bc/20/735fd53f6896e2af63af47e212601c1b8a7a80d00b6126c388c9d1233892/keras-embed-sim-0.7.0.tar.gz\n",
            "Collecting keras-self-attention==0.41.0\n",
            "  Downloading https://files.pythonhosted.org/packages/1b/1c/01599219bef7266fa43b3316e4f55bcb487734d3bafdc60ffd564f3cfe29/keras-self-attention-0.41.0.tar.gz\n",
            "Building wheels for collected packages: keras-bert, keras-transformer, keras-pos-embd, keras-multi-head, keras-layer-normalization, keras-position-wise-feed-forward, keras-embed-sim, keras-self-attention\n",
            "  Building wheel for keras-bert (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for keras-bert: filename=keras_bert-0.81.0-cp36-none-any.whl size=37913 sha256=2b5b7409bfd51be2056b29d67b168db5d1283551bf7ccc83599c4e19f76f56ab\n",
            "  Stored in directory: /root/.cache/pip/wheels/bd/27/da/ffc2d573aa48b87440ec4f98bc7c992e3a2d899edb2d22ef9e\n",
            "  Building wheel for keras-transformer (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for keras-transformer: filename=keras_transformer-0.32.0-cp36-none-any.whl size=13266 sha256=8809c477c88b494a1fb303bbd82aefdfd48004fac36dac644e2d6b4e30d45f67\n",
            "  Stored in directory: /root/.cache/pip/wheels/62/f0/ce/82fa5d024d5ef8e263f26a50dcee23820efe245680ce9c922a\n",
            "  Building wheel for keras-pos-embd (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for keras-pos-embd: filename=keras_pos_embd-0.11.0-cp36-none-any.whl size=7553 sha256=b70650b15dc7374936aa9412784ac2f9c657d03d9bf04686d51cadcbc7133ccb\n",
            "  Stored in directory: /root/.cache/pip/wheels/5b/a1/a0/ce6b1d49ba1a9a76f592e70cf297b05c96bc9f418146761032\n",
            "  Building wheel for keras-multi-head (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for keras-multi-head: filename=keras_multi_head-0.22.0-cp36-none-any.whl size=15371 sha256=416356703fb468fa9227f5b365f52c1f771ceb4405bdb902004d7ac016fd8f1b\n",
            "  Stored in directory: /root/.cache/pip/wheels/bb/df/3f/81b36f41b66e6a9cd69224c70a737de2bb6b2f7feb3272c25e\n",
            "  Building wheel for keras-layer-normalization (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for keras-layer-normalization: filename=keras_layer_normalization-0.14.0-cp36-none-any.whl size=5268 sha256=190ae184b5cfb9eb64781188f114c5dd317e7ba4d7b4c80d04f4a63b348842fe\n",
            "  Stored in directory: /root/.cache/pip/wheels/54/80/22/a638a7d406fd155e507aa33d703e3fa2612b9eb7bb4f4fe667\n",
            "  Building wheel for keras-position-wise-feed-forward (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for keras-position-wise-feed-forward: filename=keras_position_wise_feed_forward-0.6.0-cp36-none-any.whl size=5624 sha256=5f60eae4a3ca3f6ef672a274d6d8c87bd8e7668863d60db3da0930530a8bcec9\n",
            "  Stored in directory: /root/.cache/pip/wheels/39/e2/e2/3514fef126a00574b13bc0b9e23891800158df3a3c19c96e3b\n",
            "  Building wheel for keras-embed-sim (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for keras-embed-sim: filename=keras_embed_sim-0.7.0-cp36-none-any.whl size=4676 sha256=b4ed62b20ed39d2db1885f0c2d351b7267f65794482b4d5ec475b432a04379fc\n",
            "  Stored in directory: /root/.cache/pip/wheels/d1/bc/b1/b0c45cee4ca2e6c86586b0218ffafe7f0703c6d07fdf049866\n",
            "  Building wheel for keras-self-attention (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for keras-self-attention: filename=keras_self_attention-0.41.0-cp36-none-any.whl size=17290 sha256=b2743160af2935d5374b4d2fc263f82943fe9690f696030e84f8b4abcfc217a7\n",
            "  Stored in directory: /root/.cache/pip/wheels/cc/dc/17/84258b27a04cd38ac91998abe148203720ca696186635db694\n",
            "Successfully built keras-bert keras-transformer keras-pos-embd keras-multi-head keras-layer-normalization keras-position-wise-feed-forward keras-embed-sim keras-self-attention\n",
            "Installing collected packages: keras-pos-embd, keras-self-attention, keras-multi-head, keras-layer-normalization, keras-position-wise-feed-forward, keras-embed-sim, keras-transformer, keras-bert\n",
            "Successfully installed keras-bert-0.81.0 keras-embed-sim-0.7.0 keras-layer-normalization-0.14.0 keras-multi-head-0.22.0 keras-pos-embd-0.11.0 keras-position-wise-feed-forward-0.6.0 keras-self-attention-0.41.0 keras-transformer-0.32.0\n",
            "Collecting keras-radam\n",
            "  Downloading https://files.pythonhosted.org/packages/46/8d/b83ccaa94253fbc920b21981f038393041d92236bb541751b98a66a2ac1d/keras-radam-0.15.0.tar.gz\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from keras-radam) (1.17.5)\n",
            "Requirement already satisfied: Keras in /usr/local/lib/python3.6/dist-packages (from keras-radam) (2.2.5)\n",
            "Requirement already satisfied: keras-applications>=1.0.8 in /usr/local/lib/python3.6/dist-packages (from Keras->keras-radam) (1.0.8)\n",
            "Requirement already satisfied: six>=1.9.0 in /usr/local/lib/python3.6/dist-packages (from Keras->keras-radam) (1.12.0)\n",
            "Requirement already satisfied: keras-preprocessing>=1.1.0 in /usr/local/lib/python3.6/dist-packages (from Keras->keras-radam) (1.1.0)\n",
            "Requirement already satisfied: pyyaml in /usr/local/lib/python3.6/dist-packages (from Keras->keras-radam) (3.13)\n",
            "Requirement already satisfied: h5py in /usr/local/lib/python3.6/dist-packages (from Keras->keras-radam) (2.8.0)\n",
            "Requirement already satisfied: scipy>=0.14 in /usr/local/lib/python3.6/dist-packages (from Keras->keras-radam) (1.4.1)\n",
            "Building wheels for collected packages: keras-radam\n",
            "  Building wheel for keras-radam (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for keras-radam: filename=keras_radam-0.15.0-cp36-none-any.whl size=14686 sha256=72cc0262b4c6b9821b9d840ecdd9c5607823a70f944acc7d8f2faefc213d99e2\n",
            "  Stored in directory: /root/.cache/pip/wheels/79/a0/c0/670b0a118e8f078539fafec7bd02eba0af921f745660c7f83f\n",
            "Successfully built keras-radam\n",
            "Installing collected packages: keras-radam\n",
            "Successfully installed keras-radam-0.15.0\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cyIoeOBH04lW",
        "colab_type": "text"
      },
      "source": [
        "keras-bert 라이브러리에서 버트 모형 활용에 필요한 모듈들을 임포트합니다"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FZv-5ALnGdwI",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from keras_bert import load_trained_model_from_checkpoint, load_vocabulary\n",
        "from keras_bert import Tokenizer\n",
        "from keras_bert import AdamWarmup, calc_train_steps\n",
        "\n",
        "from keras_radam import RAdam"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kVGK2j5tKJgS",
        "colab_type": "text"
      },
      "source": [
        "SQUAD JSON파일을 PANDAS DATAFRAME으로 만들어주는 함수를 정의합니다.  \n",
        "출처 : https://www.kaggle.com/sanjay11100/squad-stanford-q-a-json-to-pandas-dataframe"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zgWmFtiaIZQK",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def squad_json_to_dataframe_train(input_file_path, record_path = ['data','paragraphs','qas','answers'],\n",
        "                           verbose = 1):\n",
        "    \"\"\"\n",
        "    input_file_path: path to the squad json file.\n",
        "    record_path: path to deepest level in json file default value is\n",
        "    ['data','paragraphs','qas','answers']\n",
        "    verbose: 0 to suppress it default is 1\n",
        "    \"\"\"\n",
        "    if verbose:\n",
        "        print(\"Reading the json file\")    \n",
        "    file = json.loads(open(input_file_path).read())\n",
        "    if verbose:\n",
        "        print(\"processing...\")\n",
        "    # parsing different level's in the json file\n",
        "    js = pd.io.json.json_normalize(file , record_path )\n",
        "    m = pd.io.json.json_normalize(file, record_path[:-1] )\n",
        "    r = pd.io.json.json_normalize(file,record_path[:-2])\n",
        "    \n",
        "    #combining it into single dataframe\n",
        "    idx = np.repeat(r['context'].values, r.qas.str.len())\n",
        "    ndx  = np.repeat(m['id'].values,m['answers'].str.len())\n",
        "    m['context'] = idx\n",
        "    js['q_idx'] = ndx\n",
        "    main = pd.concat([ m[['id','question','context']].set_index('id'),js.set_index('q_idx')],1,sort=False).reset_index()\n",
        "    main['c_id'] = main['context'].factorize()[0]\n",
        "    if verbose:\n",
        "        print(\"shape of the dataframe is {}\".format(main.shape))\n",
        "        print(\"Done\")\n",
        "    return main"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Eoez6gHVGeP8",
        "colab_type": "text"
      },
      "source": [
        "SQUAD 데이터를 PANDAS DATAFRAME 형식으로 로드합니다."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9j2vCEzCMW4K",
        "colab_type": "code",
        "outputId": "ab5613d5-9062-439b-f119-2a7d19ce9894",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 87
        }
      },
      "source": [
        "train = squad_json_to_dataframe_train(\"train-v1.1.json\")"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Reading the json file\n",
            "processing...\n",
            "shape of the dataframe is (87599, 6)\n",
            "Done\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PnnEcb9mQxV1",
        "colab_type": "text"
      },
      "source": [
        "SQUAD 예측을 위한 훈련 데이터가 잘 로드되었습니다.  \n",
        "**question 칼럼이 질문, context 칼럼이 문장으로 인풋으로 들어갑니다.  \n",
        "아웃풋 값(정답)은 text 칼럼에서 시작 단어와 끝 단어 두 개 입니다. 예를 들어서, text 값이 Saint Bernadette Soubirous라면, 정답은 시작 단어인 Saint와 끝 단어인 Soubrious입니다.**  \n",
        "  \n",
        "그리고 SQUAD 문제의 특징은, 정답에 해당하는 아웃풋 값(text)이 context 안에 있다는 것입니다. 참고로 answer_start는 무시하셔도 됩니다. 왜냐하면 answer_start는 context 내에서 단어를 쪼갠 다음 쪼갠 것을 하나 하나 세어서 몇번째에 정답이 위치하는지를 알려주는 것입니다. 예를 들자면, context를 abcdefg라고 가정했을시 e가 정답(text)이라면, answer_start는 5가 됩니다. \n",
        "본 SQAUD 문제에서는 단어를 쪼갠 것을 하나 하나의 위치를 예측하는 것이 아니라, 단어의 시작 위치와 끝 위치를 예측하는 것이기 때문에 answer_start를 무시하셔도 됩니다."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "waSYuY0kMfW5",
        "colab_type": "code",
        "outputId": "73b12bee-1301-49d5-f180-c10c809f06df",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 402
        }
      },
      "source": [
        "train"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>index</th>\n",
              "      <th>question</th>\n",
              "      <th>context</th>\n",
              "      <th>answer_start</th>\n",
              "      <th>text</th>\n",
              "      <th>c_id</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>5733be284776f41900661182</td>\n",
              "      <td>To whom did the Virgin Mary allegedly appear i...</td>\n",
              "      <td>Architecturally, the school has a Catholic cha...</td>\n",
              "      <td>515</td>\n",
              "      <td>Saint Bernadette Soubirous</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>5733be284776f4190066117f</td>\n",
              "      <td>What is in front of the Notre Dame Main Building?</td>\n",
              "      <td>Architecturally, the school has a Catholic cha...</td>\n",
              "      <td>188</td>\n",
              "      <td>a copper statue of Christ</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>5733be284776f41900661180</td>\n",
              "      <td>The Basilica of the Sacred heart at Notre Dame...</td>\n",
              "      <td>Architecturally, the school has a Catholic cha...</td>\n",
              "      <td>279</td>\n",
              "      <td>the Main Building</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>5733be284776f41900661181</td>\n",
              "      <td>What is the Grotto at Notre Dame?</td>\n",
              "      <td>Architecturally, the school has a Catholic cha...</td>\n",
              "      <td>381</td>\n",
              "      <td>a Marian place of prayer and reflection</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>5733be284776f4190066117e</td>\n",
              "      <td>What sits on top of the Main Building at Notre...</td>\n",
              "      <td>Architecturally, the school has a Catholic cha...</td>\n",
              "      <td>92</td>\n",
              "      <td>a golden statue of the Virgin Mary</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>87594</th>\n",
              "      <td>5735d259012e2f140011a09d</td>\n",
              "      <td>In what US state did Kathmandu first establish...</td>\n",
              "      <td>Kathmandu Metropolitan City (KMC), in order to...</td>\n",
              "      <td>229</td>\n",
              "      <td>Oregon</td>\n",
              "      <td>18890</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>87595</th>\n",
              "      <td>5735d259012e2f140011a09e</td>\n",
              "      <td>What was Yangon previously known as?</td>\n",
              "      <td>Kathmandu Metropolitan City (KMC), in order to...</td>\n",
              "      <td>414</td>\n",
              "      <td>Rangoon</td>\n",
              "      <td>18890</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>87596</th>\n",
              "      <td>5735d259012e2f140011a09f</td>\n",
              "      <td>With what Belorussian city does Kathmandu have...</td>\n",
              "      <td>Kathmandu Metropolitan City (KMC), in order to...</td>\n",
              "      <td>476</td>\n",
              "      <td>Minsk</td>\n",
              "      <td>18890</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>87597</th>\n",
              "      <td>5735d259012e2f140011a0a0</td>\n",
              "      <td>In what year did Kathmandu create its initial ...</td>\n",
              "      <td>Kathmandu Metropolitan City (KMC), in order to...</td>\n",
              "      <td>199</td>\n",
              "      <td>1975</td>\n",
              "      <td>18890</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>87598</th>\n",
              "      <td>5735d259012e2f140011a0a1</td>\n",
              "      <td>What is KMC an initialism of?</td>\n",
              "      <td>Kathmandu Metropolitan City (KMC), in order to...</td>\n",
              "      <td>0</td>\n",
              "      <td>Kathmandu Metropolitan City</td>\n",
              "      <td>18890</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>87599 rows × 6 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "                          index  ...   c_id\n",
              "0      5733be284776f41900661182  ...      0\n",
              "1      5733be284776f4190066117f  ...      0\n",
              "2      5733be284776f41900661180  ...      0\n",
              "3      5733be284776f41900661181  ...      0\n",
              "4      5733be284776f4190066117e  ...      0\n",
              "...                         ...  ...    ...\n",
              "87594  5735d259012e2f140011a09d  ...  18890\n",
              "87595  5735d259012e2f140011a09e  ...  18890\n",
              "87596  5735d259012e2f140011a09f  ...  18890\n",
              "87597  5735d259012e2f140011a0a0  ...  18890\n",
              "87598  5735d259012e2f140011a0a1  ...  18890\n",
              "\n",
              "[87599 rows x 6 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 16
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2-NebUQmG1w-",
        "colab_type": "text"
      },
      "source": [
        "\n",
        "- bert 훈련을 위한 사전 설정을 합니다. SEQ_LEN은 문장의 최대 길이입니다. SEQ_LEN 보다 문장의 길이가 작다면 남은 부분은 0이 채워지고, 만약에 SEQ_LEN보다 문장 길이가 길다면 SEQ_LEN을 초과하는 부분이 잘리게 됩니다.  \n",
        "본 문제에서는 메모리 문제 등으로 384로 정했습니다.\n",
        "- BATCH_SIZE는 메모리 초과 같은 문제를 방지하기 위해 작은 수인 10으로 정했습니다. 그리고 총 훈련 에포크 수는 2로 정했습니다. 학습율(LR;Learning rate)은 3e-5로 작게 정했습니다.\n",
        "- pretrained_path는 bert 사전학습 모형이 있는 폴더를 의미합니다.\n",
        "- 그리고 우리가 분석할 문장이 들어있는 칼럼의 제목인 document와 긍정인지 부정인지 알려주는 칼럼을 label로 정해줍니다\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tLYWgZJAtsku",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "SEQ_LEN = 384\n",
        "BATCH_SIZE = 10\n",
        "EPOCHS=2\n",
        "LR=3e-5\n",
        "\n",
        "pretrained_path =\"bert\"\n",
        "config_path = os.path.join(pretrained_path, 'bert_config.json')\n",
        "checkpoint_path = os.path.join(pretrained_path, 'bert_model.ckpt')\n",
        "vocab_path = os.path.join(pretrained_path, 'vocab.txt')\n",
        "\n",
        "DATA_COLUMN = \"context\"\n",
        "QUESTION_COLUMN = \"question\"\n",
        "TEXT = \"text\""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IaU-13a9H4YJ",
        "colab_type": "text"
      },
      "source": [
        "vocab.txt에 있는 단어에 인덱스를 추가해주는 token_dict라는 딕셔너리를 생성합니다.  \n",
        "우리가 분석할 문장이 토큰화가 되고, 그 다음에는 인덱스(숫자)로 변경되어서 버트 신경망에 인풋으로 들어게 됩니다."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-ph2YQXg2iz5",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "token_dict = {}\n",
        "with codecs.open(vocab_path, 'r', 'utf8') as reader:\n",
        "    for line in reader:\n",
        "        token = line.strip()\n",
        "        if \"_\" in token:\n",
        "          token = token.replace(\"_\",\"\")\n",
        "          token = \"##\" + token\n",
        "        token_dict[token] = len(token_dict)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0WhiwaBCIXqv",
        "colab_type": "text"
      },
      "source": [
        "- BERT의 토큰화는 단어를 분리하는 토큰화 방식입니다. wordpiece(단어조각?) 방식이라고 하는데, 이는 한국어를 형태소로 꼭 변환해야 할 문제를 해결해주며, 의미가 있는 단어는 밀접하게 연관이 되게 하는 장점까지 갖추고 있습니다.\n",
        "- 단어의 첫 시작은 ##가 붙지 않지만, 단어에 포함되면서 단어의 시작이 아닌 부분에는 ##가 붙는 것이 특징입니다."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "88T_cMA4tsk2",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "tokenizer = Tokenizer(token_dict)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "W0dmxrOhJ9vD",
        "colab_type": "text"
      },
      "source": [
        "토큰화가 잘 되었는지 확인해 봅니다.\n",
        "버트 모형은 문장 앞에 꼭 [CLS]라는 문자가 위치하고, [SEP]라는 문자가 끝에 위치합니다.  \n",
        "[CLS]는 문장의 시작, [SEP]는 문장의 끝을 의미합니다."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-4EWrSTCMkzk",
        "colab_type": "code",
        "outputId": "72fb5a3e-1755-4e8b-b63b-403c501cd00c",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "print(tokenizer.tokenize(\"keras is reall fun.\"), tokenizer.tokenize(\"we can manipulate AI.\"))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "['[CLS]', 'keras', 'is', 'real', '##l', 'fun', '.', '[SEP]'] ['[CLS]', 'we', 'can', 'mani', '##pul', '##ate', 'ai', '.', '[SEP]']\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kOuqggRsORwN",
        "colab_type": "code",
        "outputId": "b215daf9-4677-406c-d846-c6bfeda8b175",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "token_dict"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'[PAD]': 0,\n",
              " '[unused1]': 1,\n",
              " '[unused2]': 2,\n",
              " '[unused3]': 3,\n",
              " '[unused4]': 4,\n",
              " '[unused5]': 5,\n",
              " '[unused6]': 6,\n",
              " '[unused7]': 7,\n",
              " '[unused8]': 8,\n",
              " '[unused9]': 9,\n",
              " '[unused10]': 10,\n",
              " '[unused11]': 11,\n",
              " '[unused12]': 12,\n",
              " '[unused13]': 13,\n",
              " '[unused14]': 14,\n",
              " '[unused15]': 15,\n",
              " '[unused16]': 16,\n",
              " '[unused17]': 17,\n",
              " '[unused18]': 18,\n",
              " '[unused19]': 19,\n",
              " '[unused20]': 20,\n",
              " '[unused21]': 21,\n",
              " '[unused22]': 22,\n",
              " '[unused23]': 23,\n",
              " '[unused24]': 24,\n",
              " '[unused25]': 25,\n",
              " '[unused26]': 26,\n",
              " '[unused27]': 27,\n",
              " '[unused28]': 28,\n",
              " '[unused29]': 29,\n",
              " '[unused30]': 30,\n",
              " '[unused31]': 31,\n",
              " '[unused32]': 32,\n",
              " '[unused33]': 33,\n",
              " '[unused34]': 34,\n",
              " '[unused35]': 35,\n",
              " '[unused36]': 36,\n",
              " '[unused37]': 37,\n",
              " '[unused38]': 38,\n",
              " '[unused39]': 39,\n",
              " '[unused40]': 40,\n",
              " '[unused41]': 41,\n",
              " '[unused42]': 42,\n",
              " '[unused43]': 43,\n",
              " '[unused44]': 44,\n",
              " '[unused45]': 45,\n",
              " '[unused46]': 46,\n",
              " '[unused47]': 47,\n",
              " '[unused48]': 48,\n",
              " '[unused49]': 49,\n",
              " '[unused50]': 50,\n",
              " '[unused51]': 51,\n",
              " '[unused52]': 52,\n",
              " '[unused53]': 53,\n",
              " '[unused54]': 54,\n",
              " '[unused55]': 55,\n",
              " '[unused56]': 56,\n",
              " '[unused57]': 57,\n",
              " '[unused58]': 58,\n",
              " '[unused59]': 59,\n",
              " '[unused60]': 60,\n",
              " '[unused61]': 61,\n",
              " '[unused62]': 62,\n",
              " '[unused63]': 63,\n",
              " '[unused64]': 64,\n",
              " '[unused65]': 65,\n",
              " '[unused66]': 66,\n",
              " '[unused67]': 67,\n",
              " '[unused68]': 68,\n",
              " '[unused69]': 69,\n",
              " '[unused70]': 70,\n",
              " '[unused71]': 71,\n",
              " '[unused72]': 72,\n",
              " '[unused73]': 73,\n",
              " '[unused74]': 74,\n",
              " '[unused75]': 75,\n",
              " '[unused76]': 76,\n",
              " '[unused77]': 77,\n",
              " '[unused78]': 78,\n",
              " '[unused79]': 79,\n",
              " '[unused80]': 80,\n",
              " '[unused81]': 81,\n",
              " '[unused82]': 82,\n",
              " '[unused83]': 83,\n",
              " '[unused84]': 84,\n",
              " '[unused85]': 85,\n",
              " '[unused86]': 86,\n",
              " '[unused87]': 87,\n",
              " '[unused88]': 88,\n",
              " '[unused89]': 89,\n",
              " '[unused90]': 90,\n",
              " '[unused91]': 91,\n",
              " '[unused92]': 92,\n",
              " '[unused93]': 93,\n",
              " '[unused94]': 94,\n",
              " '[unused95]': 95,\n",
              " '[unused96]': 96,\n",
              " '[unused97]': 97,\n",
              " '[unused98]': 98,\n",
              " '[unused99]': 99,\n",
              " '[UNK]': 100,\n",
              " '[CLS]': 101,\n",
              " '[SEP]': 102,\n",
              " '[MASK]': 103,\n",
              " '<S>': 104,\n",
              " '<T>': 105,\n",
              " '!': 106,\n",
              " '\"': 107,\n",
              " '#': 108,\n",
              " '$': 109,\n",
              " '%': 110,\n",
              " '&': 111,\n",
              " \"'\": 112,\n",
              " '(': 113,\n",
              " ')': 114,\n",
              " '*': 115,\n",
              " '+': 116,\n",
              " ',': 117,\n",
              " '-': 118,\n",
              " '.': 119,\n",
              " '/': 120,\n",
              " '0': 121,\n",
              " '1': 122,\n",
              " '2': 123,\n",
              " '3': 124,\n",
              " '4': 125,\n",
              " '5': 126,\n",
              " '6': 127,\n",
              " '7': 128,\n",
              " '8': 129,\n",
              " '9': 130,\n",
              " ':': 131,\n",
              " ';': 132,\n",
              " '<': 133,\n",
              " '=': 134,\n",
              " '>': 135,\n",
              " '?': 136,\n",
              " '@': 137,\n",
              " 'A': 138,\n",
              " 'B': 139,\n",
              " 'C': 140,\n",
              " 'D': 141,\n",
              " 'E': 142,\n",
              " 'F': 143,\n",
              " 'G': 144,\n",
              " 'H': 145,\n",
              " 'I': 146,\n",
              " 'J': 147,\n",
              " 'K': 148,\n",
              " 'L': 149,\n",
              " 'M': 150,\n",
              " 'N': 151,\n",
              " 'O': 152,\n",
              " 'P': 153,\n",
              " 'Q': 154,\n",
              " 'R': 155,\n",
              " 'S': 156,\n",
              " 'T': 157,\n",
              " 'U': 158,\n",
              " 'V': 159,\n",
              " 'W': 160,\n",
              " 'X': 161,\n",
              " 'Y': 162,\n",
              " 'Z': 163,\n",
              " '[': 164,\n",
              " '\\\\': 165,\n",
              " ']': 166,\n",
              " '^': 167,\n",
              " '##': 168,\n",
              " 'a': 169,\n",
              " 'b': 170,\n",
              " 'c': 171,\n",
              " 'd': 172,\n",
              " 'e': 173,\n",
              " 'f': 174,\n",
              " 'g': 175,\n",
              " 'h': 176,\n",
              " 'i': 177,\n",
              " 'j': 178,\n",
              " 'k': 179,\n",
              " 'l': 180,\n",
              " 'm': 181,\n",
              " 'n': 182,\n",
              " 'o': 183,\n",
              " 'p': 184,\n",
              " 'q': 185,\n",
              " 'r': 186,\n",
              " 's': 187,\n",
              " 't': 188,\n",
              " 'u': 189,\n",
              " 'v': 190,\n",
              " 'w': 191,\n",
              " 'x': 192,\n",
              " 'y': 193,\n",
              " 'z': 194,\n",
              " '{': 195,\n",
              " '|': 196,\n",
              " '}': 197,\n",
              " '~': 198,\n",
              " '¡': 199,\n",
              " '¢': 200,\n",
              " '£': 201,\n",
              " '¥': 202,\n",
              " '¦': 203,\n",
              " '§': 204,\n",
              " '¨': 205,\n",
              " '©': 206,\n",
              " 'ª': 207,\n",
              " '«': 208,\n",
              " '¬': 209,\n",
              " '®': 210,\n",
              " '°': 211,\n",
              " '±': 212,\n",
              " '²': 213,\n",
              " '³': 214,\n",
              " 'µ': 215,\n",
              " '¶': 216,\n",
              " '·': 217,\n",
              " '¹': 218,\n",
              " 'º': 219,\n",
              " '»': 220,\n",
              " '¼': 221,\n",
              " '½': 222,\n",
              " '¾': 223,\n",
              " '¿': 224,\n",
              " 'À': 225,\n",
              " 'Á': 226,\n",
              " 'Â': 227,\n",
              " 'Ã': 228,\n",
              " 'Ä': 229,\n",
              " 'Å': 230,\n",
              " 'Æ': 231,\n",
              " 'Ç': 232,\n",
              " 'È': 233,\n",
              " 'É': 234,\n",
              " 'Ê': 235,\n",
              " 'Ë': 236,\n",
              " 'Ì': 237,\n",
              " 'Í': 238,\n",
              " 'Î': 239,\n",
              " 'Ð': 240,\n",
              " 'Ñ': 241,\n",
              " 'Ò': 242,\n",
              " 'Ó': 243,\n",
              " 'Ô': 244,\n",
              " 'Õ': 245,\n",
              " 'Ö': 246,\n",
              " '×': 247,\n",
              " 'Ø': 248,\n",
              " 'Ú': 249,\n",
              " 'Ü': 250,\n",
              " 'Ý': 251,\n",
              " 'Þ': 252,\n",
              " 'ß': 253,\n",
              " 'à': 254,\n",
              " 'á': 255,\n",
              " 'â': 256,\n",
              " 'ã': 257,\n",
              " 'ä': 258,\n",
              " 'å': 259,\n",
              " 'æ': 260,\n",
              " 'ç': 261,\n",
              " 'è': 262,\n",
              " 'é': 263,\n",
              " 'ê': 264,\n",
              " 'ë': 265,\n",
              " 'ì': 266,\n",
              " 'í': 267,\n",
              " 'î': 268,\n",
              " 'ï': 269,\n",
              " 'ð': 270,\n",
              " 'ñ': 271,\n",
              " 'ò': 272,\n",
              " 'ó': 273,\n",
              " 'ô': 274,\n",
              " 'õ': 275,\n",
              " 'ö': 276,\n",
              " '÷': 277,\n",
              " 'ø': 278,\n",
              " 'ù': 279,\n",
              " 'ú': 280,\n",
              " 'û': 281,\n",
              " 'ü': 282,\n",
              " 'ý': 283,\n",
              " 'þ': 284,\n",
              " 'ÿ': 285,\n",
              " 'Ā': 286,\n",
              " 'ā': 287,\n",
              " 'Ă': 288,\n",
              " 'ă': 289,\n",
              " 'Ą': 290,\n",
              " 'ą': 291,\n",
              " 'Ć': 292,\n",
              " 'ć': 293,\n",
              " 'Č': 294,\n",
              " 'č': 295,\n",
              " 'Ď': 296,\n",
              " 'ď': 297,\n",
              " 'Đ': 298,\n",
              " 'đ': 299,\n",
              " 'Ē': 300,\n",
              " 'ē': 301,\n",
              " 'Ĕ': 302,\n",
              " 'ĕ': 303,\n",
              " 'Ė': 304,\n",
              " 'ė': 305,\n",
              " 'ę': 306,\n",
              " 'ě': 307,\n",
              " 'Ğ': 308,\n",
              " 'ğ': 309,\n",
              " 'ġ': 310,\n",
              " 'Ģ': 311,\n",
              " 'ģ': 312,\n",
              " 'Ħ': 313,\n",
              " 'ħ': 314,\n",
              " 'ĩ': 315,\n",
              " 'Ī': 316,\n",
              " 'ī': 317,\n",
              " 'Į': 318,\n",
              " 'į': 319,\n",
              " 'İ': 320,\n",
              " 'ı': 321,\n",
              " 'Ķ': 322,\n",
              " 'ķ': 323,\n",
              " 'ĺ': 324,\n",
              " 'Ļ': 325,\n",
              " 'ļ': 326,\n",
              " 'Ľ': 327,\n",
              " 'ľ': 328,\n",
              " 'Ł': 329,\n",
              " 'ł': 330,\n",
              " 'ń': 331,\n",
              " 'Ņ': 332,\n",
              " 'ņ': 333,\n",
              " 'ň': 334,\n",
              " 'ŉ': 335,\n",
              " 'ŋ': 336,\n",
              " 'Ō': 337,\n",
              " 'ō': 338,\n",
              " 'ŏ': 339,\n",
              " 'Ő': 340,\n",
              " 'ő': 341,\n",
              " 'Œ': 342,\n",
              " 'œ': 343,\n",
              " 'ŕ': 344,\n",
              " 'Ř': 345,\n",
              " 'ř': 346,\n",
              " 'Ś': 347,\n",
              " 'ś': 348,\n",
              " 'Ş': 349,\n",
              " 'ş': 350,\n",
              " 'Š': 351,\n",
              " 'š': 352,\n",
              " 'Ţ': 353,\n",
              " 'ţ': 354,\n",
              " 'Ť': 355,\n",
              " 'ť': 356,\n",
              " 'ũ': 357,\n",
              " 'Ū': 358,\n",
              " 'ū': 359,\n",
              " 'ŭ': 360,\n",
              " 'ů': 361,\n",
              " 'Ű': 362,\n",
              " 'ű': 363,\n",
              " 'ų': 364,\n",
              " 'ŵ': 365,\n",
              " 'ŷ': 366,\n",
              " 'Ź': 367,\n",
              " 'ź': 368,\n",
              " 'Ż': 369,\n",
              " 'ż': 370,\n",
              " 'Ž': 371,\n",
              " 'ž': 372,\n",
              " 'Ə': 373,\n",
              " 'ƒ': 374,\n",
              " 'ơ': 375,\n",
              " 'Ư': 376,\n",
              " 'ư': 377,\n",
              " 'ǎ': 378,\n",
              " 'ǐ': 379,\n",
              " 'ǔ': 380,\n",
              " 'ǫ': 381,\n",
              " 'ǹ': 382,\n",
              " 'Ș': 383,\n",
              " 'ș': 384,\n",
              " 'Ț': 385,\n",
              " 'ț': 386,\n",
              " 'ɐ': 387,\n",
              " 'ɑ': 388,\n",
              " 'ɔ': 389,\n",
              " 'ɕ': 390,\n",
              " 'ə': 391,\n",
              " 'ɛ': 392,\n",
              " 'ɡ': 393,\n",
              " 'ɣ': 394,\n",
              " 'ɨ': 395,\n",
              " 'ɪ': 396,\n",
              " 'ɲ': 397,\n",
              " 'ɾ': 398,\n",
              " 'ʁ': 399,\n",
              " 'ʃ': 400,\n",
              " 'ʊ': 401,\n",
              " 'ʎ': 402,\n",
              " 'ʒ': 403,\n",
              " 'ʔ': 404,\n",
              " 'ʙ': 405,\n",
              " 'ʰ': 406,\n",
              " 'ʲ': 407,\n",
              " 'ʳ': 408,\n",
              " 'ʷ': 409,\n",
              " 'ʸ': 410,\n",
              " 'ʻ': 411,\n",
              " 'ʼ': 412,\n",
              " 'ʾ': 413,\n",
              " 'ʿ': 414,\n",
              " 'ˈ': 415,\n",
              " 'ː': 416,\n",
              " 'ˡ': 417,\n",
              " 'ˢ': 418,\n",
              " '̀': 419,\n",
              " '́': 420,\n",
              " '̃': 421,\n",
              " '̄': 422,\n",
              " '̍': 423,\n",
              " '̥': 424,\n",
              " '̧': 425,\n",
              " '̲': 426,\n",
              " '͡': 427,\n",
              " '΄': 428,\n",
              " 'Ά': 429,\n",
              " 'Έ': 430,\n",
              " 'Ή': 431,\n",
              " 'Ί': 432,\n",
              " 'Ό': 433,\n",
              " 'Ύ': 434,\n",
              " 'Ώ': 435,\n",
              " 'ΐ': 436,\n",
              " 'Α': 437,\n",
              " 'Β': 438,\n",
              " 'Γ': 439,\n",
              " 'Δ': 440,\n",
              " 'Ε': 441,\n",
              " 'Ζ': 442,\n",
              " 'Η': 443,\n",
              " 'Θ': 444,\n",
              " 'Ι': 445,\n",
              " 'Κ': 446,\n",
              " 'Λ': 447,\n",
              " 'Μ': 448,\n",
              " 'Ν': 449,\n",
              " 'Ξ': 450,\n",
              " 'Ο': 451,\n",
              " 'Π': 452,\n",
              " 'Ρ': 453,\n",
              " 'Σ': 454,\n",
              " 'Τ': 455,\n",
              " 'Υ': 456,\n",
              " 'Φ': 457,\n",
              " 'Χ': 458,\n",
              " 'Ψ': 459,\n",
              " 'Ω': 460,\n",
              " 'ά': 461,\n",
              " 'έ': 462,\n",
              " 'ή': 463,\n",
              " 'ί': 464,\n",
              " 'α': 465,\n",
              " 'β': 466,\n",
              " 'γ': 467,\n",
              " 'δ': 468,\n",
              " 'ε': 469,\n",
              " 'ζ': 470,\n",
              " 'η': 471,\n",
              " 'θ': 472,\n",
              " 'ι': 473,\n",
              " 'κ': 474,\n",
              " 'λ': 475,\n",
              " 'μ': 476,\n",
              " 'ν': 477,\n",
              " 'ξ': 478,\n",
              " 'ο': 479,\n",
              " 'π': 480,\n",
              " 'ρ': 481,\n",
              " 'ς': 482,\n",
              " 'σ': 483,\n",
              " 'τ': 484,\n",
              " 'υ': 485,\n",
              " 'φ': 486,\n",
              " 'χ': 487,\n",
              " 'ψ': 488,\n",
              " 'ω': 489,\n",
              " 'ϊ': 490,\n",
              " 'ϋ': 491,\n",
              " 'ό': 492,\n",
              " 'ύ': 493,\n",
              " 'ώ': 494,\n",
              " 'Ё': 495,\n",
              " 'Ђ': 496,\n",
              " 'Ѓ': 497,\n",
              " 'Є': 498,\n",
              " 'Ѕ': 499,\n",
              " 'І': 500,\n",
              " 'Ї': 501,\n",
              " 'Ј': 502,\n",
              " 'Љ': 503,\n",
              " 'Њ': 504,\n",
              " 'Ћ': 505,\n",
              " 'Ќ': 506,\n",
              " 'Ў': 507,\n",
              " 'Џ': 508,\n",
              " 'А': 509,\n",
              " 'Б': 510,\n",
              " 'В': 511,\n",
              " 'Г': 512,\n",
              " 'Д': 513,\n",
              " 'Е': 514,\n",
              " 'Ж': 515,\n",
              " 'З': 516,\n",
              " 'И': 517,\n",
              " 'Й': 518,\n",
              " 'К': 519,\n",
              " 'Л': 520,\n",
              " 'М': 521,\n",
              " 'Н': 522,\n",
              " 'О': 523,\n",
              " 'П': 524,\n",
              " 'Р': 525,\n",
              " 'С': 526,\n",
              " 'Т': 527,\n",
              " 'У': 528,\n",
              " 'Ф': 529,\n",
              " 'Х': 530,\n",
              " 'Ц': 531,\n",
              " 'Ч': 532,\n",
              " 'Ш': 533,\n",
              " 'Щ': 534,\n",
              " 'Ъ': 535,\n",
              " 'Ы': 536,\n",
              " 'Ь': 537,\n",
              " 'Э': 538,\n",
              " 'Ю': 539,\n",
              " 'Я': 540,\n",
              " 'а': 541,\n",
              " 'б': 542,\n",
              " 'в': 543,\n",
              " 'г': 544,\n",
              " 'д': 545,\n",
              " 'е': 546,\n",
              " 'ж': 547,\n",
              " 'з': 548,\n",
              " 'и': 549,\n",
              " 'й': 550,\n",
              " 'к': 551,\n",
              " 'л': 552,\n",
              " 'м': 553,\n",
              " 'н': 554,\n",
              " 'о': 555,\n",
              " 'п': 556,\n",
              " 'р': 557,\n",
              " 'с': 558,\n",
              " 'т': 559,\n",
              " 'у': 560,\n",
              " 'ф': 561,\n",
              " 'х': 562,\n",
              " 'ц': 563,\n",
              " 'ч': 564,\n",
              " 'ш': 565,\n",
              " 'щ': 566,\n",
              " 'ъ': 567,\n",
              " 'ы': 568,\n",
              " 'ь': 569,\n",
              " 'э': 570,\n",
              " 'ю': 571,\n",
              " 'я': 572,\n",
              " 'ѐ': 573,\n",
              " 'ё': 574,\n",
              " 'ђ': 575,\n",
              " 'ѓ': 576,\n",
              " 'є': 577,\n",
              " 'ѕ': 578,\n",
              " 'і': 579,\n",
              " 'ї': 580,\n",
              " 'ј': 581,\n",
              " 'љ': 582,\n",
              " 'њ': 583,\n",
              " 'ћ': 584,\n",
              " 'ќ': 585,\n",
              " 'ѝ': 586,\n",
              " 'ў': 587,\n",
              " 'џ': 588,\n",
              " 'ѣ': 589,\n",
              " 'Ґ': 590,\n",
              " 'ґ': 591,\n",
              " 'Ғ': 592,\n",
              " 'ғ': 593,\n",
              " 'Җ': 594,\n",
              " 'җ': 595,\n",
              " 'Ҙ': 596,\n",
              " 'ҙ': 597,\n",
              " 'Қ': 598,\n",
              " 'қ': 599,\n",
              " 'Ҡ': 600,\n",
              " 'ҡ': 601,\n",
              " 'ң': 602,\n",
              " 'ҫ': 603,\n",
              " 'Ү': 604,\n",
              " 'ү': 605,\n",
              " 'Ұ': 606,\n",
              " 'ұ': 607,\n",
              " 'Ҳ': 608,\n",
              " 'ҳ': 609,\n",
              " 'Ҷ': 610,\n",
              " 'ҷ': 611,\n",
              " 'Һ': 612,\n",
              " 'һ': 613,\n",
              " 'Ӏ': 614,\n",
              " 'ӑ': 615,\n",
              " 'ӗ': 616,\n",
              " 'Ә': 617,\n",
              " 'ә': 618,\n",
              " 'ӣ': 619,\n",
              " 'Ө': 620,\n",
              " 'ө': 621,\n",
              " 'Ӯ': 622,\n",
              " 'ӯ': 623,\n",
              " 'ӳ': 624,\n",
              " 'Ա': 625,\n",
              " 'Բ': 626,\n",
              " 'Գ': 627,\n",
              " 'Դ': 628,\n",
              " 'Ե': 629,\n",
              " 'Զ': 630,\n",
              " 'Է': 631,\n",
              " 'Ը': 632,\n",
              " 'Թ': 633,\n",
              " 'Ժ': 634,\n",
              " 'Ի': 635,\n",
              " 'Լ': 636,\n",
              " 'Խ': 637,\n",
              " 'Ծ': 638,\n",
              " 'Կ': 639,\n",
              " 'Հ': 640,\n",
              " 'Ձ': 641,\n",
              " 'Ղ': 642,\n",
              " 'Ճ': 643,\n",
              " 'Մ': 644,\n",
              " 'Յ': 645,\n",
              " 'Ն': 646,\n",
              " 'Շ': 647,\n",
              " 'Ո': 648,\n",
              " 'Չ': 649,\n",
              " 'Պ': 650,\n",
              " 'Ջ': 651,\n",
              " 'Ռ': 652,\n",
              " 'Ս': 653,\n",
              " 'Վ': 654,\n",
              " 'Տ': 655,\n",
              " 'Ր': 656,\n",
              " 'Ց': 657,\n",
              " 'Ւ': 658,\n",
              " 'Փ': 659,\n",
              " 'Ք': 660,\n",
              " 'Օ': 661,\n",
              " 'Ֆ': 662,\n",
              " '՚': 663,\n",
              " '՛': 664,\n",
              " '՜': 665,\n",
              " '՝': 666,\n",
              " '՞': 667,\n",
              " 'ա': 668,\n",
              " 'բ': 669,\n",
              " 'գ': 670,\n",
              " 'դ': 671,\n",
              " 'ե': 672,\n",
              " 'զ': 673,\n",
              " 'է': 674,\n",
              " 'ը': 675,\n",
              " 'թ': 676,\n",
              " 'ժ': 677,\n",
              " 'ի': 678,\n",
              " 'լ': 679,\n",
              " 'խ': 680,\n",
              " 'ծ': 681,\n",
              " 'կ': 682,\n",
              " 'հ': 683,\n",
              " 'ձ': 684,\n",
              " 'ղ': 685,\n",
              " 'ճ': 686,\n",
              " 'մ': 687,\n",
              " 'յ': 688,\n",
              " 'ն': 689,\n",
              " 'շ': 690,\n",
              " 'ո': 691,\n",
              " 'չ': 692,\n",
              " 'պ': 693,\n",
              " 'ջ': 694,\n",
              " 'ռ': 695,\n",
              " 'ս': 696,\n",
              " 'վ': 697,\n",
              " 'տ': 698,\n",
              " 'ր': 699,\n",
              " 'ց': 700,\n",
              " 'ւ': 701,\n",
              " 'փ': 702,\n",
              " 'ք': 703,\n",
              " 'օ': 704,\n",
              " 'ֆ': 705,\n",
              " 'և': 706,\n",
              " '։': 707,\n",
              " '֊': 708,\n",
              " 'ְ': 709,\n",
              " 'ֱ': 710,\n",
              " 'ֲ': 711,\n",
              " 'ִ': 712,\n",
              " 'ֵ': 713,\n",
              " 'ֶ': 714,\n",
              " 'ַ': 715,\n",
              " 'ָ': 716,\n",
              " 'ֹ': 717,\n",
              " 'ּ': 718,\n",
              " '־': 719,\n",
              " 'ׁ': 720,\n",
              " 'ׂ': 721,\n",
              " '׃': 722,\n",
              " 'א': 723,\n",
              " 'ב': 724,\n",
              " 'ג': 725,\n",
              " 'ד': 726,\n",
              " 'ה': 727,\n",
              " 'ו': 728,\n",
              " 'ז': 729,\n",
              " 'ח': 730,\n",
              " 'ט': 731,\n",
              " 'י': 732,\n",
              " 'ך': 733,\n",
              " 'כ': 734,\n",
              " 'ל': 735,\n",
              " 'ם': 736,\n",
              " 'מ': 737,\n",
              " 'ן': 738,\n",
              " 'נ': 739,\n",
              " 'ס': 740,\n",
              " 'ע': 741,\n",
              " 'ף': 742,\n",
              " 'פ': 743,\n",
              " 'ץ': 744,\n",
              " 'צ': 745,\n",
              " 'ק': 746,\n",
              " 'ר': 747,\n",
              " 'ש': 748,\n",
              " 'ת': 749,\n",
              " '׳': 750,\n",
              " '״': 751,\n",
              " '،': 752,\n",
              " '؍': 753,\n",
              " 'ؒ': 754,\n",
              " '؛': 755,\n",
              " '؟': 756,\n",
              " 'ء': 757,\n",
              " 'آ': 758,\n",
              " 'أ': 759,\n",
              " 'ؤ': 760,\n",
              " 'إ': 761,\n",
              " 'ئ': 762,\n",
              " 'ا': 763,\n",
              " 'ب': 764,\n",
              " 'ة': 765,\n",
              " 'ت': 766,\n",
              " 'ث': 767,\n",
              " 'ج': 768,\n",
              " 'ح': 769,\n",
              " 'خ': 770,\n",
              " 'د': 771,\n",
              " 'ذ': 772,\n",
              " 'ر': 773,\n",
              " 'ز': 774,\n",
              " 'س': 775,\n",
              " 'ش': 776,\n",
              " 'ص': 777,\n",
              " 'ض': 778,\n",
              " 'ط': 779,\n",
              " 'ظ': 780,\n",
              " 'ع': 781,\n",
              " 'غ': 782,\n",
              " 'ـ': 783,\n",
              " 'ف': 784,\n",
              " 'ق': 785,\n",
              " 'ك': 786,\n",
              " 'ل': 787,\n",
              " 'م': 788,\n",
              " 'ن': 789,\n",
              " 'ه': 790,\n",
              " 'و': 791,\n",
              " 'ى': 792,\n",
              " 'ي': 793,\n",
              " 'ً': 794,\n",
              " 'ٌ': 795,\n",
              " 'ٍ': 796,\n",
              " 'َ': 797,\n",
              " 'ُ': 798,\n",
              " 'ِ': 799,\n",
              " 'ّ': 800,\n",
              " 'ْ': 801,\n",
              " 'ٔ': 802,\n",
              " 'ٛ': 803,\n",
              " '٠': 804,\n",
              " '١': 805,\n",
              " '٢': 806,\n",
              " '٣': 807,\n",
              " '٤': 808,\n",
              " '٥': 809,\n",
              " '٩': 810,\n",
              " '٪': 811,\n",
              " '٫': 812,\n",
              " '٬': 813,\n",
              " '٭': 814,\n",
              " 'ٰ': 815,\n",
              " 'ٹ': 816,\n",
              " 'پ': 817,\n",
              " 'چ': 818,\n",
              " 'ڈ': 819,\n",
              " 'ڑ': 820,\n",
              " 'ژ': 821,\n",
              " 'ڤ': 822,\n",
              " 'ک': 823,\n",
              " 'ڭ': 824,\n",
              " 'گ': 825,\n",
              " 'ں': 826,\n",
              " 'ھ': 827,\n",
              " 'ۀ': 828,\n",
              " 'ہ': 829,\n",
              " 'ۂ': 830,\n",
              " 'ۃ': 831,\n",
              " 'ۆ': 832,\n",
              " 'ۇ': 833,\n",
              " 'ی': 834,\n",
              " 'ے': 835,\n",
              " 'ۓ': 836,\n",
              " '۔': 837,\n",
              " '۰': 838,\n",
              " '۱': 839,\n",
              " '۲': 840,\n",
              " '۳': 841,\n",
              " '۴': 842,\n",
              " '۵': 843,\n",
              " '۶': 844,\n",
              " '۷': 845,\n",
              " '۸': 846,\n",
              " '۹': 847,\n",
              " 'ँ': 848,\n",
              " 'ं': 849,\n",
              " 'ः': 850,\n",
              " 'अ': 851,\n",
              " 'आ': 852,\n",
              " 'इ': 853,\n",
              " 'ई': 854,\n",
              " 'उ': 855,\n",
              " 'ऊ': 856,\n",
              " 'ऋ': 857,\n",
              " 'ऍ': 858,\n",
              " 'ऎ': 859,\n",
              " 'ए': 860,\n",
              " 'ऐ': 861,\n",
              " 'ऑ': 862,\n",
              " 'ओ': 863,\n",
              " 'औ': 864,\n",
              " 'क': 865,\n",
              " 'ख': 866,\n",
              " 'ग': 867,\n",
              " 'घ': 868,\n",
              " 'ङ': 869,\n",
              " 'च': 870,\n",
              " 'छ': 871,\n",
              " 'ज': 872,\n",
              " 'झ': 873,\n",
              " 'ञ': 874,\n",
              " 'ट': 875,\n",
              " 'ठ': 876,\n",
              " 'ड': 877,\n",
              " 'ढ': 878,\n",
              " 'ण': 879,\n",
              " 'त': 880,\n",
              " 'थ': 881,\n",
              " 'द': 882,\n",
              " 'ध': 883,\n",
              " 'न': 884,\n",
              " 'प': 885,\n",
              " 'फ': 886,\n",
              " 'ब': 887,\n",
              " 'भ': 888,\n",
              " 'म': 889,\n",
              " 'य': 890,\n",
              " 'र': 891,\n",
              " 'ऱ': 892,\n",
              " 'ल': 893,\n",
              " 'ळ': 894,\n",
              " 'व': 895,\n",
              " 'श': 896,\n",
              " 'ष': 897,\n",
              " 'स': 898,\n",
              " 'ह': 899,\n",
              " '़': 900,\n",
              " 'ऽ': 901,\n",
              " 'ा': 902,\n",
              " 'ि': 903,\n",
              " 'ी': 904,\n",
              " 'ु': 905,\n",
              " 'ू': 906,\n",
              " 'ृ': 907,\n",
              " 'ॄ': 908,\n",
              " 'ॅ': 909,\n",
              " 'ॆ': 910,\n",
              " 'े': 911,\n",
              " 'ै': 912,\n",
              " 'ॉ': 913,\n",
              " 'ो': 914,\n",
              " 'ौ': 915,\n",
              " '्': 916,\n",
              " 'ॐ': 917,\n",
              " '॓': 918,\n",
              " 'ॠ': 919,\n",
              " '।': 920,\n",
              " '॥': 921,\n",
              " '०': 922,\n",
              " '१': 923,\n",
              " '२': 924,\n",
              " '३': 925,\n",
              " '४': 926,\n",
              " '५': 927,\n",
              " '६': 928,\n",
              " '७': 929,\n",
              " '८': 930,\n",
              " '९': 931,\n",
              " '॰': 932,\n",
              " 'ॲ': 933,\n",
              " 'ঁ': 934,\n",
              " 'ং': 935,\n",
              " 'ঃ': 936,\n",
              " 'অ': 937,\n",
              " 'আ': 938,\n",
              " 'ই': 939,\n",
              " 'ঈ': 940,\n",
              " 'উ': 941,\n",
              " 'ঊ': 942,\n",
              " 'ঋ': 943,\n",
              " 'এ': 944,\n",
              " 'ঐ': 945,\n",
              " 'ও': 946,\n",
              " 'ঔ': 947,\n",
              " 'ক': 948,\n",
              " 'খ': 949,\n",
              " 'গ': 950,\n",
              " 'ঘ': 951,\n",
              " 'ঙ': 952,\n",
              " 'চ': 953,\n",
              " 'ছ': 954,\n",
              " 'জ': 955,\n",
              " 'ঝ': 956,\n",
              " 'ঞ': 957,\n",
              " 'ট': 958,\n",
              " 'ঠ': 959,\n",
              " 'ড': 960,\n",
              " 'ঢ': 961,\n",
              " 'ণ': 962,\n",
              " 'ত': 963,\n",
              " 'থ': 964,\n",
              " 'দ': 965,\n",
              " 'ধ': 966,\n",
              " 'ন': 967,\n",
              " 'প': 968,\n",
              " 'ফ': 969,\n",
              " 'ব': 970,\n",
              " 'ভ': 971,\n",
              " 'ম': 972,\n",
              " 'য': 973,\n",
              " 'র': 974,\n",
              " 'ল': 975,\n",
              " 'শ': 976,\n",
              " 'ষ': 977,\n",
              " 'স': 978,\n",
              " 'হ': 979,\n",
              " '়': 980,\n",
              " 'া': 981,\n",
              " 'ি': 982,\n",
              " 'ী': 983,\n",
              " 'ু': 984,\n",
              " 'ূ': 985,\n",
              " 'ৃ': 986,\n",
              " 'ে': 987,\n",
              " 'ৈ': 988,\n",
              " 'ো': 989,\n",
              " 'ৌ': 990,\n",
              " '্': 991,\n",
              " 'ৎ': 992,\n",
              " '০': 993,\n",
              " '১': 994,\n",
              " '২': 995,\n",
              " '৩': 996,\n",
              " '৪': 997,\n",
              " '৫': 998,\n",
              " '৬': 999,\n",
              " ...}"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 21
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rCC_Z1TNSODT",
        "colab_type": "text"
      },
      "source": [
        "token dict의 key 값과 value 값을 바꾼 reverse_token_dict를 정의합니다."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oF_7BhoWw6_E",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "reverse_token_dict = {v : k for k, v in token_dict.items()}"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jeYV0qlaxepy",
        "colab_type": "code",
        "outputId": "819ac32b-588f-43db-fcc0-bf51d7d032f0",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "reverse_token_dict"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{0: '[PAD]',\n",
              " 1: '[unused1]',\n",
              " 2: '[unused2]',\n",
              " 3: '[unused3]',\n",
              " 4: '[unused4]',\n",
              " 5: '[unused5]',\n",
              " 6: '[unused6]',\n",
              " 7: '[unused7]',\n",
              " 8: '[unused8]',\n",
              " 9: '[unused9]',\n",
              " 10: '[unused10]',\n",
              " 11: '[unused11]',\n",
              " 12: '[unused12]',\n",
              " 13: '[unused13]',\n",
              " 14: '[unused14]',\n",
              " 15: '[unused15]',\n",
              " 16: '[unused16]',\n",
              " 17: '[unused17]',\n",
              " 18: '[unused18]',\n",
              " 19: '[unused19]',\n",
              " 20: '[unused20]',\n",
              " 21: '[unused21]',\n",
              " 22: '[unused22]',\n",
              " 23: '[unused23]',\n",
              " 24: '[unused24]',\n",
              " 25: '[unused25]',\n",
              " 26: '[unused26]',\n",
              " 27: '[unused27]',\n",
              " 28: '[unused28]',\n",
              " 29: '[unused29]',\n",
              " 30: '[unused30]',\n",
              " 31: '[unused31]',\n",
              " 32: '[unused32]',\n",
              " 33: '[unused33]',\n",
              " 34: '[unused34]',\n",
              " 35: '[unused35]',\n",
              " 36: '[unused36]',\n",
              " 37: '[unused37]',\n",
              " 38: '[unused38]',\n",
              " 39: '[unused39]',\n",
              " 40: '[unused40]',\n",
              " 41: '[unused41]',\n",
              " 42: '[unused42]',\n",
              " 43: '[unused43]',\n",
              " 44: '[unused44]',\n",
              " 45: '[unused45]',\n",
              " 46: '[unused46]',\n",
              " 47: '[unused47]',\n",
              " 48: '[unused48]',\n",
              " 49: '[unused49]',\n",
              " 50: '[unused50]',\n",
              " 51: '[unused51]',\n",
              " 52: '[unused52]',\n",
              " 53: '[unused53]',\n",
              " 54: '[unused54]',\n",
              " 55: '[unused55]',\n",
              " 56: '[unused56]',\n",
              " 57: '[unused57]',\n",
              " 58: '[unused58]',\n",
              " 59: '[unused59]',\n",
              " 60: '[unused60]',\n",
              " 61: '[unused61]',\n",
              " 62: '[unused62]',\n",
              " 63: '[unused63]',\n",
              " 64: '[unused64]',\n",
              " 65: '[unused65]',\n",
              " 66: '[unused66]',\n",
              " 67: '[unused67]',\n",
              " 68: '[unused68]',\n",
              " 69: '[unused69]',\n",
              " 70: '[unused70]',\n",
              " 71: '[unused71]',\n",
              " 72: '[unused72]',\n",
              " 73: '[unused73]',\n",
              " 74: '[unused74]',\n",
              " 75: '[unused75]',\n",
              " 76: '[unused76]',\n",
              " 77: '[unused77]',\n",
              " 78: '[unused78]',\n",
              " 79: '[unused79]',\n",
              " 80: '[unused80]',\n",
              " 81: '[unused81]',\n",
              " 82: '[unused82]',\n",
              " 83: '[unused83]',\n",
              " 84: '[unused84]',\n",
              " 85: '[unused85]',\n",
              " 86: '[unused86]',\n",
              " 87: '[unused87]',\n",
              " 88: '[unused88]',\n",
              " 89: '[unused89]',\n",
              " 90: '[unused90]',\n",
              " 91: '[unused91]',\n",
              " 92: '[unused92]',\n",
              " 93: '[unused93]',\n",
              " 94: '[unused94]',\n",
              " 95: '[unused95]',\n",
              " 96: '[unused96]',\n",
              " 97: '[unused97]',\n",
              " 98: '[unused98]',\n",
              " 99: '[unused99]',\n",
              " 100: '[UNK]',\n",
              " 101: '[CLS]',\n",
              " 102: '[SEP]',\n",
              " 103: '[MASK]',\n",
              " 104: '<S>',\n",
              " 105: '<T>',\n",
              " 106: '!',\n",
              " 107: '\"',\n",
              " 108: '#',\n",
              " 109: '$',\n",
              " 110: '%',\n",
              " 111: '&',\n",
              " 112: \"'\",\n",
              " 113: '(',\n",
              " 114: ')',\n",
              " 115: '*',\n",
              " 116: '+',\n",
              " 117: ',',\n",
              " 118: '-',\n",
              " 119: '.',\n",
              " 120: '/',\n",
              " 121: '0',\n",
              " 122: '1',\n",
              " 123: '2',\n",
              " 124: '3',\n",
              " 125: '4',\n",
              " 126: '5',\n",
              " 127: '6',\n",
              " 128: '7',\n",
              " 129: '8',\n",
              " 130: '9',\n",
              " 131: ':',\n",
              " 132: ';',\n",
              " 133: '<',\n",
              " 134: '=',\n",
              " 135: '>',\n",
              " 136: '?',\n",
              " 137: '@',\n",
              " 138: 'A',\n",
              " 139: 'B',\n",
              " 140: 'C',\n",
              " 141: 'D',\n",
              " 142: 'E',\n",
              " 143: 'F',\n",
              " 144: 'G',\n",
              " 145: 'H',\n",
              " 146: 'I',\n",
              " 147: 'J',\n",
              " 148: 'K',\n",
              " 149: 'L',\n",
              " 150: 'M',\n",
              " 151: 'N',\n",
              " 152: 'O',\n",
              " 153: 'P',\n",
              " 154: 'Q',\n",
              " 155: 'R',\n",
              " 156: 'S',\n",
              " 157: 'T',\n",
              " 158: 'U',\n",
              " 159: 'V',\n",
              " 160: 'W',\n",
              " 161: 'X',\n",
              " 162: 'Y',\n",
              " 163: 'Z',\n",
              " 164: '[',\n",
              " 165: '\\\\',\n",
              " 166: ']',\n",
              " 167: '^',\n",
              " 168: '##',\n",
              " 169: 'a',\n",
              " 170: 'b',\n",
              " 171: 'c',\n",
              " 172: 'd',\n",
              " 173: 'e',\n",
              " 174: 'f',\n",
              " 175: 'g',\n",
              " 176: 'h',\n",
              " 177: 'i',\n",
              " 178: 'j',\n",
              " 179: 'k',\n",
              " 180: 'l',\n",
              " 181: 'm',\n",
              " 182: 'n',\n",
              " 183: 'o',\n",
              " 184: 'p',\n",
              " 185: 'q',\n",
              " 186: 'r',\n",
              " 187: 's',\n",
              " 188: 't',\n",
              " 189: 'u',\n",
              " 190: 'v',\n",
              " 191: 'w',\n",
              " 192: 'x',\n",
              " 193: 'y',\n",
              " 194: 'z',\n",
              " 195: '{',\n",
              " 196: '|',\n",
              " 197: '}',\n",
              " 198: '~',\n",
              " 199: '¡',\n",
              " 200: '¢',\n",
              " 201: '£',\n",
              " 202: '¥',\n",
              " 203: '¦',\n",
              " 204: '§',\n",
              " 205: '¨',\n",
              " 206: '©',\n",
              " 207: 'ª',\n",
              " 208: '«',\n",
              " 209: '¬',\n",
              " 210: '®',\n",
              " 211: '°',\n",
              " 212: '±',\n",
              " 213: '²',\n",
              " 214: '³',\n",
              " 215: 'µ',\n",
              " 216: '¶',\n",
              " 217: '·',\n",
              " 218: '¹',\n",
              " 219: 'º',\n",
              " 220: '»',\n",
              " 221: '¼',\n",
              " 222: '½',\n",
              " 223: '¾',\n",
              " 224: '¿',\n",
              " 225: 'À',\n",
              " 226: 'Á',\n",
              " 227: 'Â',\n",
              " 228: 'Ã',\n",
              " 229: 'Ä',\n",
              " 230: 'Å',\n",
              " 231: 'Æ',\n",
              " 232: 'Ç',\n",
              " 233: 'È',\n",
              " 234: 'É',\n",
              " 235: 'Ê',\n",
              " 236: 'Ë',\n",
              " 237: 'Ì',\n",
              " 238: 'Í',\n",
              " 239: 'Î',\n",
              " 240: 'Ð',\n",
              " 241: 'Ñ',\n",
              " 242: 'Ò',\n",
              " 243: 'Ó',\n",
              " 244: 'Ô',\n",
              " 245: 'Õ',\n",
              " 246: 'Ö',\n",
              " 247: '×',\n",
              " 248: 'Ø',\n",
              " 249: 'Ú',\n",
              " 250: 'Ü',\n",
              " 251: 'Ý',\n",
              " 252: 'Þ',\n",
              " 253: 'ß',\n",
              " 254: 'à',\n",
              " 255: 'á',\n",
              " 256: 'â',\n",
              " 257: 'ã',\n",
              " 258: 'ä',\n",
              " 259: 'å',\n",
              " 260: 'æ',\n",
              " 261: 'ç',\n",
              " 262: 'è',\n",
              " 263: 'é',\n",
              " 264: 'ê',\n",
              " 265: 'ë',\n",
              " 266: 'ì',\n",
              " 267: 'í',\n",
              " 268: 'î',\n",
              " 269: 'ï',\n",
              " 270: 'ð',\n",
              " 271: 'ñ',\n",
              " 272: 'ò',\n",
              " 273: 'ó',\n",
              " 274: 'ô',\n",
              " 275: 'õ',\n",
              " 276: 'ö',\n",
              " 277: '÷',\n",
              " 278: 'ø',\n",
              " 279: 'ù',\n",
              " 280: 'ú',\n",
              " 281: 'û',\n",
              " 282: 'ü',\n",
              " 283: 'ý',\n",
              " 284: 'þ',\n",
              " 285: 'ÿ',\n",
              " 286: 'Ā',\n",
              " 287: 'ā',\n",
              " 288: 'Ă',\n",
              " 289: 'ă',\n",
              " 290: 'Ą',\n",
              " 291: 'ą',\n",
              " 292: 'Ć',\n",
              " 293: 'ć',\n",
              " 294: 'Č',\n",
              " 295: 'č',\n",
              " 296: 'Ď',\n",
              " 297: 'ď',\n",
              " 298: 'Đ',\n",
              " 299: 'đ',\n",
              " 300: 'Ē',\n",
              " 301: 'ē',\n",
              " 302: 'Ĕ',\n",
              " 303: 'ĕ',\n",
              " 304: 'Ė',\n",
              " 305: 'ė',\n",
              " 306: 'ę',\n",
              " 307: 'ě',\n",
              " 308: 'Ğ',\n",
              " 309: 'ğ',\n",
              " 310: 'ġ',\n",
              " 311: 'Ģ',\n",
              " 312: 'ģ',\n",
              " 313: 'Ħ',\n",
              " 314: 'ħ',\n",
              " 315: 'ĩ',\n",
              " 316: 'Ī',\n",
              " 317: 'ī',\n",
              " 318: 'Į',\n",
              " 319: 'į',\n",
              " 320: 'İ',\n",
              " 321: 'ı',\n",
              " 322: 'Ķ',\n",
              " 323: 'ķ',\n",
              " 324: 'ĺ',\n",
              " 325: 'Ļ',\n",
              " 326: 'ļ',\n",
              " 327: 'Ľ',\n",
              " 328: 'ľ',\n",
              " 329: 'Ł',\n",
              " 330: 'ł',\n",
              " 331: 'ń',\n",
              " 332: 'Ņ',\n",
              " 333: 'ņ',\n",
              " 334: 'ň',\n",
              " 335: 'ŉ',\n",
              " 336: 'ŋ',\n",
              " 337: 'Ō',\n",
              " 338: 'ō',\n",
              " 339: 'ŏ',\n",
              " 340: 'Ő',\n",
              " 341: 'ő',\n",
              " 342: 'Œ',\n",
              " 343: 'œ',\n",
              " 344: 'ŕ',\n",
              " 345: 'Ř',\n",
              " 346: 'ř',\n",
              " 347: 'Ś',\n",
              " 348: 'ś',\n",
              " 349: 'Ş',\n",
              " 350: 'ş',\n",
              " 351: 'Š',\n",
              " 352: 'š',\n",
              " 353: 'Ţ',\n",
              " 354: 'ţ',\n",
              " 355: 'Ť',\n",
              " 356: 'ť',\n",
              " 357: 'ũ',\n",
              " 358: 'Ū',\n",
              " 359: 'ū',\n",
              " 360: 'ŭ',\n",
              " 361: 'ů',\n",
              " 362: 'Ű',\n",
              " 363: 'ű',\n",
              " 364: 'ų',\n",
              " 365: 'ŵ',\n",
              " 366: 'ŷ',\n",
              " 367: 'Ź',\n",
              " 368: 'ź',\n",
              " 369: 'Ż',\n",
              " 370: 'ż',\n",
              " 371: 'Ž',\n",
              " 372: 'ž',\n",
              " 373: 'Ə',\n",
              " 374: 'ƒ',\n",
              " 375: 'ơ',\n",
              " 376: 'Ư',\n",
              " 377: 'ư',\n",
              " 378: 'ǎ',\n",
              " 379: 'ǐ',\n",
              " 380: 'ǔ',\n",
              " 381: 'ǫ',\n",
              " 382: 'ǹ',\n",
              " 383: 'Ș',\n",
              " 384: 'ș',\n",
              " 385: 'Ț',\n",
              " 386: 'ț',\n",
              " 387: 'ɐ',\n",
              " 388: 'ɑ',\n",
              " 389: 'ɔ',\n",
              " 390: 'ɕ',\n",
              " 391: 'ə',\n",
              " 392: 'ɛ',\n",
              " 393: 'ɡ',\n",
              " 394: 'ɣ',\n",
              " 395: 'ɨ',\n",
              " 396: 'ɪ',\n",
              " 397: 'ɲ',\n",
              " 398: 'ɾ',\n",
              " 399: 'ʁ',\n",
              " 400: 'ʃ',\n",
              " 401: 'ʊ',\n",
              " 402: 'ʎ',\n",
              " 403: 'ʒ',\n",
              " 404: 'ʔ',\n",
              " 405: 'ʙ',\n",
              " 406: 'ʰ',\n",
              " 407: 'ʲ',\n",
              " 408: 'ʳ',\n",
              " 409: 'ʷ',\n",
              " 410: 'ʸ',\n",
              " 411: 'ʻ',\n",
              " 412: 'ʼ',\n",
              " 413: 'ʾ',\n",
              " 414: 'ʿ',\n",
              " 415: 'ˈ',\n",
              " 416: 'ː',\n",
              " 417: 'ˡ',\n",
              " 418: 'ˢ',\n",
              " 419: '̀',\n",
              " 420: '́',\n",
              " 421: '̃',\n",
              " 422: '̄',\n",
              " 423: '̍',\n",
              " 424: '̥',\n",
              " 425: '̧',\n",
              " 426: '̲',\n",
              " 427: '͡',\n",
              " 428: '΄',\n",
              " 429: 'Ά',\n",
              " 430: 'Έ',\n",
              " 431: 'Ή',\n",
              " 432: 'Ί',\n",
              " 433: 'Ό',\n",
              " 434: 'Ύ',\n",
              " 435: 'Ώ',\n",
              " 436: 'ΐ',\n",
              " 437: 'Α',\n",
              " 438: 'Β',\n",
              " 439: 'Γ',\n",
              " 440: 'Δ',\n",
              " 441: 'Ε',\n",
              " 442: 'Ζ',\n",
              " 443: 'Η',\n",
              " 444: 'Θ',\n",
              " 445: 'Ι',\n",
              " 446: 'Κ',\n",
              " 447: 'Λ',\n",
              " 448: 'Μ',\n",
              " 449: 'Ν',\n",
              " 450: 'Ξ',\n",
              " 451: 'Ο',\n",
              " 452: 'Π',\n",
              " 453: 'Ρ',\n",
              " 454: 'Σ',\n",
              " 455: 'Τ',\n",
              " 456: 'Υ',\n",
              " 457: 'Φ',\n",
              " 458: 'Χ',\n",
              " 459: 'Ψ',\n",
              " 460: 'Ω',\n",
              " 461: 'ά',\n",
              " 462: 'έ',\n",
              " 463: 'ή',\n",
              " 464: 'ί',\n",
              " 465: 'α',\n",
              " 466: 'β',\n",
              " 467: 'γ',\n",
              " 468: 'δ',\n",
              " 469: 'ε',\n",
              " 470: 'ζ',\n",
              " 471: 'η',\n",
              " 472: 'θ',\n",
              " 473: 'ι',\n",
              " 474: 'κ',\n",
              " 475: 'λ',\n",
              " 476: 'μ',\n",
              " 477: 'ν',\n",
              " 478: 'ξ',\n",
              " 479: 'ο',\n",
              " 480: 'π',\n",
              " 481: 'ρ',\n",
              " 482: 'ς',\n",
              " 483: 'σ',\n",
              " 484: 'τ',\n",
              " 485: 'υ',\n",
              " 486: 'φ',\n",
              " 487: 'χ',\n",
              " 488: 'ψ',\n",
              " 489: 'ω',\n",
              " 490: 'ϊ',\n",
              " 491: 'ϋ',\n",
              " 492: 'ό',\n",
              " 493: 'ύ',\n",
              " 494: 'ώ',\n",
              " 495: 'Ё',\n",
              " 496: 'Ђ',\n",
              " 497: 'Ѓ',\n",
              " 498: 'Є',\n",
              " 499: 'Ѕ',\n",
              " 500: 'І',\n",
              " 501: 'Ї',\n",
              " 502: 'Ј',\n",
              " 503: 'Љ',\n",
              " 504: 'Њ',\n",
              " 505: 'Ћ',\n",
              " 506: 'Ќ',\n",
              " 507: 'Ў',\n",
              " 508: 'Џ',\n",
              " 509: 'А',\n",
              " 510: 'Б',\n",
              " 511: 'В',\n",
              " 512: 'Г',\n",
              " 513: 'Д',\n",
              " 514: 'Е',\n",
              " 515: 'Ж',\n",
              " 516: 'З',\n",
              " 517: 'И',\n",
              " 518: 'Й',\n",
              " 519: 'К',\n",
              " 520: 'Л',\n",
              " 521: 'М',\n",
              " 522: 'Н',\n",
              " 523: 'О',\n",
              " 524: 'П',\n",
              " 525: 'Р',\n",
              " 526: 'С',\n",
              " 527: 'Т',\n",
              " 528: 'У',\n",
              " 529: 'Ф',\n",
              " 530: 'Х',\n",
              " 531: 'Ц',\n",
              " 532: 'Ч',\n",
              " 533: 'Ш',\n",
              " 534: 'Щ',\n",
              " 535: 'Ъ',\n",
              " 536: 'Ы',\n",
              " 537: 'Ь',\n",
              " 538: 'Э',\n",
              " 539: 'Ю',\n",
              " 540: 'Я',\n",
              " 541: 'а',\n",
              " 542: 'б',\n",
              " 543: 'в',\n",
              " 544: 'г',\n",
              " 545: 'д',\n",
              " 546: 'е',\n",
              " 547: 'ж',\n",
              " 548: 'з',\n",
              " 549: 'и',\n",
              " 550: 'й',\n",
              " 551: 'к',\n",
              " 552: 'л',\n",
              " 553: 'м',\n",
              " 554: 'н',\n",
              " 555: 'о',\n",
              " 556: 'п',\n",
              " 557: 'р',\n",
              " 558: 'с',\n",
              " 559: 'т',\n",
              " 560: 'у',\n",
              " 561: 'ф',\n",
              " 562: 'х',\n",
              " 563: 'ц',\n",
              " 564: 'ч',\n",
              " 565: 'ш',\n",
              " 566: 'щ',\n",
              " 567: 'ъ',\n",
              " 568: 'ы',\n",
              " 569: 'ь',\n",
              " 570: 'э',\n",
              " 571: 'ю',\n",
              " 572: 'я',\n",
              " 573: 'ѐ',\n",
              " 574: 'ё',\n",
              " 575: 'ђ',\n",
              " 576: 'ѓ',\n",
              " 577: 'є',\n",
              " 578: 'ѕ',\n",
              " 579: 'і',\n",
              " 580: 'ї',\n",
              " 581: 'ј',\n",
              " 582: 'љ',\n",
              " 583: 'њ',\n",
              " 584: 'ћ',\n",
              " 585: 'ќ',\n",
              " 586: 'ѝ',\n",
              " 587: 'ў',\n",
              " 588: 'џ',\n",
              " 589: 'ѣ',\n",
              " 590: 'Ґ',\n",
              " 591: 'ґ',\n",
              " 592: 'Ғ',\n",
              " 593: 'ғ',\n",
              " 594: 'Җ',\n",
              " 595: 'җ',\n",
              " 596: 'Ҙ',\n",
              " 597: 'ҙ',\n",
              " 598: 'Қ',\n",
              " 599: 'қ',\n",
              " 600: 'Ҡ',\n",
              " 601: 'ҡ',\n",
              " 602: 'ң',\n",
              " 603: 'ҫ',\n",
              " 604: 'Ү',\n",
              " 605: 'ү',\n",
              " 606: 'Ұ',\n",
              " 607: 'ұ',\n",
              " 608: 'Ҳ',\n",
              " 609: 'ҳ',\n",
              " 610: 'Ҷ',\n",
              " 611: 'ҷ',\n",
              " 612: 'Һ',\n",
              " 613: 'һ',\n",
              " 614: 'Ӏ',\n",
              " 615: 'ӑ',\n",
              " 616: 'ӗ',\n",
              " 617: 'Ә',\n",
              " 618: 'ә',\n",
              " 619: 'ӣ',\n",
              " 620: 'Ө',\n",
              " 621: 'ө',\n",
              " 622: 'Ӯ',\n",
              " 623: 'ӯ',\n",
              " 624: 'ӳ',\n",
              " 625: 'Ա',\n",
              " 626: 'Բ',\n",
              " 627: 'Գ',\n",
              " 628: 'Դ',\n",
              " 629: 'Ե',\n",
              " 630: 'Զ',\n",
              " 631: 'Է',\n",
              " 632: 'Ը',\n",
              " 633: 'Թ',\n",
              " 634: 'Ժ',\n",
              " 635: 'Ի',\n",
              " 636: 'Լ',\n",
              " 637: 'Խ',\n",
              " 638: 'Ծ',\n",
              " 639: 'Կ',\n",
              " 640: 'Հ',\n",
              " 641: 'Ձ',\n",
              " 642: 'Ղ',\n",
              " 643: 'Ճ',\n",
              " 644: 'Մ',\n",
              " 645: 'Յ',\n",
              " 646: 'Ն',\n",
              " 647: 'Շ',\n",
              " 648: 'Ո',\n",
              " 649: 'Չ',\n",
              " 650: 'Պ',\n",
              " 651: 'Ջ',\n",
              " 652: 'Ռ',\n",
              " 653: 'Ս',\n",
              " 654: 'Վ',\n",
              " 655: 'Տ',\n",
              " 656: 'Ր',\n",
              " 657: 'Ց',\n",
              " 658: 'Ւ',\n",
              " 659: 'Փ',\n",
              " 660: 'Ք',\n",
              " 661: 'Օ',\n",
              " 662: 'Ֆ',\n",
              " 663: '՚',\n",
              " 664: '՛',\n",
              " 665: '՜',\n",
              " 666: '՝',\n",
              " 667: '՞',\n",
              " 668: 'ա',\n",
              " 669: 'բ',\n",
              " 670: 'գ',\n",
              " 671: 'դ',\n",
              " 672: 'ե',\n",
              " 673: 'զ',\n",
              " 674: 'է',\n",
              " 675: 'ը',\n",
              " 676: 'թ',\n",
              " 677: 'ժ',\n",
              " 678: 'ի',\n",
              " 679: 'լ',\n",
              " 680: 'խ',\n",
              " 681: 'ծ',\n",
              " 682: 'կ',\n",
              " 683: 'հ',\n",
              " 684: 'ձ',\n",
              " 685: 'ղ',\n",
              " 686: 'ճ',\n",
              " 687: 'մ',\n",
              " 688: 'յ',\n",
              " 689: 'ն',\n",
              " 690: 'շ',\n",
              " 691: 'ո',\n",
              " 692: 'չ',\n",
              " 693: 'պ',\n",
              " 694: 'ջ',\n",
              " 695: 'ռ',\n",
              " 696: 'ս',\n",
              " 697: 'վ',\n",
              " 698: 'տ',\n",
              " 699: 'ր',\n",
              " 700: 'ց',\n",
              " 701: 'ւ',\n",
              " 702: 'փ',\n",
              " 703: 'ք',\n",
              " 704: 'օ',\n",
              " 705: 'ֆ',\n",
              " 706: 'և',\n",
              " 707: '։',\n",
              " 708: '֊',\n",
              " 709: 'ְ',\n",
              " 710: 'ֱ',\n",
              " 711: 'ֲ',\n",
              " 712: 'ִ',\n",
              " 713: 'ֵ',\n",
              " 714: 'ֶ',\n",
              " 715: 'ַ',\n",
              " 716: 'ָ',\n",
              " 717: 'ֹ',\n",
              " 718: 'ּ',\n",
              " 719: '־',\n",
              " 720: 'ׁ',\n",
              " 721: 'ׂ',\n",
              " 722: '׃',\n",
              " 723: 'א',\n",
              " 724: 'ב',\n",
              " 725: 'ג',\n",
              " 726: 'ד',\n",
              " 727: 'ה',\n",
              " 728: 'ו',\n",
              " 729: 'ז',\n",
              " 730: 'ח',\n",
              " 731: 'ט',\n",
              " 732: 'י',\n",
              " 733: 'ך',\n",
              " 734: 'כ',\n",
              " 735: 'ל',\n",
              " 736: 'ם',\n",
              " 737: 'מ',\n",
              " 738: 'ן',\n",
              " 739: 'נ',\n",
              " 740: 'ס',\n",
              " 741: 'ע',\n",
              " 742: 'ף',\n",
              " 743: 'פ',\n",
              " 744: 'ץ',\n",
              " 745: 'צ',\n",
              " 746: 'ק',\n",
              " 747: 'ר',\n",
              " 748: 'ש',\n",
              " 749: 'ת',\n",
              " 750: '׳',\n",
              " 751: '״',\n",
              " 752: '،',\n",
              " 753: '؍',\n",
              " 754: 'ؒ',\n",
              " 755: '؛',\n",
              " 756: '؟',\n",
              " 757: 'ء',\n",
              " 758: 'آ',\n",
              " 759: 'أ',\n",
              " 760: 'ؤ',\n",
              " 761: 'إ',\n",
              " 762: 'ئ',\n",
              " 763: 'ا',\n",
              " 764: 'ب',\n",
              " 765: 'ة',\n",
              " 766: 'ت',\n",
              " 767: 'ث',\n",
              " 768: 'ج',\n",
              " 769: 'ح',\n",
              " 770: 'خ',\n",
              " 771: 'د',\n",
              " 772: 'ذ',\n",
              " 773: 'ر',\n",
              " 774: 'ز',\n",
              " 775: 'س',\n",
              " 776: 'ش',\n",
              " 777: 'ص',\n",
              " 778: 'ض',\n",
              " 779: 'ط',\n",
              " 780: 'ظ',\n",
              " 781: 'ع',\n",
              " 782: 'غ',\n",
              " 783: 'ـ',\n",
              " 784: 'ف',\n",
              " 785: 'ق',\n",
              " 786: 'ك',\n",
              " 787: 'ل',\n",
              " 788: 'م',\n",
              " 789: 'ن',\n",
              " 790: 'ه',\n",
              " 791: 'و',\n",
              " 792: 'ى',\n",
              " 793: 'ي',\n",
              " 794: 'ً',\n",
              " 795: 'ٌ',\n",
              " 796: 'ٍ',\n",
              " 797: 'َ',\n",
              " 798: 'ُ',\n",
              " 799: 'ِ',\n",
              " 800: 'ّ',\n",
              " 801: 'ْ',\n",
              " 802: 'ٔ',\n",
              " 803: 'ٛ',\n",
              " 804: '٠',\n",
              " 805: '١',\n",
              " 806: '٢',\n",
              " 807: '٣',\n",
              " 808: '٤',\n",
              " 809: '٥',\n",
              " 810: '٩',\n",
              " 811: '٪',\n",
              " 812: '٫',\n",
              " 813: '٬',\n",
              " 814: '٭',\n",
              " 815: 'ٰ',\n",
              " 816: 'ٹ',\n",
              " 817: 'پ',\n",
              " 818: 'چ',\n",
              " 819: 'ڈ',\n",
              " 820: 'ڑ',\n",
              " 821: 'ژ',\n",
              " 822: 'ڤ',\n",
              " 823: 'ک',\n",
              " 824: 'ڭ',\n",
              " 825: 'گ',\n",
              " 826: 'ں',\n",
              " 827: 'ھ',\n",
              " 828: 'ۀ',\n",
              " 829: 'ہ',\n",
              " 830: 'ۂ',\n",
              " 831: 'ۃ',\n",
              " 832: 'ۆ',\n",
              " 833: 'ۇ',\n",
              " 834: 'ی',\n",
              " 835: 'ے',\n",
              " 836: 'ۓ',\n",
              " 837: '۔',\n",
              " 838: '۰',\n",
              " 839: '۱',\n",
              " 840: '۲',\n",
              " 841: '۳',\n",
              " 842: '۴',\n",
              " 843: '۵',\n",
              " 844: '۶',\n",
              " 845: '۷',\n",
              " 846: '۸',\n",
              " 847: '۹',\n",
              " 848: 'ँ',\n",
              " 849: 'ं',\n",
              " 850: 'ः',\n",
              " 851: 'अ',\n",
              " 852: 'आ',\n",
              " 853: 'इ',\n",
              " 854: 'ई',\n",
              " 855: 'उ',\n",
              " 856: 'ऊ',\n",
              " 857: 'ऋ',\n",
              " 858: 'ऍ',\n",
              " 859: 'ऎ',\n",
              " 860: 'ए',\n",
              " 861: 'ऐ',\n",
              " 862: 'ऑ',\n",
              " 863: 'ओ',\n",
              " 864: 'औ',\n",
              " 865: 'क',\n",
              " 866: 'ख',\n",
              " 867: 'ग',\n",
              " 868: 'घ',\n",
              " 869: 'ङ',\n",
              " 870: 'च',\n",
              " 871: 'छ',\n",
              " 872: 'ज',\n",
              " 873: 'झ',\n",
              " 874: 'ञ',\n",
              " 875: 'ट',\n",
              " 876: 'ठ',\n",
              " 877: 'ड',\n",
              " 878: 'ढ',\n",
              " 879: 'ण',\n",
              " 880: 'त',\n",
              " 881: 'थ',\n",
              " 882: 'द',\n",
              " 883: 'ध',\n",
              " 884: 'न',\n",
              " 885: 'प',\n",
              " 886: 'फ',\n",
              " 887: 'ब',\n",
              " 888: 'भ',\n",
              " 889: 'म',\n",
              " 890: 'य',\n",
              " 891: 'र',\n",
              " 892: 'ऱ',\n",
              " 893: 'ल',\n",
              " 894: 'ळ',\n",
              " 895: 'व',\n",
              " 896: 'श',\n",
              " 897: 'ष',\n",
              " 898: 'स',\n",
              " 899: 'ह',\n",
              " 900: '़',\n",
              " 901: 'ऽ',\n",
              " 902: 'ा',\n",
              " 903: 'ि',\n",
              " 904: 'ी',\n",
              " 905: 'ु',\n",
              " 906: 'ू',\n",
              " 907: 'ृ',\n",
              " 908: 'ॄ',\n",
              " 909: 'ॅ',\n",
              " 910: 'ॆ',\n",
              " 911: 'े',\n",
              " 912: 'ै',\n",
              " 913: 'ॉ',\n",
              " 914: 'ो',\n",
              " 915: 'ौ',\n",
              " 916: '्',\n",
              " 917: 'ॐ',\n",
              " 918: '॓',\n",
              " 919: 'ॠ',\n",
              " 920: '।',\n",
              " 921: '॥',\n",
              " 922: '०',\n",
              " 923: '१',\n",
              " 924: '२',\n",
              " 925: '३',\n",
              " 926: '४',\n",
              " 927: '५',\n",
              " 928: '६',\n",
              " 929: '७',\n",
              " 930: '८',\n",
              " 931: '९',\n",
              " 932: '॰',\n",
              " 933: 'ॲ',\n",
              " 934: 'ঁ',\n",
              " 935: 'ং',\n",
              " 936: 'ঃ',\n",
              " 937: 'অ',\n",
              " 938: 'আ',\n",
              " 939: 'ই',\n",
              " 940: 'ঈ',\n",
              " 941: 'উ',\n",
              " 942: 'ঊ',\n",
              " 943: 'ঋ',\n",
              " 944: 'এ',\n",
              " 945: 'ঐ',\n",
              " 946: 'ও',\n",
              " 947: 'ঔ',\n",
              " 948: 'ক',\n",
              " 949: 'খ',\n",
              " 950: 'গ',\n",
              " 951: 'ঘ',\n",
              " 952: 'ঙ',\n",
              " 953: 'চ',\n",
              " 954: 'ছ',\n",
              " 955: 'জ',\n",
              " 956: 'ঝ',\n",
              " 957: 'ঞ',\n",
              " 958: 'ট',\n",
              " 959: 'ঠ',\n",
              " 960: 'ড',\n",
              " 961: 'ঢ',\n",
              " 962: 'ণ',\n",
              " 963: 'ত',\n",
              " 964: 'থ',\n",
              " 965: 'দ',\n",
              " 966: 'ধ',\n",
              " 967: 'ন',\n",
              " 968: 'প',\n",
              " 969: 'ফ',\n",
              " 970: 'ব',\n",
              " 971: 'ভ',\n",
              " 972: 'ম',\n",
              " 973: 'য',\n",
              " 974: 'র',\n",
              " 975: 'ল',\n",
              " 976: 'শ',\n",
              " 977: 'ষ',\n",
              " 978: 'স',\n",
              " 979: 'হ',\n",
              " 980: '়',\n",
              " 981: 'া',\n",
              " 982: 'ি',\n",
              " 983: 'ী',\n",
              " 984: 'ু',\n",
              " 985: 'ূ',\n",
              " 986: 'ৃ',\n",
              " 987: 'ে',\n",
              " 988: 'ৈ',\n",
              " 989: 'ো',\n",
              " 990: 'ৌ',\n",
              " 991: '্',\n",
              " 992: 'ৎ',\n",
              " 993: '০',\n",
              " 994: '১',\n",
              " 995: '২',\n",
              " 996: '৩',\n",
              " 997: '৪',\n",
              " 998: '৫',\n",
              " 999: '৬',\n",
              " ...}"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 23
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "I9sFKLjD-v2n",
        "colab_type": "text"
      },
      "source": [
        "**버트 모형에 들어갈 인풋은 토큰, 세그먼트, 포지션으로 구성됩니다.**  \n",
        "버트에 인풋으로 들어가는 토큰은 문장을 토크나이징 한 후, 인덱스 번호를 매긴 것입니다.  \n",
        "세그먼트는 예를 들어 문장이 두 개가 있다면, 앞의 문장과 뒤의 문장을 구분하는 것입니다.  \n",
        "포지션 임베딩은 단순히 단어의 위치를 말합니다.\n",
        "\n",
        "토큰, 세그먼트, 포지션을 인풋으로 버트 모형에 넣으면 기하학적인 문장 공간으로 임베딩이 됩니다.  \n",
        "그림을 보면 my dog is cute, he likes play ##ing 두 문장이 있는데요  \n",
        "SQUAD 문제에서는 첫번째 문장이 질문, 두번째 문장이 context가 되고, 이 두 문장이 합쳐져서 하나의 문장으로 들어가게 됩니다.\n",
        "\n",
        "![대체 텍스트](https://i.imgur.com/l9BTao3.png)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oXdjhy0cLlYA",
        "colab_type": "text"
      },
      "source": [
        "- 사전학습된 버트 모델의 인풋은 문장 토큰화가 숫자로 바뀐 것과, 앞문장인지 뒷문장인지 알려주는 문장 순서 벡터가 들어갑니다. 우리는 문장 하나를 가지고만 훈련할 것이므로 순서 벡터는 모두 0으로 통일합니다.\n",
        "\n",
        "- 그리고 파인튜닝 시에는 문장 안에 일부 단어를 가리는 마스킹은 사용하지 않습니다."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Fraw-fe9PL1k",
        "colab_type": "text"
      },
      "source": [
        "우리가 로드하였던 SQUAD 데이터를 **버트 모형의 입력에 맞게 변형해주는 함수**를 정의하도록 하겠습니다.\n",
        "\n",
        "함수 내부에 tokenizer.encode 함수가 버트 모형을 토큰화해주고 토큰화 된 단어를 인덱스에 맞게 숫자로 바꿔주게 됩니다.  \n",
        "**[CLS] 질문 [SEP] 문장 [SEP]** 이런 방식으로 질문과 문장이 인풋으로 들어가게 됩니다. SEQ_LEN이 384로 지정되어 있어서 길이가 384가 넘는 인풋은 문장 부분이 잘려서 들어가게 됩니다.  \n",
        "SQUAD 문제에서 문장(context) 내에 text(정답)이 포함된다고 미리 말씀 드렸는데요, 길이가 384가 넘는 인풋인 경우에 context 내에 정답을 포함하고 있는 context가 잘려서 정답을 포함하지 않는 경우가 생깁니다. 이번 실습에서는 이러한 경우의 인덱스를 del_list로 지정해서, 빼도록 하겠습니다. (숙제로 남겨 두겠습니다.) "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jYrlZPd3YExo",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "question = train['question'][0]\n",
        "context = train['context'][0]\n",
        "text = train['text'][0]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xjcdonPiY3U9",
        "colab_type": "code",
        "outputId": "90e9feeb-5aef-47a4-eab3-0b140b188892",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "# 질문\n",
        "question"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'To whom did the Virgin Mary allegedly appear in 1858 in Lourdes France?'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 25
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2tRbEbSCY4dx",
        "colab_type": "code",
        "outputId": "d781462f-2614-49dc-81b1-e9b6f3a927f6",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 54
        }
      },
      "source": [
        "# 문장\n",
        "context"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'Architecturally, the school has a Catholic character. Atop the Main Building\\'s gold dome is a golden statue of the Virgin Mary. Immediately in front of the Main Building and facing it, is a copper statue of Christ with arms upraised with the legend \"Venite Ad Me Omnes\". Next to the Main Building is the Basilica of the Sacred Heart. Immediately behind the basilica is the Grotto, a Marian place of prayer and reflection. It is a replica of the grotto at Lourdes, France where the Virgin Mary reputedly appeared to Saint Bernadette Soubirous in 1858. At the end of the main drive (and in a direct line that connects through 3 statues and the Gold Dome), is a simple, modern stone statue of Mary.'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 26
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZhMGIFOhY5hz",
        "colab_type": "code",
        "outputId": "eb57700d-6c6f-468d-abb3-493d8aeb3bf5",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "# 정답\n",
        "text"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'Saint Bernadette Soubirous'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 27
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "10hBmhcvY68i",
        "colab_type": "code",
        "outputId": "9cf606c7-c26f-4a2f-a0a6-edf6aacb400e",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 54
        }
      },
      "source": [
        "print(tokenizer.tokenize(question, context))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "['[CLS]', 'to', 'whom', 'did', 'the', 'vir', '##gin', 'mar', '##y', 'allegedly', 'appear', 'in', '1858', 'in', 'lo', '##urde', '##s', 'franc', '##e', '?', '[SEP]', 'architectural', '##ly', ',', 'the', 'school', 'has', 'a', 'cat', '##hol', '##ic', 'character', '.', 'ato', '##p', 'the', 'main', 'building', \"'\", 's', 'gold', 'dome', 'is', 'a', 'golden', 'statue', 'of', 'the', 'vir', '##gin', 'mar', '##y', '.', 'immediately', 'in', 'front', 'of', 'the', 'main', 'building', 'and', 'facing', 'it', ',', 'is', 'a', 'copper', 'statue', 'of', 'ch', '##rist', 'with', 'arms', 'up', '##rais', '##ed', 'with', 'the', 'legend', '\"', 'ven', '##ite', 'ad', 'me', 'om', '##nes', '\"', '.', 'next', 'to', 'the', 'main', 'building', 'is', 'the', 'basilica', 'of', 'the', 'sacred', 'heart', '.', 'immediately', 'behind', 'the', 'basilica', 'is', 'the', 'gr', '##otto', ',', 'a', 'mari', '##an', 'place', 'of', 'prayer', 'and', 'reflect', '##ion', '.', 'it', 'is', 'a', 'replica', 'of', 'the', 'gr', '##otto', 'at', 'lo', '##urde', '##s', ',', 'franc', '##e', 'where', 'the', 'vir', '##gin', 'mar', '##y', 'rep', '##uted', '##ly', 'appeared', 'to', 'saint', 'bern', '##ade', '##tte', 'sou', '##bir', '##ous', 'in', '1858', '.', 'at', 'the', 'end', 'of', 'the', 'main', 'drive', '(', 'and', 'in', 'a', 'direct', 'line', 'that', 'connects', 'through', '3', 'statues', 'and', 'the', 'gold', 'dome', ')', ',', 'is', 'a', 'simple', ',', 'modern', 'stone', 'statue', 'of', 'mar', '##y', '.', '[SEP]']\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SvahjcksZIN9",
        "colab_type": "code",
        "outputId": "7889323f-e77e-4372-e988-84f5ccade488",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "print(tokenizer.tokenize(text))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "['[CLS]', 'saint', 'bern', '##ade', '##tte', 'sou', '##bir', '##ous', '[SEP]']\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fgOlJH_4ZQpS",
        "colab_type": "text"
      },
      "source": [
        "우리의 목표는, 질문(question)과 문장(context)를 받아서, 정답(text)를 맞추는 모델을 만드는 것입니다.  \n",
        "정답을 통째로 맞추는 것이 아니라, **토큰화된 것의 맨 앞 단어와, 맨 뒷 단어입니다.**  \n",
        "토큰화된 정답은 ['[CLS]', 'saint', 'bern', '##ade', '##tte', 'sou', '##bir', '##ous', '[SEP]'] 인데, 여기서 **saint**에 해당하는 위치와 **##ous**에 해당하는 위치를 맞추는 버트 모형을 파인튜닝 하려 하는 것입니다.  \n",
        "\n",
        "  그래서 밑에 convert_data 함수에서, 정답(text) 길이만큼 문장(context)를 슬라이딩 하면서 만약에 문장이 정답을 포함하는 위치에 도달하면, 문장에서 정답의 맨 앞이 우리가 예측할 1번째 정답, 정답의 맨 뒤가 우리가 예측할 2번째 정답이 되게 됩니다."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0rJsxRZDWw4e",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def convert_data(data_df):\n",
        "    global tokenizer\n",
        "    indices, segments, target_start, target_end = [], [], [], []\n",
        "    for i in tqdm(range(len(data_df))):\n",
        "        \n",
        "        ids, segment = tokenizer.encode(data_df[QUESTION_COLUMN][i], data_df[DATA_COLUMN][i], max_len=SEQ_LEN)\n",
        "        \n",
        "\n",
        "        text = tokenizer.encode(data_df[TEXT][i])[0]\n",
        "\n",
        "        text_slide_len = len(text[1:-1])\n",
        "        for i in range(1,len(ids)-text_slide_len-1):  \n",
        "            exist_flag = 0\n",
        "            if text[1:-1] == ids[i:i+text_slide_len]:\n",
        "              ans_start = i\n",
        "              ans_end = i + text_slide_len - 1\n",
        "              exist_flag = 1\n",
        "              break\n",
        "        \n",
        "        if exist_flag == 0:\n",
        "          ans_start = SEQ_LEN\n",
        "          ans_end = SEQ_LEN\n",
        "\n",
        "        indices.append(ids)\n",
        "        segments.append(segment)\n",
        "\n",
        "        target_start.append(ans_start)\n",
        "        target_end.append(ans_end)\n",
        "\n",
        "    indices_x = np.array(indices)\n",
        "    segments = np.array(segments)\n",
        "    target_start = np.array(target_start)\n",
        "    target_end = np.array(target_end)\n",
        "    \n",
        "    del_list = np.where(target_start!=SEQ_LEN)[0]\n",
        "\n",
        "    indices_x = indices_x[del_list]\n",
        "    segments = segments[del_list]\n",
        "    target_start = target_start[del_list]\n",
        "    target_end = target_end[del_list]\n",
        "\n",
        "    train_y_0 = keras.utils.to_categorical(target_start, num_classes=SEQ_LEN, dtype='int64')\n",
        "    train_y_1 = keras.utils.to_categorical(target_end, num_classes=SEQ_LEN, dtype='int64')\n",
        "    train_y_cat = [train_y_0, train_y_1]\n",
        "    \n",
        "    return [indices_x, segments], train_y_cat\n",
        "\n",
        "def load_data(pandas_dataframe):\n",
        "    data_df = pandas_dataframe\n",
        "    \n",
        "    \n",
        "    data_df[DATA_COLUMN] = data_df[DATA_COLUMN].astype(str)\n",
        "    data_df[QUESTION_COLUMN] = data_df[QUESTION_COLUMN].astype(str)\n",
        "\n",
        "\n",
        "    data_x, data_y = convert_data(data_df)\n",
        "\n",
        "    return data_x, data_y"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rNUYX-Rntsk_",
        "colab_type": "code",
        "outputId": "5a2436d6-d63d-4759-ac06-7da3c4c8a2b4",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "train_x, train_y = load_data(train)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "100%|██████████| 87599/87599 [02:56<00:00, 496.06it/s]\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vnh0ZuikSzz6",
        "colab_type": "text"
      },
      "source": [
        "#### 이해가 안 가실 수 있는데, 버트 인풋을 문장으로 예를 들어 만들어 보겠습니다.\n",
        "#### 인풋은 총 2개가 들어갑니다\n",
        "- **(토큰)** 첫번째 인풋은 토큰화 된 것이 인덱싱되어 숫자로 변환된 것  \n",
        "\n",
        "- **(세그멘트)** 두번째 인풋은 앞문장인지 뒷문장인지 알려주는 숫자들입니다. 이번 튜토리얼에서는 파인튜닝 과정이라 앞문장 뒷문장 구분을 안하기 때문에 모두 0으로 하였습니다.  \n",
        "\n",
        "- **(포지션)** 단어 순서에 따라서 자동으로 부여됩니다.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RZ23W0s8MCBt",
        "colab_type": "text"
      },
      "source": [
        "이해가 되셨는지요?  \n",
        "구글 깃허브에서 다운받았던 사전학습된 모델을 colab으로 로드합니다.  \n",
        "Training을 False로 두어서 Bert 모델에서, 마지막 트랜스포머 계층까지만 모델이 로드되게 합니다."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "scrolled": false,
        "id": "8auSqxdHtslD",
        "colab_type": "code",
        "outputId": "bb13acbe-ac8f-4759-93a7-940154604e74",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "layer_num = 12\n",
        "model = load_trained_model_from_checkpoint(\n",
        "    config_path,\n",
        "    checkpoint_path,\n",
        "    training=False,\n",
        "    trainable=True,\n",
        "    seq_len=SEQ_LEN,)\n",
        "model.summary()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model: \"model_2\"\n",
            "__________________________________________________________________________________________________\n",
            "Layer (type)                    Output Shape         Param #     Connected to                     \n",
            "==================================================================================================\n",
            "Input-Token (InputLayer)        (None, 384)          0                                            \n",
            "__________________________________________________________________________________________________\n",
            "Input-Segment (InputLayer)      (None, 384)          0                                            \n",
            "__________________________________________________________________________________________________\n",
            "Embedding-Token (TokenEmbedding [(None, 384, 768), ( 91812096    Input-Token[0][0]                \n",
            "__________________________________________________________________________________________________\n",
            "Embedding-Segment (Embedding)   (None, 384, 768)     1536        Input-Segment[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "Embedding-Token-Segment (Add)   (None, 384, 768)     0           Embedding-Token[0][0]            \n",
            "                                                                 Embedding-Segment[0][0]          \n",
            "__________________________________________________________________________________________________\n",
            "Embedding-Position (PositionEmb (None, 384, 768)     294912      Embedding-Token-Segment[0][0]    \n",
            "__________________________________________________________________________________________________\n",
            "Embedding-Dropout (Dropout)     (None, 384, 768)     0           Embedding-Position[0][0]         \n",
            "__________________________________________________________________________________________________\n",
            "Embedding-Norm (LayerNormalizat (None, 384, 768)     1536        Embedding-Dropout[0][0]          \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-1-MultiHeadSelfAttentio (None, 384, 768)     2362368     Embedding-Norm[0][0]             \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-1-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-1-MultiHeadSelfAttention[\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-1-MultiHeadSelfAttentio (None, 384, 768)     0           Embedding-Norm[0][0]             \n",
            "                                                                 Encoder-1-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-1-MultiHeadSelfAttentio (None, 384, 768)     1536        Encoder-1-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-1-FeedForward (FeedForw (None, 384, 768)     4722432     Encoder-1-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-1-FeedForward-Dropout ( (None, 384, 768)     0           Encoder-1-FeedForward[0][0]      \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-1-FeedForward-Add (Add) (None, 384, 768)     0           Encoder-1-MultiHeadSelfAttention-\n",
            "                                                                 Encoder-1-FeedForward-Dropout[0][\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-1-FeedForward-Norm (Lay (None, 384, 768)     1536        Encoder-1-FeedForward-Add[0][0]  \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-2-MultiHeadSelfAttentio (None, 384, 768)     2362368     Encoder-1-FeedForward-Norm[0][0] \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-2-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-2-MultiHeadSelfAttention[\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-2-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-1-FeedForward-Norm[0][0] \n",
            "                                                                 Encoder-2-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-2-MultiHeadSelfAttentio (None, 384, 768)     1536        Encoder-2-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-2-FeedForward (FeedForw (None, 384, 768)     4722432     Encoder-2-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-2-FeedForward-Dropout ( (None, 384, 768)     0           Encoder-2-FeedForward[0][0]      \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-2-FeedForward-Add (Add) (None, 384, 768)     0           Encoder-2-MultiHeadSelfAttention-\n",
            "                                                                 Encoder-2-FeedForward-Dropout[0][\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-2-FeedForward-Norm (Lay (None, 384, 768)     1536        Encoder-2-FeedForward-Add[0][0]  \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-3-MultiHeadSelfAttentio (None, 384, 768)     2362368     Encoder-2-FeedForward-Norm[0][0] \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-3-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-3-MultiHeadSelfAttention[\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-3-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-2-FeedForward-Norm[0][0] \n",
            "                                                                 Encoder-3-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-3-MultiHeadSelfAttentio (None, 384, 768)     1536        Encoder-3-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-3-FeedForward (FeedForw (None, 384, 768)     4722432     Encoder-3-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-3-FeedForward-Dropout ( (None, 384, 768)     0           Encoder-3-FeedForward[0][0]      \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-3-FeedForward-Add (Add) (None, 384, 768)     0           Encoder-3-MultiHeadSelfAttention-\n",
            "                                                                 Encoder-3-FeedForward-Dropout[0][\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-3-FeedForward-Norm (Lay (None, 384, 768)     1536        Encoder-3-FeedForward-Add[0][0]  \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-4-MultiHeadSelfAttentio (None, 384, 768)     2362368     Encoder-3-FeedForward-Norm[0][0] \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-4-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-4-MultiHeadSelfAttention[\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-4-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-3-FeedForward-Norm[0][0] \n",
            "                                                                 Encoder-4-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-4-MultiHeadSelfAttentio (None, 384, 768)     1536        Encoder-4-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-4-FeedForward (FeedForw (None, 384, 768)     4722432     Encoder-4-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-4-FeedForward-Dropout ( (None, 384, 768)     0           Encoder-4-FeedForward[0][0]      \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-4-FeedForward-Add (Add) (None, 384, 768)     0           Encoder-4-MultiHeadSelfAttention-\n",
            "                                                                 Encoder-4-FeedForward-Dropout[0][\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-4-FeedForward-Norm (Lay (None, 384, 768)     1536        Encoder-4-FeedForward-Add[0][0]  \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-5-MultiHeadSelfAttentio (None, 384, 768)     2362368     Encoder-4-FeedForward-Norm[0][0] \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-5-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-5-MultiHeadSelfAttention[\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-5-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-4-FeedForward-Norm[0][0] \n",
            "                                                                 Encoder-5-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-5-MultiHeadSelfAttentio (None, 384, 768)     1536        Encoder-5-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-5-FeedForward (FeedForw (None, 384, 768)     4722432     Encoder-5-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-5-FeedForward-Dropout ( (None, 384, 768)     0           Encoder-5-FeedForward[0][0]      \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-5-FeedForward-Add (Add) (None, 384, 768)     0           Encoder-5-MultiHeadSelfAttention-\n",
            "                                                                 Encoder-5-FeedForward-Dropout[0][\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-5-FeedForward-Norm (Lay (None, 384, 768)     1536        Encoder-5-FeedForward-Add[0][0]  \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-6-MultiHeadSelfAttentio (None, 384, 768)     2362368     Encoder-5-FeedForward-Norm[0][0] \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-6-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-6-MultiHeadSelfAttention[\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-6-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-5-FeedForward-Norm[0][0] \n",
            "                                                                 Encoder-6-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-6-MultiHeadSelfAttentio (None, 384, 768)     1536        Encoder-6-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-6-FeedForward (FeedForw (None, 384, 768)     4722432     Encoder-6-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-6-FeedForward-Dropout ( (None, 384, 768)     0           Encoder-6-FeedForward[0][0]      \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-6-FeedForward-Add (Add) (None, 384, 768)     0           Encoder-6-MultiHeadSelfAttention-\n",
            "                                                                 Encoder-6-FeedForward-Dropout[0][\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-6-FeedForward-Norm (Lay (None, 384, 768)     1536        Encoder-6-FeedForward-Add[0][0]  \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-7-MultiHeadSelfAttentio (None, 384, 768)     2362368     Encoder-6-FeedForward-Norm[0][0] \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-7-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-7-MultiHeadSelfAttention[\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-7-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-6-FeedForward-Norm[0][0] \n",
            "                                                                 Encoder-7-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-7-MultiHeadSelfAttentio (None, 384, 768)     1536        Encoder-7-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-7-FeedForward (FeedForw (None, 384, 768)     4722432     Encoder-7-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-7-FeedForward-Dropout ( (None, 384, 768)     0           Encoder-7-FeedForward[0][0]      \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-7-FeedForward-Add (Add) (None, 384, 768)     0           Encoder-7-MultiHeadSelfAttention-\n",
            "                                                                 Encoder-7-FeedForward-Dropout[0][\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-7-FeedForward-Norm (Lay (None, 384, 768)     1536        Encoder-7-FeedForward-Add[0][0]  \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-8-MultiHeadSelfAttentio (None, 384, 768)     2362368     Encoder-7-FeedForward-Norm[0][0] \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-8-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-8-MultiHeadSelfAttention[\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-8-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-7-FeedForward-Norm[0][0] \n",
            "                                                                 Encoder-8-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-8-MultiHeadSelfAttentio (None, 384, 768)     1536        Encoder-8-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-8-FeedForward (FeedForw (None, 384, 768)     4722432     Encoder-8-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-8-FeedForward-Dropout ( (None, 384, 768)     0           Encoder-8-FeedForward[0][0]      \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-8-FeedForward-Add (Add) (None, 384, 768)     0           Encoder-8-MultiHeadSelfAttention-\n",
            "                                                                 Encoder-8-FeedForward-Dropout[0][\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-8-FeedForward-Norm (Lay (None, 384, 768)     1536        Encoder-8-FeedForward-Add[0][0]  \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-9-MultiHeadSelfAttentio (None, 384, 768)     2362368     Encoder-8-FeedForward-Norm[0][0] \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-9-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-9-MultiHeadSelfAttention[\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-9-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-8-FeedForward-Norm[0][0] \n",
            "                                                                 Encoder-9-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-9-MultiHeadSelfAttentio (None, 384, 768)     1536        Encoder-9-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-9-FeedForward (FeedForw (None, 384, 768)     4722432     Encoder-9-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-9-FeedForward-Dropout ( (None, 384, 768)     0           Encoder-9-FeedForward[0][0]      \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-9-FeedForward-Add (Add) (None, 384, 768)     0           Encoder-9-MultiHeadSelfAttention-\n",
            "                                                                 Encoder-9-FeedForward-Dropout[0][\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-9-FeedForward-Norm (Lay (None, 384, 768)     1536        Encoder-9-FeedForward-Add[0][0]  \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-10-MultiHeadSelfAttenti (None, 384, 768)     2362368     Encoder-9-FeedForward-Norm[0][0] \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-10-MultiHeadSelfAttenti (None, 384, 768)     0           Encoder-10-MultiHeadSelfAttention\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-10-MultiHeadSelfAttenti (None, 384, 768)     0           Encoder-9-FeedForward-Norm[0][0] \n",
            "                                                                 Encoder-10-MultiHeadSelfAttention\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-10-MultiHeadSelfAttenti (None, 384, 768)     1536        Encoder-10-MultiHeadSelfAttention\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-10-FeedForward (FeedFor (None, 384, 768)     4722432     Encoder-10-MultiHeadSelfAttention\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-10-FeedForward-Dropout  (None, 384, 768)     0           Encoder-10-FeedForward[0][0]     \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-10-FeedForward-Add (Add (None, 384, 768)     0           Encoder-10-MultiHeadSelfAttention\n",
            "                                                                 Encoder-10-FeedForward-Dropout[0]\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-10-FeedForward-Norm (La (None, 384, 768)     1536        Encoder-10-FeedForward-Add[0][0] \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-11-MultiHeadSelfAttenti (None, 384, 768)     2362368     Encoder-10-FeedForward-Norm[0][0]\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-11-MultiHeadSelfAttenti (None, 384, 768)     0           Encoder-11-MultiHeadSelfAttention\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-11-MultiHeadSelfAttenti (None, 384, 768)     0           Encoder-10-FeedForward-Norm[0][0]\n",
            "                                                                 Encoder-11-MultiHeadSelfAttention\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-11-MultiHeadSelfAttenti (None, 384, 768)     1536        Encoder-11-MultiHeadSelfAttention\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-11-FeedForward (FeedFor (None, 384, 768)     4722432     Encoder-11-MultiHeadSelfAttention\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-11-FeedForward-Dropout  (None, 384, 768)     0           Encoder-11-FeedForward[0][0]     \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-11-FeedForward-Add (Add (None, 384, 768)     0           Encoder-11-MultiHeadSelfAttention\n",
            "                                                                 Encoder-11-FeedForward-Dropout[0]\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-11-FeedForward-Norm (La (None, 384, 768)     1536        Encoder-11-FeedForward-Add[0][0] \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-12-MultiHeadSelfAttenti (None, 384, 768)     2362368     Encoder-11-FeedForward-Norm[0][0]\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-12-MultiHeadSelfAttenti (None, 384, 768)     0           Encoder-12-MultiHeadSelfAttention\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-12-MultiHeadSelfAttenti (None, 384, 768)     0           Encoder-11-FeedForward-Norm[0][0]\n",
            "                                                                 Encoder-12-MultiHeadSelfAttention\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-12-MultiHeadSelfAttenti (None, 384, 768)     1536        Encoder-12-MultiHeadSelfAttention\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-12-FeedForward (FeedFor (None, 384, 768)     4722432     Encoder-12-MultiHeadSelfAttention\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-12-FeedForward-Dropout  (None, 384, 768)     0           Encoder-12-FeedForward[0][0]     \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-12-FeedForward-Add (Add (None, 384, 768)     0           Encoder-12-MultiHeadSelfAttention\n",
            "                                                                 Encoder-12-FeedForward-Dropout[0]\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-12-FeedForward-Norm (La (None, 384, 768)     1536        Encoder-12-FeedForward-Add[0][0] \n",
            "==================================================================================================\n",
            "Total params: 177,164,544\n",
            "Trainable params: 177,164,544\n",
            "Non-trainable params: 0\n",
            "__________________________________________________________________________________________________\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_ptok2n0MHoV",
        "colab_type": "text"
      },
      "source": [
        "모델의 구조를 확인합니다.  \n",
        "총 12층의 트랜스포머 계층이 있음을 확인할 수 있습니다.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "g1N8hZvNMMWL",
        "colab_type": "text"
      },
      "source": [
        "Transfer learning을 위해 Custom Layer를 작성해 줍니다.  \n",
        "NonMasking 함수를 지정해서, Bert 모형의 자체 Masking 된 텐서들을 풀어줘야 합니다.  \n",
        "이번 튜토리얼에서 만약 NonMasking 클래스를 만들지 않는다면, Bert 모형을 훈련할 수 없습니다."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "H99O5l4X8xyb",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class NonMasking(Layer):   \n",
        "    def __init__(self, **kwargs):   \n",
        "        self.supports_masking = True  \n",
        "        super(NonMasking, self).__init__(**kwargs)   \n",
        "  \n",
        "    def build(self, input_shape):   \n",
        "        input_shape = input_shape   \n",
        "  \n",
        "    def compute_mask(self, input, input_mask=None):   \n",
        "        return None   \n",
        "  \n",
        "    def call(self, x, mask=None):   \n",
        "        return x   \n",
        "  \n",
        "    def get_output_shape_for(self, input_shape):   \n",
        "        return input_shape  "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0BgB9WtubcgQ",
        "colab_type": "text"
      },
      "source": [
        "Keras Custom Layer 두 개를 생성합니다.  \n",
        "MyLayer_Start는 정답의 첫 번째 단어를 예측하는 것을 담당하고,  \n",
        "MyLaer_End는 정답의 마지막 단어를 예측하는 것을 담당합니다.  \n",
        "  \n",
        "사실 두 레이어는 동일한 역할을 합니다.  \n",
        "Bert 모형의 마지막 입력을 받아서, (batch_size, 384, 768)의 텐서 모양을 (batch_size, 384, 2)로 만들어주는 텐서를 곱해줍니다.  \n",
        "이 다음에 i) (batch_size, 384), ii) (batch_size, 384)의 아웃풋을 출력할 수 있게 하나의 텐서를 두개로 잘라줍니다.  \n",
        "  \n",
        "왜 끝이 384냐면, 384개의 위치를 예측하기 때문입니다. 단어의 위치의 최대 개수는 384개로 앞서 지정하였습니다.(SEQ_LEN)\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gOBsah1UgIPq",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class MyLayer_Start(Layer):\n",
        "\n",
        "    def __init__(self,seq_len, **kwargs):\n",
        "        \n",
        "        self.seq_len = seq_len\n",
        "        self.supports_masking = True\n",
        "        super(MyLayer_Start, self).__init__(**kwargs)\n",
        "\n",
        "    def build(self, input_shape):\n",
        "        \n",
        "        self.W = self.add_weight(name='kernel', \n",
        "                                 shape=(768,2),\n",
        "                                 initializer='uniform',\n",
        "                                 trainable=True)\n",
        "        super(MyLayer_Start, self).build(input_shape)\n",
        "\n",
        "    def call(self, x):\n",
        "        \n",
        "        x = K.reshape(x, shape=(-1,384,768))\n",
        "        x = K.dot(x, self.W)\n",
        "        \n",
        "        x = K.permute_dimensions(x, (2,0,1))\n",
        "\n",
        "        self.start_logits, self.end_logits = x[0], x[1]\n",
        "        \n",
        "        self.start_logits = K.softmax(self.start_logits, axis=-1)\n",
        "        \n",
        "        return self.start_logits\n",
        "\n",
        "    def compute_output_shape(self, input_shape):\n",
        "        return (input_shape[0], self.seq_len)\n",
        "\n",
        "\n",
        "class MyLayer_End(Layer):\n",
        "  def __init__(self,seq_len, **kwargs):\n",
        "        \n",
        "        self.seq_len = seq_len\n",
        "        self.supports_masking = True\n",
        "        super(MyLayer_End, self).__init__(**kwargs)\n",
        "  \n",
        "  def build(self, input_shape):\n",
        "        \n",
        "        self.W = self.add_weight(name='kernel', \n",
        "                                 shape=(768, 2),\n",
        "                                 initializer='uniform',\n",
        "                                 trainable=True)\n",
        "        super(MyLayer_End, self).build(input_shape)\n",
        "\n",
        "  \n",
        "  def call(self, x):\n",
        "\n",
        "        \n",
        "        x = K.reshape(x, shape=(-1,384,768))\n",
        "        x = K.dot(x, self.W)\n",
        "        x = K.permute_dimensions(x, (2,0,1))\n",
        "        \n",
        "        self.start_logits, self.end_logits = x[0], x[1]\n",
        "        \n",
        "        self.end_logits = K.softmax(self.end_logits, axis=-1)\n",
        "        \n",
        "        return self.end_logits\n",
        "\n",
        "  def compute_output_shape(self, input_shape):\n",
        "        return (input_shape[0], self.seq_len)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3uYTT6WYgiBG",
        "colab_type": "text"
      },
      "source": [
        "BERT 모델을 출력하는 함수를 지정합니다.  \n",
        "start_answer, end_answer를 예측하게 됩니다."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NcbOzpEb0wxU",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from keras.layers import merge, dot, concatenate\n",
        "from keras import metrics\n",
        "def get_bert_finetuning_model(model):\n",
        "  inputs = model.inputs[:2]\n",
        "  dense = model.output\n",
        "  x = NonMasking()(dense)\n",
        "  outputs_start = MyLayer_Start(384)(x)\n",
        "  outputs_end = MyLayer_End(384)(x)\n",
        "  bert_model = keras.models.Model(inputs, [outputs_start, outputs_end])\n",
        "  bert_model.compile(\n",
        "      optimizer=RAdam(learning_rate=LR, decay=0.001),\n",
        "      loss='categorical_crossentropy',\n",
        "      metrics=['accuracy'])\n",
        "  \n",
        "  return bert_model"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "H2LAk6d5dm4A",
        "colab_type": "text"
      },
      "source": [
        "**모델의 FLOW를 확인해 보도록 하겠습니다.**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "scWw12ysbPZ9",
        "colab_type": "code",
        "outputId": "71310051-3d63-4791-f162-82ccc634a283",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "from IPython.display import SVG\n",
        "from keras.utils import model_to_dot\n",
        "\n",
        "\n",
        "SVG(model_to_dot(get_bert_finetuning_model(model), dpi=65).create(prog='dot', format='svg'))\n"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<IPython.core.display.SVG object>"
            ],
            "image/svg+xml": "<svg height=\"6829pt\" viewBox=\"0.00 0.00 549.50 7564.00\" width=\"496pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n<g class=\"graph\" id=\"graph0\" transform=\"scale(.9028 .9028) rotate(0) translate(4 7560)\">\n<title>G</title>\n<polygon fill=\"#ffffff\" points=\"-4,4 -4,-7560 545.5,-7560 545.5,4 -4,4\" stroke=\"transparent\"/>\n<!-- 139879773576496 -->\n<g class=\"node\" id=\"node1\">\n<title>139879773576496</title>\n<polygon fill=\"none\" points=\"110.5,-7519.5 110.5,-7555.5 270.5,-7555.5 270.5,-7519.5 110.5,-7519.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"190.5\" y=\"-7533.8\">Input-Token: InputLayer</text>\n</g>\n<!-- 139879773576608 -->\n<g class=\"node\" id=\"node3\">\n<title>139879773576608</title>\n<polygon fill=\"none\" points=\"72,-7446.5 72,-7482.5 309,-7482.5 309,-7446.5 72,-7446.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"190.5\" y=\"-7460.8\">Embedding-Token: TokenEmbedding</text>\n</g>\n<!-- 139879773576496&#45;&gt;139879773576608 -->\n<g class=\"edge\" id=\"edge1\">\n<title>139879773576496-&gt;139879773576608</title>\n<path d=\"M190.5,-7519.4551C190.5,-7511.3828 190.5,-7501.6764 190.5,-7492.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"194.0001,-7492.5903 190.5,-7482.5904 187.0001,-7492.5904 194.0001,-7492.5903\" stroke=\"#000000\"/>\n</g>\n<!-- 139879773575432 -->\n<g class=\"node\" id=\"node2\">\n<title>139879773575432</title>\n<polygon fill=\"none\" points=\"348,-7519.5 348,-7555.5 521,-7555.5 521,-7519.5 348,-7519.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"434.5\" y=\"-7533.8\">Input-Segment: InputLayer</text>\n</g>\n<!-- 139879750808128 -->\n<g class=\"node\" id=\"node4\">\n<title>139879750808128</title>\n<polygon fill=\"none\" points=\"327.5,-7446.5 327.5,-7482.5 541.5,-7482.5 541.5,-7446.5 327.5,-7446.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"434.5\" y=\"-7460.8\">Embedding-Segment: Embedding</text>\n</g>\n<!-- 139879773575432&#45;&gt;139879750808128 -->\n<g class=\"edge\" id=\"edge2\">\n<title>139879773575432-&gt;139879750808128</title>\n<path d=\"M434.5,-7519.4551C434.5,-7511.3828 434.5,-7501.6764 434.5,-7492.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"438.0001,-7492.5903 434.5,-7482.5904 431.0001,-7492.5904 438.0001,-7492.5903\" stroke=\"#000000\"/>\n</g>\n<!-- 139879750808520 -->\n<g class=\"node\" id=\"node5\">\n<title>139879750808520</title>\n<polygon fill=\"none\" points=\"206.5,-7373.5 206.5,-7409.5 418.5,-7409.5 418.5,-7373.5 206.5,-7373.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"312.5\" y=\"-7387.8\">Embedding-Token-Segment: Add</text>\n</g>\n<!-- 139879773576608&#45;&gt;139879750808520 -->\n<g class=\"edge\" id=\"edge3\">\n<title>139879773576608-&gt;139879750808520</title>\n<path d=\"M220.6573,-7446.4551C236.6407,-7436.8912 256.4579,-7425.0334 273.5882,-7414.7833\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"275.4828,-7417.7284 282.2668,-7409.5904 271.8885,-7411.7216 275.4828,-7417.7284\" stroke=\"#000000\"/>\n</g>\n<!-- 139879750808128&#45;&gt;139879750808520 -->\n<g class=\"edge\" id=\"edge4\">\n<title>139879750808128-&gt;139879750808520</title>\n<path d=\"M404.3427,-7446.4551C388.3593,-7436.8912 368.5421,-7425.0334 351.4118,-7414.7833\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"353.1115,-7411.7216 342.7332,-7409.5904 349.5172,-7417.7284 353.1115,-7411.7216\" stroke=\"#000000\"/>\n</g>\n<!-- 139879750808016 -->\n<g class=\"node\" id=\"node6\">\n<title>139879750808016</title>\n<polygon fill=\"none\" points=\"183.5,-7300.5 183.5,-7336.5 441.5,-7336.5 441.5,-7300.5 183.5,-7300.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"312.5\" y=\"-7314.8\">Embedding-Position: PositionEmbedding</text>\n</g>\n<!-- 139879750808520&#45;&gt;139879750808016 -->\n<g class=\"edge\" id=\"edge5\">\n<title>139879750808520-&gt;139879750808016</title>\n<path d=\"M312.5,-7373.4551C312.5,-7365.3828 312.5,-7355.6764 312.5,-7346.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"316.0001,-7346.5903 312.5,-7336.5904 309.0001,-7346.5904 316.0001,-7346.5903\" stroke=\"#000000\"/>\n</g>\n<!-- 139879697053904 -->\n<g class=\"node\" id=\"node7\">\n<title>139879697053904</title>\n<polygon fill=\"none\" points=\"215.5,-7227.5 215.5,-7263.5 409.5,-7263.5 409.5,-7227.5 215.5,-7227.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"312.5\" y=\"-7241.8\">Embedding-Dropout: Dropout</text>\n</g>\n<!-- 139879750808016&#45;&gt;139879697053904 -->\n<g class=\"edge\" id=\"edge6\">\n<title>139879750808016-&gt;139879697053904</title>\n<path d=\"M312.5,-7300.4551C312.5,-7292.3828 312.5,-7282.6764 312.5,-7273.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"316.0001,-7273.5903 312.5,-7263.5904 309.0001,-7273.5904 316.0001,-7273.5903\" stroke=\"#000000\"/>\n</g>\n<!-- 139879767120304 -->\n<g class=\"node\" id=\"node8\">\n<title>139879767120304</title>\n<polygon fill=\"none\" points=\"190,-7154.5 190,-7190.5 435,-7190.5 435,-7154.5 190,-7154.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"312.5\" y=\"-7168.8\">Embedding-Norm: LayerNormalization</text>\n</g>\n<!-- 139879697053904&#45;&gt;139879767120304 -->\n<g class=\"edge\" id=\"edge7\">\n<title>139879697053904-&gt;139879767120304</title>\n<path d=\"M312.5,-7227.4551C312.5,-7219.3828 312.5,-7209.6764 312.5,-7200.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"316.0001,-7200.5903 312.5,-7190.5904 309.0001,-7200.5904 316.0001,-7200.5903\" stroke=\"#000000\"/>\n</g>\n<!-- 139879767179216 -->\n<g class=\"node\" id=\"node9\">\n<title>139879767179216</title>\n<polygon fill=\"none\" points=\"42.5,-7081.5 42.5,-7117.5 384.5,-7117.5 384.5,-7081.5 42.5,-7081.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"213.5\" y=\"-7095.8\">Encoder-1-MultiHeadSelfAttention: MultiHeadAttention</text>\n</g>\n<!-- 139879767120304&#45;&gt;139879767179216 -->\n<g class=\"edge\" id=\"edge8\">\n<title>139879767120304-&gt;139879767179216</title>\n<path d=\"M288.0281,-7154.4551C275.4149,-7145.1545 259.8597,-7133.6844 246.2286,-7123.6332\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"248.1592,-7120.7082 238.0335,-7117.5904 244.0049,-7126.3421 248.1592,-7120.7082\" stroke=\"#000000\"/>\n</g>\n<!-- 139879724252576 -->\n<g class=\"node\" id=\"node11\">\n<title>139879724252576</title>\n<polygon fill=\"none\" points=\"173,-6935.5 173,-6971.5 454,-6971.5 454,-6935.5 173,-6935.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"313.5\" y=\"-6949.8\">Encoder-1-MultiHeadSelfAttention-Add: Add</text>\n</g>\n<!-- 139879767120304&#45;&gt;139879724252576 -->\n<g class=\"edge\" id=\"edge10\">\n<title>139879767120304-&gt;139879724252576</title>\n<path d=\"M355.0685,-7154.3804C370.0065,-7145.7 385.1359,-7133.7088 393.5,-7118 401.1793,-7103.5774 400.9885,-7027.2923 390.5,-7008 383.7016,-6995.4953 372.6452,-6985.1998 361.0052,-6977.0548\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"362.7688,-6974.0266 352.4788,-6971.5031 358.9492,-6979.8927 362.7688,-6974.0266\" stroke=\"#000000\"/>\n</g>\n<!-- 139879767260632 -->\n<g class=\"node\" id=\"node10\">\n<title>139879767260632</title>\n<polygon fill=\"none\" points=\"53,-7008.5 53,-7044.5 382,-7044.5 382,-7008.5 53,-7008.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"217.5\" y=\"-7022.8\">Encoder-1-MultiHeadSelfAttention-Dropout: Dropout</text>\n</g>\n<!-- 139879767179216&#45;&gt;139879767260632 -->\n<g class=\"edge\" id=\"edge9\">\n<title>139879767179216-&gt;139879767260632</title>\n<path d=\"M214.4888,-7081.4551C214.9311,-7073.3828 215.4629,-7063.6764 215.9558,-7054.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"219.4563,-7054.7669 216.5087,-7044.5904 212.4668,-7054.3839 219.4563,-7054.7669\" stroke=\"#000000\"/>\n</g>\n<!-- 139879767260632&#45;&gt;139879724252576 -->\n<g class=\"edge\" id=\"edge11\">\n<title>139879767260632-&gt;139879724252576</title>\n<path d=\"M241.2303,-7008.4551C253.3459,-6999.2422 268.2608,-6987.9006 281.3886,-6977.918\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"283.8684,-6980.4293 289.7099,-6971.5904 279.6314,-6974.8573 283.8684,-6980.4293\" stroke=\"#000000\"/>\n</g>\n<!-- 139879733866112 -->\n<g class=\"node\" id=\"node12\">\n<title>139879733866112</title>\n<polygon fill=\"none\" points=\"123.5,-6862.5 123.5,-6898.5 503.5,-6898.5 503.5,-6862.5 123.5,-6862.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"313.5\" y=\"-6876.8\">Encoder-1-MultiHeadSelfAttention-Norm: LayerNormalization</text>\n</g>\n<!-- 139879724252576&#45;&gt;139879733866112 -->\n<g class=\"edge\" id=\"edge12\">\n<title>139879724252576-&gt;139879733866112</title>\n<path d=\"M313.5,-6935.4551C313.5,-6927.3828 313.5,-6917.6764 313.5,-6908.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"317.0001,-6908.5903 313.5,-6898.5904 310.0001,-6908.5904 317.0001,-6908.5903\" stroke=\"#000000\"/>\n</g>\n<!-- 139879733864880 -->\n<g class=\"node\" id=\"node13\">\n<title>139879733864880</title>\n<polygon fill=\"none\" points=\"118.5,-6789.5 118.5,-6825.5 360.5,-6825.5 360.5,-6789.5 118.5,-6789.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"239.5\" y=\"-6803.8\">Encoder-1-FeedForward: FeedForward</text>\n</g>\n<!-- 139879733866112&#45;&gt;139879733864880 -->\n<g class=\"edge\" id=\"edge13\">\n<title>139879733866112-&gt;139879733864880</title>\n<path d=\"M295.2079,-6862.4551C286.1356,-6853.5054 275.0271,-6842.547 265.1232,-6832.7769\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"267.4152,-6830.1215 257.8382,-6825.5904 262.4992,-6835.1049 267.4152,-6830.1215\" stroke=\"#000000\"/>\n</g>\n<!-- 139879732760416 -->\n<g class=\"node\" id=\"node15\">\n<title>139879732760416</title>\n<polygon fill=\"none\" points=\"203.5,-6643.5 203.5,-6679.5 423.5,-6679.5 423.5,-6643.5 203.5,-6643.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"313.5\" y=\"-6657.8\">Encoder-1-FeedForward-Add: Add</text>\n</g>\n<!-- 139879733866112&#45;&gt;139879732760416 -->\n<g class=\"edge\" id=\"edge15\">\n<title>139879733866112-&gt;139879732760416</title>\n<path d=\"M339.1147,-6862.4444C350.4534,-6852.9752 362.7322,-6840.3575 369.5,-6826 390.3764,-6781.7122 395.6809,-6760.609 375.5,-6716 370.2852,-6704.4728 361.2939,-6694.4147 351.751,-6686.1833\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"353.6666,-6683.2331 343.676,-6679.7064 349.2868,-6688.6936 353.6666,-6683.2331\" stroke=\"#000000\"/>\n</g>\n<!-- 139879733817128 -->\n<g class=\"node\" id=\"node14\">\n<title>139879733817128</title>\n<polygon fill=\"none\" points=\"99,-6716.5 99,-6752.5 366,-6752.5 366,-6716.5 99,-6716.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"232.5\" y=\"-6730.8\">Encoder-1-FeedForward-Dropout: Dropout</text>\n</g>\n<!-- 139879733864880&#45;&gt;139879733817128 -->\n<g class=\"edge\" id=\"edge14\">\n<title>139879733864880-&gt;139879733817128</title>\n<path d=\"M237.7697,-6789.4551C236.9956,-6781.3828 236.0649,-6771.6764 235.2024,-6762.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"238.6733,-6762.2106 234.2347,-6752.5904 231.7053,-6762.8788 238.6733,-6762.2106\" stroke=\"#000000\"/>\n</g>\n<!-- 139879733817128&#45;&gt;139879732760416 -->\n<g class=\"edge\" id=\"edge16\">\n<title>139879733817128-&gt;139879732760416</title>\n<path d=\"M252.5225,-6716.4551C262.5502,-6707.4177 274.8509,-6696.3319 285.7714,-6686.4899\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"288.3419,-6688.885 293.4271,-6679.5904 283.6556,-6683.6852 288.3419,-6688.885\" stroke=\"#000000\"/>\n</g>\n<!-- 139879732944176 -->\n<g class=\"node\" id=\"node16\">\n<title>139879732944176</title>\n<polygon fill=\"none\" points=\"154.5,-6570.5 154.5,-6606.5 472.5,-6606.5 472.5,-6570.5 154.5,-6570.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"313.5\" y=\"-6584.8\">Encoder-1-FeedForward-Norm: LayerNormalization</text>\n</g>\n<!-- 139879732760416&#45;&gt;139879732944176 -->\n<g class=\"edge\" id=\"edge17\">\n<title>139879732760416-&gt;139879732944176</title>\n<path d=\"M313.5,-6643.4551C313.5,-6635.3828 313.5,-6625.6764 313.5,-6616.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"317.0001,-6616.5903 313.5,-6606.5904 310.0001,-6616.5904 317.0001,-6616.5903\" stroke=\"#000000\"/>\n</g>\n<!-- 139879730978544 -->\n<g class=\"node\" id=\"node17\">\n<title>139879730978544</title>\n<polygon fill=\"none\" points=\"43.5,-6497.5 43.5,-6533.5 385.5,-6533.5 385.5,-6497.5 43.5,-6497.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"214.5\" y=\"-6511.8\">Encoder-2-MultiHeadSelfAttention: MultiHeadAttention</text>\n</g>\n<!-- 139879732944176&#45;&gt;139879730978544 -->\n<g class=\"edge\" id=\"edge18\">\n<title>139879732944176-&gt;139879730978544</title>\n<path d=\"M289.0281,-6570.4551C276.4149,-6561.1545 260.8597,-6549.6844 247.2286,-6539.6332\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"249.1592,-6536.7082 239.0335,-6533.5904 245.0049,-6542.3421 249.1592,-6536.7082\" stroke=\"#000000\"/>\n</g>\n<!-- 139879730998520 -->\n<g class=\"node\" id=\"node19\">\n<title>139879730998520</title>\n<polygon fill=\"none\" points=\"175,-6351.5 175,-6387.5 456,-6387.5 456,-6351.5 175,-6351.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"315.5\" y=\"-6365.8\">Encoder-2-MultiHeadSelfAttention-Add: Add</text>\n</g>\n<!-- 139879732944176&#45;&gt;139879730998520 -->\n<g class=\"edge\" id=\"edge20\">\n<title>139879732944176-&gt;139879730998520</title>\n<path d=\"M356.0685,-6570.3804C371.0065,-6561.7 386.1359,-6549.7088 394.5,-6534 402.5894,-6518.8071 400.2511,-6441.9102 390.5,-6424 383.7494,-6411.601 372.8204,-6401.3006 361.3687,-6393.1162\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"363.2521,-6390.1654 352.99,-6387.531 359.3694,-6395.9899 363.2521,-6390.1654\" stroke=\"#000000\"/>\n</g>\n<!-- 139879731020856 -->\n<g class=\"node\" id=\"node18\">\n<title>139879731020856</title>\n<polygon fill=\"none\" points=\"53,-6424.5 53,-6460.5 382,-6460.5 382,-6424.5 53,-6424.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"217.5\" y=\"-6438.8\">Encoder-2-MultiHeadSelfAttention-Dropout: Dropout</text>\n</g>\n<!-- 139879730978544&#45;&gt;139879731020856 -->\n<g class=\"edge\" id=\"edge19\">\n<title>139879730978544-&gt;139879731020856</title>\n<path d=\"M215.2416,-6497.4551C215.5733,-6489.3828 215.9722,-6479.6764 216.3418,-6470.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"219.8429,-6470.7257 216.7566,-6460.5904 212.8488,-6470.4382 219.8429,-6470.7257\" stroke=\"#000000\"/>\n</g>\n<!-- 139879731020856&#45;&gt;139879730998520 -->\n<g class=\"edge\" id=\"edge21\">\n<title>139879731020856-&gt;139879730998520</title>\n<path d=\"M241.7247,-6424.4551C254.2105,-6415.1545 269.6086,-6403.6844 283.102,-6393.6332\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"285.2855,-6396.371 291.2143,-6387.5904 281.1039,-6390.7573 285.2855,-6396.371\" stroke=\"#000000\"/>\n</g>\n<!-- 139879718887320 -->\n<g class=\"node\" id=\"node20\">\n<title>139879718887320</title>\n<polygon fill=\"none\" points=\"125.5,-6278.5 125.5,-6314.5 505.5,-6314.5 505.5,-6278.5 125.5,-6278.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"315.5\" y=\"-6292.8\">Encoder-2-MultiHeadSelfAttention-Norm: LayerNormalization</text>\n</g>\n<!-- 139879730998520&#45;&gt;139879718887320 -->\n<g class=\"edge\" id=\"edge22\">\n<title>139879730998520-&gt;139879718887320</title>\n<path d=\"M315.5,-6351.4551C315.5,-6343.3828 315.5,-6333.6764 315.5,-6324.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"319.0001,-6324.5903 315.5,-6314.5904 312.0001,-6324.5904 319.0001,-6324.5903\" stroke=\"#000000\"/>\n</g>\n<!-- 139879718956560 -->\n<g class=\"node\" id=\"node21\">\n<title>139879718956560</title>\n<polygon fill=\"none\" points=\"113.5,-6205.5 113.5,-6241.5 355.5,-6241.5 355.5,-6205.5 113.5,-6205.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"234.5\" y=\"-6219.8\">Encoder-2-FeedForward: FeedForward</text>\n</g>\n<!-- 139879718887320&#45;&gt;139879718956560 -->\n<g class=\"edge\" id=\"edge23\">\n<title>139879718887320-&gt;139879718956560</title>\n<path d=\"M295.4775,-6278.4551C285.4498,-6269.4177 273.1491,-6258.3319 262.2286,-6248.4899\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"264.3444,-6245.6852 254.5729,-6241.5904 259.6581,-6250.885 264.3444,-6245.6852\" stroke=\"#000000\"/>\n</g>\n<!-- 139879719038144 -->\n<g class=\"node\" id=\"node23\">\n<title>139879719038144</title>\n<polygon fill=\"none\" points=\"199.5,-6059.5 199.5,-6095.5 419.5,-6095.5 419.5,-6059.5 199.5,-6059.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"309.5\" y=\"-6073.8\">Encoder-2-FeedForward-Add: Add</text>\n</g>\n<!-- 139879718887320&#45;&gt;139879719038144 -->\n<g class=\"edge\" id=\"edge25\">\n<title>139879718887320-&gt;139879719038144</title>\n<path d=\"M337.3471,-6278.291C347.3936,-6268.6134 358.4097,-6255.8624 364.5,-6242 384.2044,-6197.1498 391.6917,-6176.6329 371.5,-6132 366.2852,-6120.4728 357.2939,-6110.4147 347.751,-6102.1833\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"349.6666,-6099.2331 339.676,-6095.7064 345.2868,-6104.6936 349.6666,-6099.2331\" stroke=\"#000000\"/>\n</g>\n<!-- 139879719017776 -->\n<g class=\"node\" id=\"node22\">\n<title>139879719017776</title>\n<polygon fill=\"none\" points=\"95,-6132.5 95,-6168.5 362,-6168.5 362,-6132.5 95,-6132.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"228.5\" y=\"-6146.8\">Encoder-2-FeedForward-Dropout: Dropout</text>\n</g>\n<!-- 139879718956560&#45;&gt;139879719017776 -->\n<g class=\"edge\" id=\"edge24\">\n<title>139879718956560-&gt;139879719017776</title>\n<path d=\"M233.0169,-6205.4551C232.3534,-6197.3828 231.5556,-6187.6764 230.8163,-6178.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"234.2944,-6178.27 229.9869,-6168.5904 227.3179,-6178.8435 234.2944,-6178.27\" stroke=\"#000000\"/>\n</g>\n<!-- 139879719017776&#45;&gt;139879719038144 -->\n<g class=\"edge\" id=\"edge26\">\n<title>139879719017776-&gt;139879719038144</title>\n<path d=\"M248.5225,-6132.4551C258.5502,-6123.4177 270.8509,-6112.3319 281.7714,-6102.4899\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"284.3419,-6104.885 289.4271,-6095.5904 279.6556,-6099.6852 284.3419,-6104.885\" stroke=\"#000000\"/>\n</g>\n<!-- 139879717448952 -->\n<g class=\"node\" id=\"node24\">\n<title>139879717448952</title>\n<polygon fill=\"none\" points=\"150.5,-5986.5 150.5,-6022.5 468.5,-6022.5 468.5,-5986.5 150.5,-5986.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"309.5\" y=\"-6000.8\">Encoder-2-FeedForward-Norm: LayerNormalization</text>\n</g>\n<!-- 139879719038144&#45;&gt;139879717448952 -->\n<g class=\"edge\" id=\"edge27\">\n<title>139879719038144-&gt;139879717448952</title>\n<path d=\"M309.5,-6059.4551C309.5,-6051.3828 309.5,-6041.6764 309.5,-6032.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"313.0001,-6032.5903 309.5,-6022.5904 306.0001,-6032.5904 313.0001,-6032.5903\" stroke=\"#000000\"/>\n</g>\n<!-- 139879717449288 -->\n<g class=\"node\" id=\"node25\">\n<title>139879717449288</title>\n<polygon fill=\"none\" points=\"39.5,-5913.5 39.5,-5949.5 381.5,-5949.5 381.5,-5913.5 39.5,-5913.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"210.5\" y=\"-5927.8\">Encoder-3-MultiHeadSelfAttention: MultiHeadAttention</text>\n</g>\n<!-- 139879717448952&#45;&gt;139879717449288 -->\n<g class=\"edge\" id=\"edge28\">\n<title>139879717448952-&gt;139879717449288</title>\n<path d=\"M285.0281,-5986.4551C272.4149,-5977.1545 256.8597,-5965.6844 243.2286,-5955.6332\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"245.1592,-5952.7082 235.0335,-5949.5904 241.0049,-5958.3421 245.1592,-5952.7082\" stroke=\"#000000\"/>\n</g>\n<!-- 139879716600464 -->\n<g class=\"node\" id=\"node27\">\n<title>139879716600464</title>\n<polygon fill=\"none\" points=\"171,-5767.5 171,-5803.5 452,-5803.5 452,-5767.5 171,-5767.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"311.5\" y=\"-5781.8\">Encoder-3-MultiHeadSelfAttention-Add: Add</text>\n</g>\n<!-- 139879717448952&#45;&gt;139879716600464 -->\n<g class=\"edge\" id=\"edge30\">\n<title>139879717448952-&gt;139879716600464</title>\n<path d=\"M352.0685,-5986.3804C367.0065,-5977.7 382.1359,-5965.7088 390.5,-5950 398.5894,-5934.8071 396.2511,-5857.9102 386.5,-5840 379.7494,-5827.601 368.8204,-5817.3006 357.3687,-5809.1162\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"359.2521,-5806.1654 348.99,-5803.531 355.3694,-5811.9899 359.2521,-5806.1654\" stroke=\"#000000\"/>\n</g>\n<!-- 139879716513832 -->\n<g class=\"node\" id=\"node26\">\n<title>139879716513832</title>\n<polygon fill=\"none\" points=\"49,-5840.5 49,-5876.5 378,-5876.5 378,-5840.5 49,-5840.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"213.5\" y=\"-5854.8\">Encoder-3-MultiHeadSelfAttention-Dropout: Dropout</text>\n</g>\n<!-- 139879717449288&#45;&gt;139879716513832 -->\n<g class=\"edge\" id=\"edge29\">\n<title>139879717449288-&gt;139879716513832</title>\n<path d=\"M211.2416,-5913.4551C211.5733,-5905.3828 211.9722,-5895.6764 212.3418,-5886.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"215.8429,-5886.7257 212.7566,-5876.5904 208.8488,-5886.4382 215.8429,-5886.7257\" stroke=\"#000000\"/>\n</g>\n<!-- 139879716513832&#45;&gt;139879716600464 -->\n<g class=\"edge\" id=\"edge31\">\n<title>139879716513832-&gt;139879716600464</title>\n<path d=\"M237.7247,-5840.4551C250.2105,-5831.1545 265.6086,-5819.6844 279.102,-5809.6332\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"281.2855,-5812.371 287.2143,-5803.5904 277.1039,-5806.7573 281.2855,-5812.371\" stroke=\"#000000\"/>\n</g>\n<!-- 139879701032408 -->\n<g class=\"node\" id=\"node28\">\n<title>139879701032408</title>\n<polygon fill=\"none\" points=\"121.5,-5694.5 121.5,-5730.5 501.5,-5730.5 501.5,-5694.5 121.5,-5694.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"311.5\" y=\"-5708.8\">Encoder-3-MultiHeadSelfAttention-Norm: LayerNormalization</text>\n</g>\n<!-- 139879716600464&#45;&gt;139879701032408 -->\n<g class=\"edge\" id=\"edge32\">\n<title>139879716600464-&gt;139879701032408</title>\n<path d=\"M311.5,-5767.4551C311.5,-5759.3828 311.5,-5749.6764 311.5,-5740.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"315.0001,-5740.5903 311.5,-5730.5904 308.0001,-5740.5904 315.0001,-5740.5903\" stroke=\"#000000\"/>\n</g>\n<!-- 139879701032072 -->\n<g class=\"node\" id=\"node29\">\n<title>139879701032072</title>\n<polygon fill=\"none\" points=\"109.5,-5621.5 109.5,-5657.5 351.5,-5657.5 351.5,-5621.5 109.5,-5621.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"230.5\" y=\"-5635.8\">Encoder-3-FeedForward: FeedForward</text>\n</g>\n<!-- 139879701032408&#45;&gt;139879701032072 -->\n<g class=\"edge\" id=\"edge33\">\n<title>139879701032408-&gt;139879701032072</title>\n<path d=\"M291.4775,-5694.4551C281.4498,-5685.4177 269.1491,-5674.3319 258.2286,-5664.4899\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"260.3444,-5661.6852 250.5729,-5657.5904 255.6581,-5666.885 260.3444,-5661.6852\" stroke=\"#000000\"/>\n</g>\n<!-- 139879701174872 -->\n<g class=\"node\" id=\"node31\">\n<title>139879701174872</title>\n<polygon fill=\"none\" points=\"195.5,-5475.5 195.5,-5511.5 415.5,-5511.5 415.5,-5475.5 195.5,-5475.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"305.5\" y=\"-5489.8\">Encoder-3-FeedForward-Add: Add</text>\n</g>\n<!-- 139879701032408&#45;&gt;139879701174872 -->\n<g class=\"edge\" id=\"edge35\">\n<title>139879701032408-&gt;139879701174872</title>\n<path d=\"M333.3471,-5694.291C343.3936,-5684.6134 354.4097,-5671.8624 360.5,-5658 380.2044,-5613.1498 387.6917,-5592.6329 367.5,-5548 362.2852,-5536.4728 353.2939,-5526.4147 343.751,-5518.1833\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"345.6666,-5515.2331 335.676,-5511.7064 341.2868,-5520.6936 345.6666,-5515.2331\" stroke=\"#000000\"/>\n</g>\n<!-- 139879701096432 -->\n<g class=\"node\" id=\"node30\">\n<title>139879701096432</title>\n<polygon fill=\"none\" points=\"91,-5548.5 91,-5584.5 358,-5584.5 358,-5548.5 91,-5548.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"224.5\" y=\"-5562.8\">Encoder-3-FeedForward-Dropout: Dropout</text>\n</g>\n<!-- 139879701032072&#45;&gt;139879701096432 -->\n<g class=\"edge\" id=\"edge34\">\n<title>139879701032072-&gt;139879701096432</title>\n<path d=\"M229.0169,-5621.4551C228.3534,-5613.3828 227.5556,-5603.6764 226.8163,-5594.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"230.2944,-5594.27 225.9869,-5584.5904 223.3179,-5594.8435 230.2944,-5594.27\" stroke=\"#000000\"/>\n</g>\n<!-- 139879701096432&#45;&gt;139879701174872 -->\n<g class=\"edge\" id=\"edge36\">\n<title>139879701096432-&gt;139879701174872</title>\n<path d=\"M244.5225,-5548.4551C254.5502,-5539.4177 266.8509,-5528.3319 277.7714,-5518.4899\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"280.3419,-5520.885 285.4271,-5511.5904 275.6556,-5515.6852 280.3419,-5520.885\" stroke=\"#000000\"/>\n</g>\n<!-- 139879699299744 -->\n<g class=\"node\" id=\"node32\">\n<title>139879699299744</title>\n<polygon fill=\"none\" points=\"146.5,-5402.5 146.5,-5438.5 464.5,-5438.5 464.5,-5402.5 146.5,-5402.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"305.5\" y=\"-5416.8\">Encoder-3-FeedForward-Norm: LayerNormalization</text>\n</g>\n<!-- 139879701174872&#45;&gt;139879699299744 -->\n<g class=\"edge\" id=\"edge37\">\n<title>139879701174872-&gt;139879699299744</title>\n<path d=\"M305.5,-5475.4551C305.5,-5467.3828 305.5,-5457.6764 305.5,-5448.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"309.0001,-5448.5903 305.5,-5438.5904 302.0001,-5448.5904 309.0001,-5448.5903\" stroke=\"#000000\"/>\n</g>\n<!-- 139879699298960 -->\n<g class=\"node\" id=\"node33\">\n<title>139879699298960</title>\n<polygon fill=\"none\" points=\"35.5,-5329.5 35.5,-5365.5 377.5,-5365.5 377.5,-5329.5 35.5,-5329.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"206.5\" y=\"-5343.8\">Encoder-4-MultiHeadSelfAttention: MultiHeadAttention</text>\n</g>\n<!-- 139879699299744&#45;&gt;139879699298960 -->\n<g class=\"edge\" id=\"edge38\">\n<title>139879699299744-&gt;139879699298960</title>\n<path d=\"M281.0281,-5402.4551C268.4149,-5393.1545 252.8597,-5381.6844 239.2286,-5371.6332\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"241.1592,-5368.7082 231.0335,-5365.5904 237.0049,-5374.3421 241.1592,-5368.7082\" stroke=\"#000000\"/>\n</g>\n<!-- 139879766087720 -->\n<g class=\"node\" id=\"node35\">\n<title>139879766087720</title>\n<polygon fill=\"none\" points=\"167,-5183.5 167,-5219.5 448,-5219.5 448,-5183.5 167,-5183.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"307.5\" y=\"-5197.8\">Encoder-4-MultiHeadSelfAttention-Add: Add</text>\n</g>\n<!-- 139879699299744&#45;&gt;139879766087720 -->\n<g class=\"edge\" id=\"edge40\">\n<title>139879699299744-&gt;139879766087720</title>\n<path d=\"M348.0685,-5402.3804C363.0065,-5393.7 378.1359,-5381.7088 386.5,-5366 394.5894,-5350.8071 392.2511,-5273.9102 382.5,-5256 375.7494,-5243.601 364.8204,-5233.3006 353.3687,-5225.1162\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"355.2521,-5222.1654 344.99,-5219.531 351.3694,-5227.9899 355.2521,-5222.1654\" stroke=\"#000000\"/>\n</g>\n<!-- 139879699401584 -->\n<g class=\"node\" id=\"node34\">\n<title>139879699401584</title>\n<polygon fill=\"none\" points=\"45,-5256.5 45,-5292.5 374,-5292.5 374,-5256.5 45,-5256.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"209.5\" y=\"-5270.8\">Encoder-4-MultiHeadSelfAttention-Dropout: Dropout</text>\n</g>\n<!-- 139879699298960&#45;&gt;139879699401584 -->\n<g class=\"edge\" id=\"edge39\">\n<title>139879699298960-&gt;139879699401584</title>\n<path d=\"M207.2416,-5329.4551C207.5733,-5321.3828 207.9722,-5311.6764 208.3418,-5302.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"211.8429,-5302.7257 208.7566,-5292.5904 204.8488,-5302.4382 211.8429,-5302.7257\" stroke=\"#000000\"/>\n</g>\n<!-- 139879699401584&#45;&gt;139879766087720 -->\n<g class=\"edge\" id=\"edge41\">\n<title>139879699401584-&gt;139879766087720</title>\n<path d=\"M233.7247,-5256.4551C246.2105,-5247.1545 261.6086,-5235.6844 275.102,-5225.6332\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"277.2855,-5228.371 283.2143,-5219.5904 273.1039,-5222.7573 277.2855,-5228.371\" stroke=\"#000000\"/>\n</g>\n<!-- 139879778282968 -->\n<g class=\"node\" id=\"node36\">\n<title>139879778282968</title>\n<polygon fill=\"none\" points=\"117.5,-5110.5 117.5,-5146.5 497.5,-5146.5 497.5,-5110.5 117.5,-5110.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"307.5\" y=\"-5124.8\">Encoder-4-MultiHeadSelfAttention-Norm: LayerNormalization</text>\n</g>\n<!-- 139879766087720&#45;&gt;139879778282968 -->\n<g class=\"edge\" id=\"edge42\">\n<title>139879766087720-&gt;139879778282968</title>\n<path d=\"M307.5,-5183.4551C307.5,-5175.3828 307.5,-5165.6764 307.5,-5156.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"311.0001,-5156.5903 307.5,-5146.5904 304.0001,-5156.5904 311.0001,-5156.5903\" stroke=\"#000000\"/>\n</g>\n<!-- 139879778294584 -->\n<g class=\"node\" id=\"node37\">\n<title>139879778294584</title>\n<polygon fill=\"none\" points=\"105.5,-5037.5 105.5,-5073.5 347.5,-5073.5 347.5,-5037.5 105.5,-5037.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"226.5\" y=\"-5051.8\">Encoder-4-FeedForward: FeedForward</text>\n</g>\n<!-- 139879778282968&#45;&gt;139879778294584 -->\n<g class=\"edge\" id=\"edge43\">\n<title>139879778282968-&gt;139879778294584</title>\n<path d=\"M287.4775,-5110.4551C277.4498,-5101.4177 265.1491,-5090.3319 254.2286,-5080.4899\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"256.3444,-5077.6852 246.5729,-5073.5904 251.6581,-5082.885 256.3444,-5077.6852\" stroke=\"#000000\"/>\n</g>\n<!-- 139879776364528 -->\n<g class=\"node\" id=\"node39\">\n<title>139879776364528</title>\n<polygon fill=\"none\" points=\"191.5,-4891.5 191.5,-4927.5 411.5,-4927.5 411.5,-4891.5 191.5,-4891.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"301.5\" y=\"-4905.8\">Encoder-4-FeedForward-Add: Add</text>\n</g>\n<!-- 139879778282968&#45;&gt;139879776364528 -->\n<g class=\"edge\" id=\"edge45\">\n<title>139879778282968-&gt;139879776364528</title>\n<path d=\"M329.3471,-5110.291C339.3936,-5100.6134 350.4097,-5087.8624 356.5,-5074 376.2044,-5029.1498 383.6917,-5008.6329 363.5,-4964 358.2852,-4952.4728 349.2939,-4942.4147 339.751,-4934.1833\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"341.6666,-4931.2331 331.676,-4927.7064 337.2868,-4936.6936 341.6666,-4931.2331\" stroke=\"#000000\"/>\n</g>\n<!-- 139879776271104 -->\n<g class=\"node\" id=\"node38\">\n<title>139879776271104</title>\n<polygon fill=\"none\" points=\"87,-4964.5 87,-5000.5 354,-5000.5 354,-4964.5 87,-4964.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"220.5\" y=\"-4978.8\">Encoder-4-FeedForward-Dropout: Dropout</text>\n</g>\n<!-- 139879778294584&#45;&gt;139879776271104 -->\n<g class=\"edge\" id=\"edge44\">\n<title>139879778294584-&gt;139879776271104</title>\n<path d=\"M225.0169,-5037.4551C224.3534,-5029.3828 223.5556,-5019.6764 222.8163,-5010.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"226.2944,-5010.27 221.9869,-5000.5904 219.3179,-5010.8435 226.2944,-5010.27\" stroke=\"#000000\"/>\n</g>\n<!-- 139879776271104&#45;&gt;139879776364528 -->\n<g class=\"edge\" id=\"edge46\">\n<title>139879776271104-&gt;139879776364528</title>\n<path d=\"M240.5225,-4964.4551C250.5502,-4955.4177 262.8509,-4944.3319 273.7714,-4934.4899\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"276.3419,-4936.885 281.4271,-4927.5904 271.6556,-4931.6852 276.3419,-4936.885\" stroke=\"#000000\"/>\n</g>\n<!-- 139879770577720 -->\n<g class=\"node\" id=\"node40\">\n<title>139879770577720</title>\n<polygon fill=\"none\" points=\"142.5,-4818.5 142.5,-4854.5 460.5,-4854.5 460.5,-4818.5 142.5,-4818.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"301.5\" y=\"-4832.8\">Encoder-4-FeedForward-Norm: LayerNormalization</text>\n</g>\n<!-- 139879776364528&#45;&gt;139879770577720 -->\n<g class=\"edge\" id=\"edge47\">\n<title>139879776364528-&gt;139879770577720</title>\n<path d=\"M301.5,-4891.4551C301.5,-4883.3828 301.5,-4873.6764 301.5,-4864.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"305.0001,-4864.5903 301.5,-4854.5904 298.0001,-4864.5904 305.0001,-4864.5903\" stroke=\"#000000\"/>\n</g>\n<!-- 139879770576936 -->\n<g class=\"node\" id=\"node41\">\n<title>139879770576936</title>\n<polygon fill=\"none\" points=\"31.5,-4745.5 31.5,-4781.5 373.5,-4781.5 373.5,-4745.5 31.5,-4745.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"202.5\" y=\"-4759.8\">Encoder-5-MultiHeadSelfAttention: MultiHeadAttention</text>\n</g>\n<!-- 139879770577720&#45;&gt;139879770576936 -->\n<g class=\"edge\" id=\"edge48\">\n<title>139879770577720-&gt;139879770576936</title>\n<path d=\"M277.0281,-4818.4551C264.4149,-4809.1545 248.8597,-4797.6844 235.2286,-4787.6332\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"237.1592,-4784.7082 227.0335,-4781.5904 233.0049,-4790.3421 237.1592,-4784.7082\" stroke=\"#000000\"/>\n</g>\n<!-- 139879768917440 -->\n<g class=\"node\" id=\"node43\">\n<title>139879768917440</title>\n<polygon fill=\"none\" points=\"163,-4599.5 163,-4635.5 444,-4635.5 444,-4599.5 163,-4599.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"303.5\" y=\"-4613.8\">Encoder-5-MultiHeadSelfAttention-Add: Add</text>\n</g>\n<!-- 139879770577720&#45;&gt;139879768917440 -->\n<g class=\"edge\" id=\"edge50\">\n<title>139879770577720-&gt;139879768917440</title>\n<path d=\"M344.0685,-4818.3804C359.0065,-4809.7 374.1359,-4797.7088 382.5,-4782 390.5894,-4766.8071 388.2511,-4689.9102 378.5,-4672 371.7494,-4659.601 360.8204,-4649.3006 349.3687,-4641.1162\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"351.2521,-4638.1654 340.99,-4635.531 347.3694,-4643.9899 351.2521,-4638.1654\" stroke=\"#000000\"/>\n</g>\n<!-- 139879770644264 -->\n<g class=\"node\" id=\"node42\">\n<title>139879770644264</title>\n<polygon fill=\"none\" points=\"41,-4672.5 41,-4708.5 370,-4708.5 370,-4672.5 41,-4672.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"205.5\" y=\"-4686.8\">Encoder-5-MultiHeadSelfAttention-Dropout: Dropout</text>\n</g>\n<!-- 139879770576936&#45;&gt;139879770644264 -->\n<g class=\"edge\" id=\"edge49\">\n<title>139879770576936-&gt;139879770644264</title>\n<path d=\"M203.2416,-4745.4551C203.5733,-4737.3828 203.9722,-4727.6764 204.3418,-4718.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"207.8429,-4718.7257 204.7566,-4708.5904 200.8488,-4718.4382 207.8429,-4718.7257\" stroke=\"#000000\"/>\n</g>\n<!-- 139879770644264&#45;&gt;139879768917440 -->\n<g class=\"edge\" id=\"edge51\">\n<title>139879770644264-&gt;139879768917440</title>\n<path d=\"M229.7247,-4672.4551C242.2105,-4663.1545 257.6086,-4651.6844 271.102,-4641.6332\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"273.2855,-4644.371 279.2143,-4635.5904 269.1039,-4638.7573 273.2855,-4644.371\" stroke=\"#000000\"/>\n</g>\n<!-- 139879765265264 -->\n<g class=\"node\" id=\"node44\">\n<title>139879765265264</title>\n<polygon fill=\"none\" points=\"113.5,-4526.5 113.5,-4562.5 493.5,-4562.5 493.5,-4526.5 113.5,-4526.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"303.5\" y=\"-4540.8\">Encoder-5-MultiHeadSelfAttention-Norm: LayerNormalization</text>\n</g>\n<!-- 139879768917440&#45;&gt;139879765265264 -->\n<g class=\"edge\" id=\"edge52\">\n<title>139879768917440-&gt;139879765265264</title>\n<path d=\"M303.5,-4599.4551C303.5,-4591.3828 303.5,-4581.6764 303.5,-4572.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"307.0001,-4572.5903 303.5,-4562.5904 300.0001,-4572.5904 307.0001,-4572.5903\" stroke=\"#000000\"/>\n</g>\n<!-- 139879765266160 -->\n<g class=\"node\" id=\"node45\">\n<title>139879765266160</title>\n<polygon fill=\"none\" points=\"101.5,-4453.5 101.5,-4489.5 343.5,-4489.5 343.5,-4453.5 101.5,-4453.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"222.5\" y=\"-4467.8\">Encoder-5-FeedForward: FeedForward</text>\n</g>\n<!-- 139879765265264&#45;&gt;139879765266160 -->\n<g class=\"edge\" id=\"edge53\">\n<title>139879765265264-&gt;139879765266160</title>\n<path d=\"M283.4775,-4526.4551C273.4498,-4517.4177 261.1491,-4506.3319 250.2286,-4496.4899\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"252.3444,-4493.6852 242.5729,-4489.5904 247.6581,-4498.885 252.3444,-4493.6852\" stroke=\"#000000\"/>\n</g>\n<!-- 139879758968032 -->\n<g class=\"node\" id=\"node47\">\n<title>139879758968032</title>\n<polygon fill=\"none\" points=\"187.5,-4307.5 187.5,-4343.5 407.5,-4343.5 407.5,-4307.5 187.5,-4307.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"297.5\" y=\"-4321.8\">Encoder-5-FeedForward-Add: Add</text>\n</g>\n<!-- 139879765265264&#45;&gt;139879758968032 -->\n<g class=\"edge\" id=\"edge55\">\n<title>139879765265264-&gt;139879758968032</title>\n<path d=\"M325.3471,-4526.291C335.3936,-4516.6134 346.4097,-4503.8624 352.5,-4490 372.2044,-4445.1498 379.6917,-4424.6329 359.5,-4380 354.2852,-4368.4728 345.2939,-4358.4147 335.751,-4350.1833\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"337.6666,-4347.2331 327.676,-4343.7064 333.2868,-4352.6936 337.6666,-4347.2331\" stroke=\"#000000\"/>\n</g>\n<!-- 139879765389040 -->\n<g class=\"node\" id=\"node46\">\n<title>139879765389040</title>\n<polygon fill=\"none\" points=\"83,-4380.5 83,-4416.5 350,-4416.5 350,-4380.5 83,-4380.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"216.5\" y=\"-4394.8\">Encoder-5-FeedForward-Dropout: Dropout</text>\n</g>\n<!-- 139879765266160&#45;&gt;139879765389040 -->\n<g class=\"edge\" id=\"edge54\">\n<title>139879765266160-&gt;139879765389040</title>\n<path d=\"M221.0169,-4453.4551C220.3534,-4445.3828 219.5556,-4435.6764 218.8163,-4426.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"222.2944,-4426.27 217.9869,-4416.5904 215.3179,-4426.8435 222.2944,-4426.27\" stroke=\"#000000\"/>\n</g>\n<!-- 139879765389040&#45;&gt;139879758968032 -->\n<g class=\"edge\" id=\"edge56\">\n<title>139879765389040-&gt;139879758968032</title>\n<path d=\"M236.5225,-4380.4551C246.5502,-4371.4177 258.8509,-4360.3319 269.7714,-4350.4899\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"272.3419,-4352.885 277.4271,-4343.5904 267.6556,-4347.6852 272.3419,-4352.885\" stroke=\"#000000\"/>\n</g>\n<!-- 139879759165648 -->\n<g class=\"node\" id=\"node48\">\n<title>139879759165648</title>\n<polygon fill=\"none\" points=\"138.5,-4234.5 138.5,-4270.5 456.5,-4270.5 456.5,-4234.5 138.5,-4234.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"297.5\" y=\"-4248.8\">Encoder-5-FeedForward-Norm: LayerNormalization</text>\n</g>\n<!-- 139879758968032&#45;&gt;139879759165648 -->\n<g class=\"edge\" id=\"edge57\">\n<title>139879758968032-&gt;139879759165648</title>\n<path d=\"M297.5,-4307.4551C297.5,-4299.3828 297.5,-4289.6764 297.5,-4280.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"301.0001,-4280.5903 297.5,-4270.5904 294.0001,-4280.5904 301.0001,-4280.5903\" stroke=\"#000000\"/>\n</g>\n<!-- 139879759164864 -->\n<g class=\"node\" id=\"node49\">\n<title>139879759164864</title>\n<polygon fill=\"none\" points=\"27.5,-4161.5 27.5,-4197.5 369.5,-4197.5 369.5,-4161.5 27.5,-4161.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"198.5\" y=\"-4175.8\">Encoder-6-MultiHeadSelfAttention: MultiHeadAttention</text>\n</g>\n<!-- 139879759165648&#45;&gt;139879759164864 -->\n<g class=\"edge\" id=\"edge58\">\n<title>139879759165648-&gt;139879759164864</title>\n<path d=\"M273.0281,-4234.4551C260.4149,-4225.1545 244.8597,-4213.6844 231.2286,-4203.6332\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"233.1592,-4200.7082 223.0335,-4197.5904 229.0049,-4206.3421 233.1592,-4200.7082\" stroke=\"#000000\"/>\n</g>\n<!-- 139879758586432 -->\n<g class=\"node\" id=\"node51\">\n<title>139879758586432</title>\n<polygon fill=\"none\" points=\"158,-4015.5 158,-4051.5 439,-4051.5 439,-4015.5 158,-4015.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"298.5\" y=\"-4029.8\">Encoder-6-MultiHeadSelfAttention-Add: Add</text>\n</g>\n<!-- 139879759165648&#45;&gt;139879758586432 -->\n<g class=\"edge\" id=\"edge60\">\n<title>139879759165648-&gt;139879758586432</title>\n<path d=\"M340.0685,-4234.3804C355.0065,-4225.7 370.1359,-4213.7088 378.5,-4198 386.1793,-4183.5774 385.9885,-4107.2923 375.5,-4088 368.7016,-4075.4953 357.6452,-4065.1998 346.0052,-4057.0548\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"347.7688,-4054.0266 337.4788,-4051.5031 343.9492,-4059.8927 347.7688,-4054.0266\" stroke=\"#000000\"/>\n</g>\n<!-- 139879758462144 -->\n<g class=\"node\" id=\"node50\">\n<title>139879758462144</title>\n<polygon fill=\"none\" points=\"38,-4088.5 38,-4124.5 367,-4124.5 367,-4088.5 38,-4088.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"202.5\" y=\"-4102.8\">Encoder-6-MultiHeadSelfAttention-Dropout: Dropout</text>\n</g>\n<!-- 139879759164864&#45;&gt;139879758462144 -->\n<g class=\"edge\" id=\"edge59\">\n<title>139879759164864-&gt;139879758462144</title>\n<path d=\"M199.4888,-4161.4551C199.9311,-4153.3828 200.4629,-4143.6764 200.9558,-4134.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"204.4563,-4134.7669 201.5087,-4124.5904 197.4668,-4134.3839 204.4563,-4134.7669\" stroke=\"#000000\"/>\n</g>\n<!-- 139879758462144&#45;&gt;139879758586432 -->\n<g class=\"edge\" id=\"edge61\">\n<title>139879758462144-&gt;139879758586432</title>\n<path d=\"M226.2303,-4088.4551C238.3459,-4079.2422 253.2608,-4067.9006 266.3886,-4057.918\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"268.8684,-4060.4293 274.7099,-4051.5904 264.6314,-4054.8573 268.8684,-4060.4293\" stroke=\"#000000\"/>\n</g>\n<!-- 139879752861960 -->\n<g class=\"node\" id=\"node52\">\n<title>139879752861960</title>\n<polygon fill=\"none\" points=\"108.5,-3942.5 108.5,-3978.5 488.5,-3978.5 488.5,-3942.5 108.5,-3942.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"298.5\" y=\"-3956.8\">Encoder-6-MultiHeadSelfAttention-Norm: LayerNormalization</text>\n</g>\n<!-- 139879758586432&#45;&gt;139879752861960 -->\n<g class=\"edge\" id=\"edge62\">\n<title>139879758586432-&gt;139879752861960</title>\n<path d=\"M298.5,-4015.4551C298.5,-4007.3828 298.5,-3997.6764 298.5,-3988.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"302.0001,-3988.5903 298.5,-3978.5904 295.0001,-3988.5904 302.0001,-3988.5903\" stroke=\"#000000\"/>\n</g>\n<!-- 139879752862856 -->\n<g class=\"node\" id=\"node53\">\n<title>139879752862856</title>\n<polygon fill=\"none\" points=\"96.5,-3869.5 96.5,-3905.5 338.5,-3905.5 338.5,-3869.5 96.5,-3869.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"217.5\" y=\"-3883.8\">Encoder-6-FeedForward: FeedForward</text>\n</g>\n<!-- 139879752861960&#45;&gt;139879752862856 -->\n<g class=\"edge\" id=\"edge63\">\n<title>139879752861960-&gt;139879752862856</title>\n<path d=\"M278.4775,-3942.4551C268.4498,-3933.4177 256.1491,-3922.3319 245.2286,-3912.4899\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"247.3444,-3909.6852 237.5729,-3905.5904 242.6581,-3914.885 247.3444,-3909.6852\" stroke=\"#000000\"/>\n</g>\n<!-- 139879752564976 -->\n<g class=\"node\" id=\"node55\">\n<title>139879752564976</title>\n<polygon fill=\"none\" points=\"182.5,-3723.5 182.5,-3759.5 402.5,-3759.5 402.5,-3723.5 182.5,-3723.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"292.5\" y=\"-3737.8\">Encoder-6-FeedForward-Add: Add</text>\n</g>\n<!-- 139879752861960&#45;&gt;139879752564976 -->\n<g class=\"edge\" id=\"edge65\">\n<title>139879752861960-&gt;139879752564976</title>\n<path d=\"M320.3471,-3942.291C330.3936,-3932.6134 341.4097,-3919.8624 347.5,-3906 367.2044,-3861.1498 374.6917,-3840.6329 354.5,-3796 349.2852,-3784.4728 340.2939,-3774.4147 330.751,-3766.1833\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"332.6666,-3763.2331 322.676,-3759.7064 328.2868,-3768.6936 332.6666,-3763.2331\" stroke=\"#000000\"/>\n</g>\n<!-- 139879752515536 -->\n<g class=\"node\" id=\"node54\">\n<title>139879752515536</title>\n<polygon fill=\"none\" points=\"78,-3796.5 78,-3832.5 345,-3832.5 345,-3796.5 78,-3796.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"211.5\" y=\"-3810.8\">Encoder-6-FeedForward-Dropout: Dropout</text>\n</g>\n<!-- 139879752862856&#45;&gt;139879752515536 -->\n<g class=\"edge\" id=\"edge64\">\n<title>139879752862856-&gt;139879752515536</title>\n<path d=\"M216.0169,-3869.4551C215.3534,-3861.3828 214.5556,-3851.6764 213.8163,-3842.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"217.2944,-3842.27 212.9869,-3832.5904 210.3179,-3842.8435 217.2944,-3842.27\" stroke=\"#000000\"/>\n</g>\n<!-- 139879752515536&#45;&gt;139879752564976 -->\n<g class=\"edge\" id=\"edge66\">\n<title>139879752515536-&gt;139879752564976</title>\n<path d=\"M231.5225,-3796.4551C241.5502,-3787.4177 253.8509,-3776.3319 264.7714,-3766.4899\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"267.3419,-3768.885 272.4271,-3759.5904 262.6556,-3763.6852 267.3419,-3768.885\" stroke=\"#000000\"/>\n</g>\n<!-- 139879751937664 -->\n<g class=\"node\" id=\"node56\">\n<title>139879751937664</title>\n<polygon fill=\"none\" points=\"133.5,-3650.5 133.5,-3686.5 451.5,-3686.5 451.5,-3650.5 133.5,-3650.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"292.5\" y=\"-3664.8\">Encoder-6-FeedForward-Norm: LayerNormalization</text>\n</g>\n<!-- 139879752564976&#45;&gt;139879751937664 -->\n<g class=\"edge\" id=\"edge67\">\n<title>139879752564976-&gt;139879751937664</title>\n<path d=\"M292.5,-3723.4551C292.5,-3715.3828 292.5,-3705.6764 292.5,-3696.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"296.0001,-3696.5903 292.5,-3686.5904 289.0001,-3696.5904 296.0001,-3696.5903\" stroke=\"#000000\"/>\n</g>\n<!-- 139879751937272 -->\n<g class=\"node\" id=\"node57\">\n<title>139879751937272</title>\n<polygon fill=\"none\" points=\"22.5,-3577.5 22.5,-3613.5 364.5,-3613.5 364.5,-3577.5 22.5,-3577.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"193.5\" y=\"-3591.8\">Encoder-7-MultiHeadSelfAttention: MultiHeadAttention</text>\n</g>\n<!-- 139879751937664&#45;&gt;139879751937272 -->\n<g class=\"edge\" id=\"edge68\">\n<title>139879751937664-&gt;139879751937272</title>\n<path d=\"M268.0281,-3650.4551C255.4149,-3641.1545 239.8597,-3629.6844 226.2286,-3619.6332\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"228.1592,-3616.7082 218.0335,-3613.5904 224.0049,-3622.3421 228.1592,-3616.7082\" stroke=\"#000000\"/>\n</g>\n<!-- 139879750585144 -->\n<g class=\"node\" id=\"node59\">\n<title>139879750585144</title>\n<polygon fill=\"none\" points=\"153,-3431.5 153,-3467.5 434,-3467.5 434,-3431.5 153,-3431.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"293.5\" y=\"-3445.8\">Encoder-7-MultiHeadSelfAttention-Add: Add</text>\n</g>\n<!-- 139879751937664&#45;&gt;139879750585144 -->\n<g class=\"edge\" id=\"edge70\">\n<title>139879751937664-&gt;139879750585144</title>\n<path d=\"M335.0685,-3650.3804C350.0065,-3641.7 365.1359,-3629.7088 373.5,-3614 381.1793,-3599.5774 380.9885,-3523.2923 370.5,-3504 363.7016,-3491.4953 352.6452,-3481.1998 341.0052,-3473.0548\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"342.7688,-3470.0266 332.4788,-3467.5031 338.9492,-3475.8927 342.7688,-3470.0266\" stroke=\"#000000\"/>\n</g>\n<!-- 139879752029856 -->\n<g class=\"node\" id=\"node58\">\n<title>139879752029856</title>\n<polygon fill=\"none\" points=\"33,-3504.5 33,-3540.5 362,-3540.5 362,-3504.5 33,-3504.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"197.5\" y=\"-3518.8\">Encoder-7-MultiHeadSelfAttention-Dropout: Dropout</text>\n</g>\n<!-- 139879751937272&#45;&gt;139879752029856 -->\n<g class=\"edge\" id=\"edge69\">\n<title>139879751937272-&gt;139879752029856</title>\n<path d=\"M194.4888,-3577.4551C194.9311,-3569.3828 195.4629,-3559.6764 195.9558,-3550.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"199.4563,-3550.7669 196.5087,-3540.5904 192.4668,-3550.3839 199.4563,-3550.7669\" stroke=\"#000000\"/>\n</g>\n<!-- 139879752029856&#45;&gt;139879750585144 -->\n<g class=\"edge\" id=\"edge71\">\n<title>139879752029856-&gt;139879750585144</title>\n<path d=\"M221.2303,-3504.4551C233.3459,-3495.2422 248.2608,-3483.9006 261.3886,-3473.918\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"263.8684,-3476.4293 269.7099,-3467.5904 259.6314,-3470.8573 263.8684,-3476.4293\" stroke=\"#000000\"/>\n</g>\n<!-- 139879748460216 -->\n<g class=\"node\" id=\"node60\">\n<title>139879748460216</title>\n<polygon fill=\"none\" points=\"103.5,-3358.5 103.5,-3394.5 483.5,-3394.5 483.5,-3358.5 103.5,-3358.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"293.5\" y=\"-3372.8\">Encoder-7-MultiHeadSelfAttention-Norm: LayerNormalization</text>\n</g>\n<!-- 139879750585144&#45;&gt;139879748460216 -->\n<g class=\"edge\" id=\"edge72\">\n<title>139879750585144-&gt;139879748460216</title>\n<path d=\"M293.5,-3431.4551C293.5,-3423.3828 293.5,-3413.6764 293.5,-3404.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"297.0001,-3404.5903 293.5,-3394.5904 290.0001,-3404.5904 297.0001,-3404.5903\" stroke=\"#000000\"/>\n</g>\n<!-- 139879748458984 -->\n<g class=\"node\" id=\"node61\">\n<title>139879748458984</title>\n<polygon fill=\"none\" points=\"91.5,-3285.5 91.5,-3321.5 333.5,-3321.5 333.5,-3285.5 91.5,-3285.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"212.5\" y=\"-3299.8\">Encoder-7-FeedForward: FeedForward</text>\n</g>\n<!-- 139879748460216&#45;&gt;139879748458984 -->\n<g class=\"edge\" id=\"edge73\">\n<title>139879748460216-&gt;139879748458984</title>\n<path d=\"M273.4775,-3358.4551C263.4498,-3349.4177 251.1491,-3338.3319 240.2286,-3328.4899\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"242.3444,-3325.6852 232.5729,-3321.5904 237.6581,-3330.885 242.3444,-3325.6852\" stroke=\"#000000\"/>\n</g>\n<!-- 139879747427128 -->\n<g class=\"node\" id=\"node63\">\n<title>139879747427128</title>\n<polygon fill=\"none\" points=\"177.5,-3139.5 177.5,-3175.5 397.5,-3175.5 397.5,-3139.5 177.5,-3139.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"287.5\" y=\"-3153.8\">Encoder-7-FeedForward-Add: Add</text>\n</g>\n<!-- 139879748460216&#45;&gt;139879747427128 -->\n<g class=\"edge\" id=\"edge75\">\n<title>139879748460216-&gt;139879747427128</title>\n<path d=\"M315.3471,-3358.291C325.3936,-3348.6134 336.4097,-3335.8624 342.5,-3322 362.2044,-3277.1498 369.6917,-3256.6329 349.5,-3212 344.2852,-3200.4728 335.2939,-3190.4147 325.751,-3182.1833\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"327.6666,-3179.2331 317.676,-3175.7064 323.2868,-3184.6936 327.6666,-3179.2331\" stroke=\"#000000\"/>\n</g>\n<!-- 139879748671080 -->\n<g class=\"node\" id=\"node62\">\n<title>139879748671080</title>\n<polygon fill=\"none\" points=\"73,-3212.5 73,-3248.5 340,-3248.5 340,-3212.5 73,-3212.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"206.5\" y=\"-3226.8\">Encoder-7-FeedForward-Dropout: Dropout</text>\n</g>\n<!-- 139879748458984&#45;&gt;139879748671080 -->\n<g class=\"edge\" id=\"edge74\">\n<title>139879748458984-&gt;139879748671080</title>\n<path d=\"M211.0169,-3285.4551C210.3534,-3277.3828 209.5556,-3267.6764 208.8163,-3258.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"212.2944,-3258.27 207.9869,-3248.5904 205.3179,-3258.8435 212.2944,-3258.27\" stroke=\"#000000\"/>\n</g>\n<!-- 139879748671080&#45;&gt;139879747427128 -->\n<g class=\"edge\" id=\"edge76\">\n<title>139879748671080-&gt;139879747427128</title>\n<path d=\"M226.5225,-3212.4551C236.5502,-3203.4177 248.8509,-3192.3319 259.7714,-3182.4899\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"262.3419,-3184.885 267.4271,-3175.5904 257.6556,-3179.6852 262.3419,-3184.885\" stroke=\"#000000\"/>\n</g>\n<!-- 139879747634736 -->\n<g class=\"node\" id=\"node64\">\n<title>139879747634736</title>\n<polygon fill=\"none\" points=\"128.5,-3066.5 128.5,-3102.5 446.5,-3102.5 446.5,-3066.5 128.5,-3066.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"287.5\" y=\"-3080.8\">Encoder-7-FeedForward-Norm: LayerNormalization</text>\n</g>\n<!-- 139879747427128&#45;&gt;139879747634736 -->\n<g class=\"edge\" id=\"edge77\">\n<title>139879747427128-&gt;139879747634736</title>\n<path d=\"M287.5,-3139.4551C287.5,-3131.3828 287.5,-3121.6764 287.5,-3112.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"291.0001,-3112.5903 287.5,-3102.5904 284.0001,-3112.5904 291.0001,-3112.5903\" stroke=\"#000000\"/>\n</g>\n<!-- 139879741145040 -->\n<g class=\"node\" id=\"node65\">\n<title>139879741145040</title>\n<polygon fill=\"none\" points=\"17.5,-2993.5 17.5,-3029.5 359.5,-3029.5 359.5,-2993.5 17.5,-2993.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"188.5\" y=\"-3007.8\">Encoder-8-MultiHeadSelfAttention: MultiHeadAttention</text>\n</g>\n<!-- 139879747634736&#45;&gt;139879741145040 -->\n<g class=\"edge\" id=\"edge78\">\n<title>139879747634736-&gt;139879741145040</title>\n<path d=\"M263.0281,-3066.4551C250.4149,-3057.1545 234.8597,-3045.6844 221.2286,-3035.6332\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"223.1592,-3032.7082 213.0335,-3029.5904 219.0049,-3038.3421 223.1592,-3032.7082\" stroke=\"#000000\"/>\n</g>\n<!-- 139879741242952 -->\n<g class=\"node\" id=\"node67\">\n<title>139879741242952</title>\n<polygon fill=\"none\" points=\"149,-2847.5 149,-2883.5 430,-2883.5 430,-2847.5 149,-2847.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"289.5\" y=\"-2861.8\">Encoder-8-MultiHeadSelfAttention-Add: Add</text>\n</g>\n<!-- 139879747634736&#45;&gt;139879741242952 -->\n<g class=\"edge\" id=\"edge80\">\n<title>139879747634736-&gt;139879741242952</title>\n<path d=\"M330.0685,-3066.3804C345.0065,-3057.7 360.1359,-3045.7088 368.5,-3030 376.5894,-3014.8071 374.2511,-2937.9102 364.5,-2920 357.7494,-2907.601 346.8204,-2897.3006 335.3687,-2889.1162\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"337.2521,-2886.1654 326.99,-2883.531 333.3694,-2891.9899 337.2521,-2886.1654\" stroke=\"#000000\"/>\n</g>\n<!-- 139879741195656 -->\n<g class=\"node\" id=\"node66\">\n<title>139879741195656</title>\n<polygon fill=\"none\" points=\"27,-2920.5 27,-2956.5 356,-2956.5 356,-2920.5 27,-2920.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"191.5\" y=\"-2934.8\">Encoder-8-MultiHeadSelfAttention-Dropout: Dropout</text>\n</g>\n<!-- 139879741145040&#45;&gt;139879741195656 -->\n<g class=\"edge\" id=\"edge79\">\n<title>139879741145040-&gt;139879741195656</title>\n<path d=\"M189.2416,-2993.4551C189.5733,-2985.3828 189.9722,-2975.6764 190.3418,-2966.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"193.8429,-2966.7257 190.7566,-2956.5904 186.8488,-2966.4382 193.8429,-2966.7257\" stroke=\"#000000\"/>\n</g>\n<!-- 139879741195656&#45;&gt;139879741242952 -->\n<g class=\"edge\" id=\"edge81\">\n<title>139879741195656-&gt;139879741242952</title>\n<path d=\"M215.7247,-2920.4551C228.2105,-2911.1545 243.6086,-2899.6844 257.102,-2889.6332\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"259.2855,-2892.371 265.2143,-2883.5904 255.1039,-2886.7573 259.2855,-2892.371\" stroke=\"#000000\"/>\n</g>\n<!-- 139879737941912 -->\n<g class=\"node\" id=\"node68\">\n<title>139879737941912</title>\n<polygon fill=\"none\" points=\"99.5,-2774.5 99.5,-2810.5 479.5,-2810.5 479.5,-2774.5 99.5,-2774.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"289.5\" y=\"-2788.8\">Encoder-8-MultiHeadSelfAttention-Norm: LayerNormalization</text>\n</g>\n<!-- 139879741242952&#45;&gt;139879737941912 -->\n<g class=\"edge\" id=\"edge82\">\n<title>139879741242952-&gt;139879737941912</title>\n<path d=\"M289.5,-2847.4551C289.5,-2839.3828 289.5,-2829.6764 289.5,-2820.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"293.0001,-2820.5903 289.5,-2810.5904 286.0001,-2820.5904 293.0001,-2820.5903\" stroke=\"#000000\"/>\n</g>\n<!-- 139879732227936 -->\n<g class=\"node\" id=\"node69\">\n<title>139879732227936</title>\n<polygon fill=\"none\" points=\"87.5,-2701.5 87.5,-2737.5 329.5,-2737.5 329.5,-2701.5 87.5,-2701.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"208.5\" y=\"-2715.8\">Encoder-8-FeedForward: FeedForward</text>\n</g>\n<!-- 139879737941912&#45;&gt;139879732227936 -->\n<g class=\"edge\" id=\"edge83\">\n<title>139879737941912-&gt;139879732227936</title>\n<path d=\"M269.4775,-2774.4551C259.4498,-2765.4177 247.1491,-2754.3319 236.2286,-2744.4899\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"238.3444,-2741.6852 228.5729,-2737.5904 233.6581,-2746.885 238.3444,-2741.6852\" stroke=\"#000000\"/>\n</g>\n<!-- 139879732301328 -->\n<g class=\"node\" id=\"node71\">\n<title>139879732301328</title>\n<polygon fill=\"none\" points=\"173.5,-2555.5 173.5,-2591.5 393.5,-2591.5 393.5,-2555.5 173.5,-2555.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"283.5\" y=\"-2569.8\">Encoder-8-FeedForward-Add: Add</text>\n</g>\n<!-- 139879737941912&#45;&gt;139879732301328 -->\n<g class=\"edge\" id=\"edge85\">\n<title>139879737941912-&gt;139879732301328</title>\n<path d=\"M311.3471,-2774.291C321.3936,-2764.6134 332.4097,-2751.8624 338.5,-2738 358.2044,-2693.1498 365.6917,-2672.6329 345.5,-2628 340.2852,-2616.4728 331.2939,-2606.4147 321.751,-2598.1833\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"323.6666,-2595.2331 313.676,-2591.7064 319.2868,-2600.6936 323.6666,-2595.2331\" stroke=\"#000000\"/>\n</g>\n<!-- 139879732286744 -->\n<g class=\"node\" id=\"node70\">\n<title>139879732286744</title>\n<polygon fill=\"none\" points=\"69,-2628.5 69,-2664.5 336,-2664.5 336,-2628.5 69,-2628.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"202.5\" y=\"-2642.8\">Encoder-8-FeedForward-Dropout: Dropout</text>\n</g>\n<!-- 139879732227936&#45;&gt;139879732286744 -->\n<g class=\"edge\" id=\"edge84\">\n<title>139879732227936-&gt;139879732286744</title>\n<path d=\"M207.0169,-2701.4551C206.3534,-2693.3828 205.5556,-2683.6764 204.8163,-2674.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"208.2944,-2674.27 203.9869,-2664.5904 201.3179,-2674.8435 208.2944,-2674.27\" stroke=\"#000000\"/>\n</g>\n<!-- 139879732286744&#45;&gt;139879732301328 -->\n<g class=\"edge\" id=\"edge86\">\n<title>139879732286744-&gt;139879732301328</title>\n<path d=\"M222.5225,-2628.4551C232.5502,-2619.4177 244.8509,-2608.3319 255.7714,-2598.4899\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"258.3419,-2600.885 263.4271,-2591.5904 253.6556,-2595.6852 258.3419,-2600.885\" stroke=\"#000000\"/>\n</g>\n<!-- 139879731547664 -->\n<g class=\"node\" id=\"node72\">\n<title>139879731547664</title>\n<polygon fill=\"none\" points=\"124.5,-2482.5 124.5,-2518.5 442.5,-2518.5 442.5,-2482.5 124.5,-2482.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"283.5\" y=\"-2496.8\">Encoder-8-FeedForward-Norm: LayerNormalization</text>\n</g>\n<!-- 139879732301328&#45;&gt;139879731547664 -->\n<g class=\"edge\" id=\"edge87\">\n<title>139879732301328-&gt;139879731547664</title>\n<path d=\"M283.5,-2555.4551C283.5,-2547.3828 283.5,-2537.6764 283.5,-2528.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"287.0001,-2528.5903 283.5,-2518.5904 280.0001,-2528.5904 287.0001,-2528.5903\" stroke=\"#000000\"/>\n</g>\n<!-- 139879731608880 -->\n<g class=\"node\" id=\"node73\">\n<title>139879731608880</title>\n<polygon fill=\"none\" points=\"13.5,-2409.5 13.5,-2445.5 355.5,-2445.5 355.5,-2409.5 13.5,-2409.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"184.5\" y=\"-2423.8\">Encoder-9-MultiHeadSelfAttention: MultiHeadAttention</text>\n</g>\n<!-- 139879731547664&#45;&gt;139879731608880 -->\n<g class=\"edge\" id=\"edge88\">\n<title>139879731547664-&gt;139879731608880</title>\n<path d=\"M259.0281,-2482.4551C246.4149,-2473.1545 230.8597,-2461.6844 217.2286,-2451.6332\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"219.1592,-2448.7082 209.0335,-2445.5904 215.0049,-2454.3421 219.1592,-2448.7082\" stroke=\"#000000\"/>\n</g>\n<!-- 139879728286688 -->\n<g class=\"node\" id=\"node75\">\n<title>139879728286688</title>\n<polygon fill=\"none\" points=\"145,-2263.5 145,-2299.5 426,-2299.5 426,-2263.5 145,-2263.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"285.5\" y=\"-2277.8\">Encoder-9-MultiHeadSelfAttention-Add: Add</text>\n</g>\n<!-- 139879731547664&#45;&gt;139879728286688 -->\n<g class=\"edge\" id=\"edge90\">\n<title>139879731547664-&gt;139879728286688</title>\n<path d=\"M326.0685,-2482.3804C341.0065,-2473.7 356.1359,-2461.7088 364.5,-2446 372.5894,-2430.8071 370.2511,-2353.9102 360.5,-2336 353.7494,-2323.601 342.8204,-2313.3006 331.3687,-2305.1162\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"333.2521,-2302.1654 322.99,-2299.531 329.3694,-2307.9899 333.2521,-2302.1654\" stroke=\"#000000\"/>\n</g>\n<!-- 139879731657808 -->\n<g class=\"node\" id=\"node74\">\n<title>139879731657808</title>\n<polygon fill=\"none\" points=\"23,-2336.5 23,-2372.5 352,-2372.5 352,-2336.5 23,-2336.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"187.5\" y=\"-2350.8\">Encoder-9-MultiHeadSelfAttention-Dropout: Dropout</text>\n</g>\n<!-- 139879731608880&#45;&gt;139879731657808 -->\n<g class=\"edge\" id=\"edge89\">\n<title>139879731608880-&gt;139879731657808</title>\n<path d=\"M185.2416,-2409.4551C185.5733,-2401.3828 185.9722,-2391.6764 186.3418,-2382.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"189.8429,-2382.7257 186.7566,-2372.5904 182.8488,-2382.4382 189.8429,-2382.7257\" stroke=\"#000000\"/>\n</g>\n<!-- 139879731657808&#45;&gt;139879728286688 -->\n<g class=\"edge\" id=\"edge91\">\n<title>139879731657808-&gt;139879728286688</title>\n<path d=\"M211.7247,-2336.4551C224.2105,-2327.1545 239.6086,-2315.6844 253.102,-2305.6332\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"255.2855,-2308.371 261.2143,-2299.5904 251.1039,-2302.7573 255.2855,-2308.371\" stroke=\"#000000\"/>\n</g>\n<!-- 139879723130328 -->\n<g class=\"node\" id=\"node76\">\n<title>139879723130328</title>\n<polygon fill=\"none\" points=\"95.5,-2190.5 95.5,-2226.5 475.5,-2226.5 475.5,-2190.5 95.5,-2190.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"285.5\" y=\"-2204.8\">Encoder-9-MultiHeadSelfAttention-Norm: LayerNormalization</text>\n</g>\n<!-- 139879728286688&#45;&gt;139879723130328 -->\n<g class=\"edge\" id=\"edge92\">\n<title>139879728286688-&gt;139879723130328</title>\n<path d=\"M285.5,-2263.4551C285.5,-2255.3828 285.5,-2245.6764 285.5,-2236.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"289.0001,-2236.5903 285.5,-2226.5904 282.0001,-2236.5904 289.0001,-2236.5903\" stroke=\"#000000\"/>\n</g>\n<!-- 139879723248944 -->\n<g class=\"node\" id=\"node77\">\n<title>139879723248944</title>\n<polygon fill=\"none\" points=\"83.5,-2117.5 83.5,-2153.5 325.5,-2153.5 325.5,-2117.5 83.5,-2117.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"204.5\" y=\"-2131.8\">Encoder-9-FeedForward: FeedForward</text>\n</g>\n<!-- 139879723130328&#45;&gt;139879723248944 -->\n<g class=\"edge\" id=\"edge93\">\n<title>139879723130328-&gt;139879723248944</title>\n<path d=\"M265.4775,-2190.4551C255.4498,-2181.4177 243.1491,-2170.3319 232.2286,-2160.4899\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"234.3444,-2157.6852 224.5729,-2153.5904 229.6581,-2162.885 234.3444,-2157.6852\" stroke=\"#000000\"/>\n</g>\n<!-- 139879719148456 -->\n<g class=\"node\" id=\"node79\">\n<title>139879719148456</title>\n<polygon fill=\"none\" points=\"169.5,-1971.5 169.5,-2007.5 389.5,-2007.5 389.5,-1971.5 169.5,-1971.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"279.5\" y=\"-1985.8\">Encoder-9-FeedForward-Add: Add</text>\n</g>\n<!-- 139879723130328&#45;&gt;139879719148456 -->\n<g class=\"edge\" id=\"edge95\">\n<title>139879723130328-&gt;139879719148456</title>\n<path d=\"M307.3471,-2190.291C317.3936,-2180.6134 328.4097,-2167.8624 334.5,-2154 354.2044,-2109.1498 361.6917,-2088.6329 341.5,-2044 336.2852,-2032.4728 327.2939,-2022.4147 317.751,-2014.1833\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"319.6666,-2011.2331 309.676,-2007.7064 315.2868,-2016.6936 319.6666,-2011.2331\" stroke=\"#000000\"/>\n</g>\n<!-- 139879723277392 -->\n<g class=\"node\" id=\"node78\">\n<title>139879723277392</title>\n<polygon fill=\"none\" points=\"65,-2044.5 65,-2080.5 332,-2080.5 332,-2044.5 65,-2044.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"198.5\" y=\"-2058.8\">Encoder-9-FeedForward-Dropout: Dropout</text>\n</g>\n<!-- 139879723248944&#45;&gt;139879723277392 -->\n<g class=\"edge\" id=\"edge94\">\n<title>139879723248944-&gt;139879723277392</title>\n<path d=\"M203.0169,-2117.4551C202.3534,-2109.3828 201.5556,-2099.6764 200.8163,-2090.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"204.2944,-2090.27 199.9869,-2080.5904 197.3179,-2090.8435 204.2944,-2090.27\" stroke=\"#000000\"/>\n</g>\n<!-- 139879723277392&#45;&gt;139879719148456 -->\n<g class=\"edge\" id=\"edge96\">\n<title>139879723277392-&gt;139879719148456</title>\n<path d=\"M218.5225,-2044.4551C228.5502,-2035.4177 240.8509,-2024.3319 251.7714,-2014.4899\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"254.3419,-2016.885 259.4271,-2007.5904 249.6556,-2011.6852 254.3419,-2016.885\" stroke=\"#000000\"/>\n</g>\n<!-- 139879718584048 -->\n<g class=\"node\" id=\"node80\">\n<title>139879718584048</title>\n<polygon fill=\"none\" points=\"120.5,-1898.5 120.5,-1934.5 438.5,-1934.5 438.5,-1898.5 120.5,-1898.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"279.5\" y=\"-1912.8\">Encoder-9-FeedForward-Norm: LayerNormalization</text>\n</g>\n<!-- 139879719148456&#45;&gt;139879718584048 -->\n<g class=\"edge\" id=\"edge97\">\n<title>139879719148456-&gt;139879718584048</title>\n<path d=\"M279.5,-1971.4551C279.5,-1963.3828 279.5,-1953.6764 279.5,-1944.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"283.0001,-1944.5903 279.5,-1934.5904 276.0001,-1944.5904 283.0001,-1944.5903\" stroke=\"#000000\"/>\n</g>\n<!-- 139879718583264 -->\n<g class=\"node\" id=\"node81\">\n<title>139879718583264</title>\n<polygon fill=\"none\" points=\"3.5,-1825.5 3.5,-1861.5 353.5,-1861.5 353.5,-1825.5 3.5,-1825.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"178.5\" y=\"-1839.8\">Encoder-10-MultiHeadSelfAttention: MultiHeadAttention</text>\n</g>\n<!-- 139879718584048&#45;&gt;139879718583264 -->\n<g class=\"edge\" id=\"edge98\">\n<title>139879718584048-&gt;139879718583264</title>\n<path d=\"M254.5337,-1898.4551C241.6658,-1889.1545 225.7962,-1877.6844 211.8898,-1867.6332\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"213.6841,-1864.6116 203.5291,-1861.5904 209.5836,-1870.2849 213.6841,-1864.6116\" stroke=\"#000000\"/>\n</g>\n<!-- 139879718771064 -->\n<g class=\"node\" id=\"node83\">\n<title>139879718771064</title>\n<polygon fill=\"none\" points=\"137,-1679.5 137,-1715.5 426,-1715.5 426,-1679.5 137,-1679.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"281.5\" y=\"-1693.8\">Encoder-10-MultiHeadSelfAttention-Add: Add</text>\n</g>\n<!-- 139879718584048&#45;&gt;139879718771064 -->\n<g class=\"edge\" id=\"edge100\">\n<title>139879718584048-&gt;139879718771064</title>\n<path d=\"M323.5299,-1898.3618C338.6906,-1889.7299 353.9854,-1877.7771 362.5,-1862 370.6814,-1846.8405 368.4328,-1769.9762 358.5,-1752 351.6163,-1739.542 340.5325,-1729.2616 328.8935,-1721.116\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"330.6612,-1718.0903 320.3726,-1715.5614 326.8385,-1723.9544 330.6612,-1718.0903\" stroke=\"#000000\"/>\n</g>\n<!-- 139879718681792 -->\n<g class=\"node\" id=\"node82\">\n<title>139879718681792</title>\n<polygon fill=\"none\" points=\"13.5,-1752.5 13.5,-1788.5 349.5,-1788.5 349.5,-1752.5 13.5,-1752.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"181.5\" y=\"-1766.8\">Encoder-10-MultiHeadSelfAttention-Dropout: Dropout</text>\n</g>\n<!-- 139879718583264&#45;&gt;139879718681792 -->\n<g class=\"edge\" id=\"edge99\">\n<title>139879718583264-&gt;139879718681792</title>\n<path d=\"M179.2416,-1825.4551C179.5733,-1817.3828 179.9722,-1807.6764 180.3418,-1798.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"183.8429,-1798.7257 180.7566,-1788.5904 176.8488,-1798.4382 183.8429,-1798.7257\" stroke=\"#000000\"/>\n</g>\n<!-- 139879718681792&#45;&gt;139879718771064 -->\n<g class=\"edge\" id=\"edge101\">\n<title>139879718681792-&gt;139879718771064</title>\n<path d=\"M206.2191,-1752.4551C218.9597,-1743.1545 234.6721,-1731.6844 248.4408,-1721.6332\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"250.7055,-1724.3134 256.7187,-1715.5904 246.5781,-1718.6596 250.7055,-1724.3134\" stroke=\"#000000\"/>\n</g>\n<!-- 139879715880016 -->\n<g class=\"node\" id=\"node84\">\n<title>139879715880016</title>\n<polygon fill=\"none\" points=\"88,-1606.5 88,-1642.5 475,-1642.5 475,-1606.5 88,-1606.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"281.5\" y=\"-1620.8\">Encoder-10-MultiHeadSelfAttention-Norm: LayerNormalization</text>\n</g>\n<!-- 139879718771064&#45;&gt;139879715880016 -->\n<g class=\"edge\" id=\"edge102\">\n<title>139879718771064-&gt;139879715880016</title>\n<path d=\"M281.5,-1679.4551C281.5,-1671.3828 281.5,-1661.6764 281.5,-1652.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"285.0001,-1652.5903 281.5,-1642.5904 278.0001,-1652.5904 285.0001,-1652.5903\" stroke=\"#000000\"/>\n</g>\n<!-- 139879715441856 -->\n<g class=\"node\" id=\"node85\">\n<title>139879715441856</title>\n<polygon fill=\"none\" points=\"74,-1533.5 74,-1569.5 323,-1569.5 323,-1533.5 74,-1533.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"198.5\" y=\"-1547.8\">Encoder-10-FeedForward: FeedForward</text>\n</g>\n<!-- 139879715880016&#45;&gt;139879715441856 -->\n<g class=\"edge\" id=\"edge103\">\n<title>139879715880016-&gt;139879715441856</title>\n<path d=\"M260.9831,-1606.4551C250.608,-1597.3299 237.8584,-1586.1165 226.5877,-1576.2036\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"228.8889,-1573.5665 219.0685,-1569.5904 224.2659,-1578.8228 228.8889,-1573.5665\" stroke=\"#000000\"/>\n</g>\n<!-- 139879715629376 -->\n<g class=\"node\" id=\"node87\">\n<title>139879715629376</title>\n<polygon fill=\"none\" points=\"162,-1387.5 162,-1423.5 389,-1423.5 389,-1387.5 162,-1387.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"275.5\" y=\"-1401.8\">Encoder-10-FeedForward-Add: Add</text>\n</g>\n<!-- 139879715880016&#45;&gt;139879715629376 -->\n<g class=\"edge\" id=\"edge105\">\n<title>139879715880016-&gt;139879715629376</title>\n<path d=\"M303.7102,-1606.3371C313.9414,-1596.6709 325.1891,-1583.916 331.5,-1570 351.745,-1525.358 360.0543,-1504.5004 339.5,-1460 334.0436,-1448.1867 324.6242,-1437.9911 314.6643,-1429.7193\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"316.678,-1426.8521 306.6262,-1423.5038 312.396,-1432.3897 316.678,-1426.8521\" stroke=\"#000000\"/>\n</g>\n<!-- 139879715527816 -->\n<g class=\"node\" id=\"node86\">\n<title>139879715527816</title>\n<polygon fill=\"none\" points=\"55,-1460.5 55,-1496.5 330,-1496.5 330,-1460.5 55,-1460.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"192.5\" y=\"-1474.8\">Encoder-10-FeedForward-Dropout: Dropout</text>\n</g>\n<!-- 139879715441856&#45;&gt;139879715527816 -->\n<g class=\"edge\" id=\"edge104\">\n<title>139879715441856-&gt;139879715527816</title>\n<path d=\"M197.0169,-1533.4551C196.3534,-1525.3828 195.5556,-1515.6764 194.8163,-1506.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"198.2944,-1506.27 193.9869,-1496.5904 191.3179,-1506.8435 198.2944,-1506.27\" stroke=\"#000000\"/>\n</g>\n<!-- 139879715527816&#45;&gt;139879715629376 -->\n<g class=\"edge\" id=\"edge106\">\n<title>139879715527816-&gt;139879715629376</title>\n<path d=\"M213.0169,-1460.4551C223.392,-1451.3299 236.1416,-1440.1165 247.4123,-1430.2036\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"249.7341,-1432.8228 254.9315,-1423.5904 245.1111,-1427.5665 249.7341,-1432.8228\" stroke=\"#000000\"/>\n</g>\n<!-- 139879713741960 -->\n<g class=\"node\" id=\"node88\">\n<title>139879713741960</title>\n<polygon fill=\"none\" points=\"112.5,-1314.5 112.5,-1350.5 438.5,-1350.5 438.5,-1314.5 112.5,-1314.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"275.5\" y=\"-1328.8\">Encoder-10-FeedForward-Norm: LayerNormalization</text>\n</g>\n<!-- 139879715629376&#45;&gt;139879713741960 -->\n<g class=\"edge\" id=\"edge107\">\n<title>139879715629376-&gt;139879713741960</title>\n<path d=\"M275.5,-1387.4551C275.5,-1379.3828 275.5,-1369.6764 275.5,-1360.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"279.0001,-1360.5903 275.5,-1350.5904 272.0001,-1360.5904 279.0001,-1360.5903\" stroke=\"#000000\"/>\n</g>\n<!-- 139879713741176 -->\n<g class=\"node\" id=\"node89\">\n<title>139879713741176</title>\n<polygon fill=\"none\" points=\"0,-1241.5 0,-1277.5 349,-1277.5 349,-1241.5 0,-1241.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"174.5\" y=\"-1255.8\">Encoder-11-MultiHeadSelfAttention: MultiHeadAttention</text>\n</g>\n<!-- 139879713741960&#45;&gt;139879713741176 -->\n<g class=\"edge\" id=\"edge108\">\n<title>139879713741960-&gt;139879713741176</title>\n<path d=\"M250.5337,-1314.4551C237.6658,-1305.1545 221.7962,-1293.6844 207.8898,-1283.6332\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"209.6841,-1280.6116 199.5291,-1277.5904 205.5836,-1286.2849 209.6841,-1280.6116\" stroke=\"#000000\"/>\n</g>\n<!-- 139879710508816 -->\n<g class=\"node\" id=\"node91\">\n<title>139879710508816</title>\n<polygon fill=\"none\" points=\"133.5,-1095.5 133.5,-1131.5 421.5,-1131.5 421.5,-1095.5 133.5,-1095.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"277.5\" y=\"-1109.8\">Encoder-11-MultiHeadSelfAttention-Add: Add</text>\n</g>\n<!-- 139879713741960&#45;&gt;139879710508816 -->\n<g class=\"edge\" id=\"edge110\">\n<title>139879713741960-&gt;139879710508816</title>\n<path d=\"M319.5299,-1314.3618C334.6906,-1305.7299 349.9854,-1293.7771 358.5,-1278 366.6814,-1262.8405 364.4328,-1185.9762 354.5,-1168 347.6163,-1155.542 336.5325,-1145.2616 324.8935,-1137.116\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"326.6612,-1134.0903 316.3726,-1131.5614 322.8385,-1139.9544 326.6612,-1134.0903\" stroke=\"#000000\"/>\n</g>\n<!-- 139879713823320 -->\n<g class=\"node\" id=\"node90\">\n<title>139879713823320</title>\n<polygon fill=\"none\" points=\"10,-1168.5 10,-1204.5 345,-1204.5 345,-1168.5 10,-1168.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"177.5\" y=\"-1182.8\">Encoder-11-MultiHeadSelfAttention-Dropout: Dropout</text>\n</g>\n<!-- 139879713741176&#45;&gt;139879713823320 -->\n<g class=\"edge\" id=\"edge109\">\n<title>139879713741176-&gt;139879713823320</title>\n<path d=\"M175.2416,-1241.4551C175.5733,-1233.3828 175.9722,-1223.6764 176.3418,-1214.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"179.8429,-1214.7257 176.7566,-1204.5904 172.8488,-1214.4382 179.8429,-1214.7257\" stroke=\"#000000\"/>\n</g>\n<!-- 139879713823320&#45;&gt;139879710508816 -->\n<g class=\"edge\" id=\"edge111\">\n<title>139879713823320-&gt;139879710508816</title>\n<path d=\"M202.2191,-1168.4551C214.9597,-1159.1545 230.6721,-1147.6844 244.4408,-1137.6332\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"246.7055,-1140.3134 252.7187,-1131.5904 242.5781,-1134.6596 246.7055,-1140.3134\" stroke=\"#000000\"/>\n</g>\n<!-- 139879707438216 -->\n<g class=\"node\" id=\"node92\">\n<title>139879707438216</title>\n<polygon fill=\"none\" points=\"84.5,-1022.5 84.5,-1058.5 470.5,-1058.5 470.5,-1022.5 84.5,-1022.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"277.5\" y=\"-1036.8\">Encoder-11-MultiHeadSelfAttention-Norm: LayerNormalization</text>\n</g>\n<!-- 139879710508816&#45;&gt;139879707438216 -->\n<g class=\"edge\" id=\"edge112\">\n<title>139879710508816-&gt;139879707438216</title>\n<path d=\"M277.5,-1095.4551C277.5,-1087.3828 277.5,-1077.6764 277.5,-1068.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"281.0001,-1068.5903 277.5,-1058.5904 274.0001,-1068.5904 281.0001,-1068.5903\" stroke=\"#000000\"/>\n</g>\n<!-- 139879707438720 -->\n<g class=\"node\" id=\"node93\">\n<title>139879707438720</title>\n<polygon fill=\"none\" points=\"77.5,-949.5 77.5,-985.5 325.5,-985.5 325.5,-949.5 77.5,-949.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"201.5\" y=\"-963.8\">Encoder-11-FeedForward: FeedForward</text>\n</g>\n<!-- 139879707438216&#45;&gt;139879707438720 -->\n<g class=\"edge\" id=\"edge113\">\n<title>139879707438216-&gt;139879707438720</title>\n<path d=\"M258.7135,-1022.4551C249.3961,-1013.5054 237.9873,-1002.547 227.8157,-992.7769\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"229.9704,-989.9935 220.3338,-985.5904 225.1212,-995.0419 229.9704,-989.9935\" stroke=\"#000000\"/>\n</g>\n<!-- 139879705851608 -->\n<g class=\"node\" id=\"node95\">\n<title>139879705851608</title>\n<polygon fill=\"none\" points=\"163,-803.5 163,-839.5 390,-839.5 390,-803.5 163,-803.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"276.5\" y=\"-817.8\">Encoder-11-FeedForward-Add: Add</text>\n</g>\n<!-- 139879707438216&#45;&gt;139879705851608 -->\n<g class=\"edge\" id=\"edge115\">\n<title>139879707438216-&gt;139879705851608</title>\n<path d=\"M303.8634,-1022.2916C315.3206,-1012.8699 327.6647,-1000.3358 334.5,-986 355.572,-941.8049 361.0306,-920.4492 340.5,-876 335.0436,-864.1867 325.6242,-853.9911 315.6643,-845.7193\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"317.678,-842.8521 307.6262,-839.5038 313.396,-848.3897 317.678,-842.8521\" stroke=\"#000000\"/>\n</g>\n<!-- 139879705737704 -->\n<g class=\"node\" id=\"node94\">\n<title>139879705737704</title>\n<polygon fill=\"none\" points=\"57.5,-876.5 57.5,-912.5 331.5,-912.5 331.5,-876.5 57.5,-876.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"194.5\" y=\"-890.8\">Encoder-11-FeedForward-Dropout: Dropout</text>\n</g>\n<!-- 139879707438720&#45;&gt;139879705737704 -->\n<g class=\"edge\" id=\"edge114\">\n<title>139879707438720-&gt;139879705737704</title>\n<path d=\"M199.7697,-949.4551C198.9956,-941.3828 198.0649,-931.6764 197.2024,-922.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"200.6733,-922.2106 196.2347,-912.5904 193.7053,-922.8788 200.6733,-922.2106\" stroke=\"#000000\"/>\n</g>\n<!-- 139879705737704&#45;&gt;139879705851608 -->\n<g class=\"edge\" id=\"edge116\">\n<title>139879705737704-&gt;139879705851608</title>\n<path d=\"M214.7697,-876.4551C224.9212,-867.4177 237.3738,-856.3319 248.4291,-846.4899\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"251.0375,-848.8539 256.1793,-839.5904 246.383,-843.6255 251.0375,-848.8539\" stroke=\"#000000\"/>\n</g>\n<!-- 139879700019744 -->\n<g class=\"node\" id=\"node96\">\n<title>139879700019744</title>\n<polygon fill=\"none\" points=\"114,-730.5 114,-766.5 439,-766.5 439,-730.5 114,-730.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"276.5\" y=\"-744.8\">Encoder-11-FeedForward-Norm: LayerNormalization</text>\n</g>\n<!-- 139879705851608&#45;&gt;139879700019744 -->\n<g class=\"edge\" id=\"edge117\">\n<title>139879705851608-&gt;139879700019744</title>\n<path d=\"M276.5,-803.4551C276.5,-795.3828 276.5,-785.6764 276.5,-776.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"280.0001,-776.5903 276.5,-766.5904 273.0001,-776.5904 280.0001,-776.5903\" stroke=\"#000000\"/>\n</g>\n<!-- 139879700018960 -->\n<g class=\"node\" id=\"node97\">\n<title>139879700018960</title>\n<polygon fill=\"none\" points=\".5,-657.5 .5,-693.5 350.5,-693.5 350.5,-657.5 .5,-657.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"175.5\" y=\"-671.8\">Encoder-12-MultiHeadSelfAttention: MultiHeadAttention</text>\n</g>\n<!-- 139879700019744&#45;&gt;139879700018960 -->\n<g class=\"edge\" id=\"edge118\">\n<title>139879700019744-&gt;139879700018960</title>\n<path d=\"M251.5337,-730.4551C238.6658,-721.1545 222.7962,-709.6844 208.8898,-699.6332\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"210.6841,-696.6116 200.5291,-693.5904 206.5836,-702.2849 210.6841,-696.6116\" stroke=\"#000000\"/>\n</g>\n<!-- 139879700161536 -->\n<g class=\"node\" id=\"node99\">\n<title>139879700161536</title>\n<polygon fill=\"none\" points=\"134,-511.5 134,-547.5 423,-547.5 423,-511.5 134,-511.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"278.5\" y=\"-525.8\">Encoder-12-MultiHeadSelfAttention-Add: Add</text>\n</g>\n<!-- 139879700019744&#45;&gt;139879700161536 -->\n<g class=\"edge\" id=\"edge120\">\n<title>139879700019744-&gt;139879700161536</title>\n<path d=\"M320.5299,-730.3618C335.6906,-721.7299 350.9854,-709.7771 359.5,-694 367.6814,-678.8405 365.4328,-601.9762 355.5,-584 348.6163,-571.542 337.5325,-561.2616 325.8935,-553.116\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"327.6612,-550.0903 317.3726,-547.5614 323.8385,-555.9544 327.6612,-550.0903\" stroke=\"#000000\"/>\n</g>\n<!-- 139879700094480 -->\n<g class=\"node\" id=\"node98\">\n<title>139879700094480</title>\n<polygon fill=\"none\" points=\"10.5,-584.5 10.5,-620.5 346.5,-620.5 346.5,-584.5 10.5,-584.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"178.5\" y=\"-598.8\">Encoder-12-MultiHeadSelfAttention-Dropout: Dropout</text>\n</g>\n<!-- 139879700018960&#45;&gt;139879700094480 -->\n<g class=\"edge\" id=\"edge119\">\n<title>139879700018960-&gt;139879700094480</title>\n<path d=\"M176.2416,-657.4551C176.5733,-649.3828 176.9722,-639.6764 177.3418,-630.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"180.8429,-630.7257 177.7566,-620.5904 173.8488,-630.4382 180.8429,-630.7257\" stroke=\"#000000\"/>\n</g>\n<!-- 139879700094480&#45;&gt;139879700161536 -->\n<g class=\"edge\" id=\"edge121\">\n<title>139879700094480-&gt;139879700161536</title>\n<path d=\"M203.2191,-584.4551C215.9597,-575.1545 231.6721,-563.6844 245.4408,-553.6332\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"247.7055,-556.3134 253.7187,-547.5904 243.5781,-550.6596 247.7055,-556.3134\" stroke=\"#000000\"/>\n</g>\n<!-- 139879777073752 -->\n<g class=\"node\" id=\"node100\">\n<title>139879777073752</title>\n<polygon fill=\"none\" points=\"85,-438.5 85,-474.5 472,-474.5 472,-438.5 85,-438.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"278.5\" y=\"-452.8\">Encoder-12-MultiHeadSelfAttention-Norm: LayerNormalization</text>\n</g>\n<!-- 139879700161536&#45;&gt;139879777073752 -->\n<g class=\"edge\" id=\"edge122\">\n<title>139879700161536-&gt;139879777073752</title>\n<path d=\"M278.5,-511.4551C278.5,-503.3828 278.5,-493.6764 278.5,-484.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"282.0001,-484.5903 278.5,-474.5904 275.0001,-484.5904 282.0001,-484.5903\" stroke=\"#000000\"/>\n</g>\n<!-- 139879777074648 -->\n<g class=\"node\" id=\"node101\">\n<title>139879777074648</title>\n<polygon fill=\"none\" points=\"71,-365.5 71,-401.5 320,-401.5 320,-365.5 71,-365.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"195.5\" y=\"-379.8\">Encoder-12-FeedForward: FeedForward</text>\n</g>\n<!-- 139879777073752&#45;&gt;139879777074648 -->\n<g class=\"edge\" id=\"edge123\">\n<title>139879777073752-&gt;139879777074648</title>\n<path d=\"M257.9831,-438.4551C247.608,-429.3299 234.8584,-418.1165 223.5877,-408.2036\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"225.8889,-405.5665 216.0685,-401.5904 221.2659,-410.8228 225.8889,-405.5665\" stroke=\"#000000\"/>\n</g>\n<!-- 139879775720392 -->\n<g class=\"node\" id=\"node103\">\n<title>139879775720392</title>\n<polygon fill=\"none\" points=\"159,-219.5 159,-255.5 386,-255.5 386,-219.5 159,-219.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"272.5\" y=\"-233.8\">Encoder-12-FeedForward-Add: Add</text>\n</g>\n<!-- 139879777073752&#45;&gt;139879775720392 -->\n<g class=\"edge\" id=\"edge125\">\n<title>139879777073752-&gt;139879775720392</title>\n<path d=\"M300.7102,-438.3371C310.9414,-428.6709 322.1891,-415.916 328.5,-402 348.745,-357.358 357.0543,-336.5004 336.5,-292 331.0436,-280.1867 321.6242,-269.9911 311.6643,-261.7193\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"313.678,-258.8521 303.6262,-255.5038 309.396,-264.3897 313.678,-258.8521\" stroke=\"#000000\"/>\n</g>\n<!-- 139879777185184 -->\n<g class=\"node\" id=\"node102\">\n<title>139879777185184</title>\n<polygon fill=\"none\" points=\"52,-292.5 52,-328.5 327,-328.5 327,-292.5 52,-292.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"189.5\" y=\"-306.8\">Encoder-12-FeedForward-Dropout: Dropout</text>\n</g>\n<!-- 139879777074648&#45;&gt;139879777185184 -->\n<g class=\"edge\" id=\"edge124\">\n<title>139879777074648-&gt;139879777185184</title>\n<path d=\"M194.0169,-365.4551C193.3534,-357.3828 192.5556,-347.6764 191.8163,-338.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"195.2944,-338.27 190.9869,-328.5904 188.3179,-338.8435 195.2944,-338.27\" stroke=\"#000000\"/>\n</g>\n<!-- 139879777185184&#45;&gt;139879775720392 -->\n<g class=\"edge\" id=\"edge126\">\n<title>139879777185184-&gt;139879775720392</title>\n<path d=\"M210.0169,-292.4551C220.392,-283.3299 233.1416,-272.1165 244.4123,-262.2036\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"246.7341,-264.8228 251.9315,-255.5904 242.1111,-259.5665 246.7341,-264.8228\" stroke=\"#000000\"/>\n</g>\n<!-- 139879775915144 -->\n<g class=\"node\" id=\"node104\">\n<title>139879775915144</title>\n<polygon fill=\"none\" points=\"109.5,-146.5 109.5,-182.5 435.5,-182.5 435.5,-146.5 109.5,-146.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"272.5\" y=\"-160.8\">Encoder-12-FeedForward-Norm: LayerNormalization</text>\n</g>\n<!-- 139879775720392&#45;&gt;139879775915144 -->\n<g class=\"edge\" id=\"edge127\">\n<title>139879775720392-&gt;139879775915144</title>\n<path d=\"M272.5,-219.4551C272.5,-211.3828 272.5,-201.6764 272.5,-192.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"276.0001,-192.5903 272.5,-182.5904 269.0001,-192.5904 276.0001,-192.5903\" stroke=\"#000000\"/>\n</g>\n<!-- 139879765660896 -->\n<g class=\"node\" id=\"node105\">\n<title>139879765660896</title>\n<polygon fill=\"none\" points=\"175.5,-73.5 175.5,-109.5 369.5,-109.5 369.5,-73.5 175.5,-73.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"272.5\" y=\"-87.8\">non_masking_1: NonMasking</text>\n</g>\n<!-- 139879775915144&#45;&gt;139879765660896 -->\n<g class=\"edge\" id=\"edge128\">\n<title>139879775915144-&gt;139879765660896</title>\n<path d=\"M272.5,-146.4551C272.5,-138.3828 272.5,-128.6764 272.5,-119.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"276.0001,-119.5903 272.5,-109.5904 269.0001,-119.5904 276.0001,-119.5903\" stroke=\"#000000\"/>\n</g>\n<!-- 139879765660392 -->\n<g class=\"node\" id=\"node106\">\n<title>139879765660392</title>\n<polygon fill=\"none\" points=\"46.5,-.5 46.5,-36.5 264.5,-36.5 264.5,-.5 46.5,-.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"155.5\" y=\"-14.8\">my_layer__start_1: MyLayer_Start</text>\n</g>\n<!-- 139879765660896&#45;&gt;139879765660392 -->\n<g class=\"edge\" id=\"edge129\">\n<title>139879765660896-&gt;139879765660392</title>\n<path d=\"M243.5786,-73.4551C228.391,-63.979 209.5937,-52.2508 193.2699,-42.0658\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"194.8309,-38.9145 184.4941,-36.5904 191.1255,-44.8533 194.8309,-38.9145\" stroke=\"#000000\"/>\n</g>\n<!-- 139879765660448 -->\n<g class=\"node\" id=\"node107\">\n<title>139879765660448</title>\n<polygon fill=\"none\" points=\"282.5,-.5 282.5,-36.5 496.5,-36.5 496.5,-.5 282.5,-.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"389.5\" y=\"-14.8\">my_layer__end_1: MyLayer_End</text>\n</g>\n<!-- 139879765660896&#45;&gt;139879765660448 -->\n<g class=\"edge\" id=\"edge130\">\n<title>139879765660896-&gt;139879765660448</title>\n<path d=\"M301.4214,-73.4551C316.609,-63.979 335.4063,-52.2508 351.7301,-42.0658\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"353.8745,-44.8533 360.5059,-36.5904 350.1691,-38.9145 353.8745,-44.8533\" stroke=\"#000000\"/>\n</g>\n</g>\n</svg>"
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 36
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gLLayijHNEjz",
        "colab_type": "text"
      },
      "source": [
        "훈련을 시작합니다.  \n",
        "1epoch을 훈련해 보고, 결과를 확인하고 훈련을 다시 시작할 예정입니다."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-7On1hY3I7o1",
        "colab_type": "code",
        "outputId": "09700577-26d7-4641-f2ed-94f8f2fda042",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "sess = K.get_session()\n",
        "uninitialized_variables = set([i.decode('ascii') for i in sess.run(tf.report_uninitialized_variables())])\n",
        "init = tf.variables_initializer([v for v in tf.global_variables() if v.name.split(':')[0] in uninitialized_variables])\n",
        "sess.run(init)\n",
        "\n",
        "bert_model = get_bert_finetuning_model(model)\n",
        "bert_model.summary()\n",
        "history = bert_model.fit(train_x, train_y, batch_size=10, validation_split=0.05, shuffle=False, verbose=1)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/array_ops.py:1475: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use tf.where in 2.0, which has the same broadcast rule as np.where\n",
            "well\n",
            "(?, 384, 768)\n",
            "(?, 384, 2)\n",
            "(2, ?, 384)\n",
            "(?, 384) (?, 384)\n",
            "No error\n",
            "(?, 384, 768)\n",
            "(2, ?, 384)\n",
            "(?, 384) (?, 384)\n",
            "No error\n",
            "Model: \"model_4\"\n",
            "__________________________________________________________________________________________________\n",
            "Layer (type)                    Output Shape         Param #     Connected to                     \n",
            "==================================================================================================\n",
            "Input-Token (InputLayer)        (None, 384)          0                                            \n",
            "__________________________________________________________________________________________________\n",
            "Input-Segment (InputLayer)      (None, 384)          0                                            \n",
            "__________________________________________________________________________________________________\n",
            "Embedding-Token (TokenEmbedding [(None, 384, 768), ( 91812096    Input-Token[0][0]                \n",
            "__________________________________________________________________________________________________\n",
            "Embedding-Segment (Embedding)   (None, 384, 768)     1536        Input-Segment[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "Embedding-Token-Segment (Add)   (None, 384, 768)     0           Embedding-Token[0][0]            \n",
            "                                                                 Embedding-Segment[0][0]          \n",
            "__________________________________________________________________________________________________\n",
            "Embedding-Position (PositionEmb (None, 384, 768)     294912      Embedding-Token-Segment[0][0]    \n",
            "__________________________________________________________________________________________________\n",
            "Embedding-Dropout (Dropout)     (None, 384, 768)     0           Embedding-Position[0][0]         \n",
            "__________________________________________________________________________________________________\n",
            "Embedding-Norm (LayerNormalizat (None, 384, 768)     1536        Embedding-Dropout[0][0]          \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-1-MultiHeadSelfAttentio (None, 384, 768)     2362368     Embedding-Norm[0][0]             \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-1-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-1-MultiHeadSelfAttention[\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-1-MultiHeadSelfAttentio (None, 384, 768)     0           Embedding-Norm[0][0]             \n",
            "                                                                 Encoder-1-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-1-MultiHeadSelfAttentio (None, 384, 768)     1536        Encoder-1-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-1-FeedForward (FeedForw (None, 384, 768)     4722432     Encoder-1-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-1-FeedForward-Dropout ( (None, 384, 768)     0           Encoder-1-FeedForward[0][0]      \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-1-FeedForward-Add (Add) (None, 384, 768)     0           Encoder-1-MultiHeadSelfAttention-\n",
            "                                                                 Encoder-1-FeedForward-Dropout[0][\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-1-FeedForward-Norm (Lay (None, 384, 768)     1536        Encoder-1-FeedForward-Add[0][0]  \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-2-MultiHeadSelfAttentio (None, 384, 768)     2362368     Encoder-1-FeedForward-Norm[0][0] \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-2-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-2-MultiHeadSelfAttention[\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-2-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-1-FeedForward-Norm[0][0] \n",
            "                                                                 Encoder-2-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-2-MultiHeadSelfAttentio (None, 384, 768)     1536        Encoder-2-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-2-FeedForward (FeedForw (None, 384, 768)     4722432     Encoder-2-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-2-FeedForward-Dropout ( (None, 384, 768)     0           Encoder-2-FeedForward[0][0]      \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-2-FeedForward-Add (Add) (None, 384, 768)     0           Encoder-2-MultiHeadSelfAttention-\n",
            "                                                                 Encoder-2-FeedForward-Dropout[0][\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-2-FeedForward-Norm (Lay (None, 384, 768)     1536        Encoder-2-FeedForward-Add[0][0]  \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-3-MultiHeadSelfAttentio (None, 384, 768)     2362368     Encoder-2-FeedForward-Norm[0][0] \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-3-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-3-MultiHeadSelfAttention[\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-3-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-2-FeedForward-Norm[0][0] \n",
            "                                                                 Encoder-3-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-3-MultiHeadSelfAttentio (None, 384, 768)     1536        Encoder-3-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-3-FeedForward (FeedForw (None, 384, 768)     4722432     Encoder-3-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-3-FeedForward-Dropout ( (None, 384, 768)     0           Encoder-3-FeedForward[0][0]      \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-3-FeedForward-Add (Add) (None, 384, 768)     0           Encoder-3-MultiHeadSelfAttention-\n",
            "                                                                 Encoder-3-FeedForward-Dropout[0][\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-3-FeedForward-Norm (Lay (None, 384, 768)     1536        Encoder-3-FeedForward-Add[0][0]  \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-4-MultiHeadSelfAttentio (None, 384, 768)     2362368     Encoder-3-FeedForward-Norm[0][0] \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-4-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-4-MultiHeadSelfAttention[\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-4-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-3-FeedForward-Norm[0][0] \n",
            "                                                                 Encoder-4-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-4-MultiHeadSelfAttentio (None, 384, 768)     1536        Encoder-4-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-4-FeedForward (FeedForw (None, 384, 768)     4722432     Encoder-4-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-4-FeedForward-Dropout ( (None, 384, 768)     0           Encoder-4-FeedForward[0][0]      \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-4-FeedForward-Add (Add) (None, 384, 768)     0           Encoder-4-MultiHeadSelfAttention-\n",
            "                                                                 Encoder-4-FeedForward-Dropout[0][\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-4-FeedForward-Norm (Lay (None, 384, 768)     1536        Encoder-4-FeedForward-Add[0][0]  \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-5-MultiHeadSelfAttentio (None, 384, 768)     2362368     Encoder-4-FeedForward-Norm[0][0] \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-5-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-5-MultiHeadSelfAttention[\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-5-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-4-FeedForward-Norm[0][0] \n",
            "                                                                 Encoder-5-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-5-MultiHeadSelfAttentio (None, 384, 768)     1536        Encoder-5-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-5-FeedForward (FeedForw (None, 384, 768)     4722432     Encoder-5-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-5-FeedForward-Dropout ( (None, 384, 768)     0           Encoder-5-FeedForward[0][0]      \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-5-FeedForward-Add (Add) (None, 384, 768)     0           Encoder-5-MultiHeadSelfAttention-\n",
            "                                                                 Encoder-5-FeedForward-Dropout[0][\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-5-FeedForward-Norm (Lay (None, 384, 768)     1536        Encoder-5-FeedForward-Add[0][0]  \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-6-MultiHeadSelfAttentio (None, 384, 768)     2362368     Encoder-5-FeedForward-Norm[0][0] \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-6-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-6-MultiHeadSelfAttention[\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-6-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-5-FeedForward-Norm[0][0] \n",
            "                                                                 Encoder-6-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-6-MultiHeadSelfAttentio (None, 384, 768)     1536        Encoder-6-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-6-FeedForward (FeedForw (None, 384, 768)     4722432     Encoder-6-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-6-FeedForward-Dropout ( (None, 384, 768)     0           Encoder-6-FeedForward[0][0]      \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-6-FeedForward-Add (Add) (None, 384, 768)     0           Encoder-6-MultiHeadSelfAttention-\n",
            "                                                                 Encoder-6-FeedForward-Dropout[0][\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-6-FeedForward-Norm (Lay (None, 384, 768)     1536        Encoder-6-FeedForward-Add[0][0]  \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-7-MultiHeadSelfAttentio (None, 384, 768)     2362368     Encoder-6-FeedForward-Norm[0][0] \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-7-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-7-MultiHeadSelfAttention[\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-7-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-6-FeedForward-Norm[0][0] \n",
            "                                                                 Encoder-7-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-7-MultiHeadSelfAttentio (None, 384, 768)     1536        Encoder-7-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-7-FeedForward (FeedForw (None, 384, 768)     4722432     Encoder-7-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-7-FeedForward-Dropout ( (None, 384, 768)     0           Encoder-7-FeedForward[0][0]      \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-7-FeedForward-Add (Add) (None, 384, 768)     0           Encoder-7-MultiHeadSelfAttention-\n",
            "                                                                 Encoder-7-FeedForward-Dropout[0][\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-7-FeedForward-Norm (Lay (None, 384, 768)     1536        Encoder-7-FeedForward-Add[0][0]  \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-8-MultiHeadSelfAttentio (None, 384, 768)     2362368     Encoder-7-FeedForward-Norm[0][0] \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-8-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-8-MultiHeadSelfAttention[\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-8-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-7-FeedForward-Norm[0][0] \n",
            "                                                                 Encoder-8-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-8-MultiHeadSelfAttentio (None, 384, 768)     1536        Encoder-8-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-8-FeedForward (FeedForw (None, 384, 768)     4722432     Encoder-8-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-8-FeedForward-Dropout ( (None, 384, 768)     0           Encoder-8-FeedForward[0][0]      \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-8-FeedForward-Add (Add) (None, 384, 768)     0           Encoder-8-MultiHeadSelfAttention-\n",
            "                                                                 Encoder-8-FeedForward-Dropout[0][\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-8-FeedForward-Norm (Lay (None, 384, 768)     1536        Encoder-8-FeedForward-Add[0][0]  \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-9-MultiHeadSelfAttentio (None, 384, 768)     2362368     Encoder-8-FeedForward-Norm[0][0] \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-9-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-9-MultiHeadSelfAttention[\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-9-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-8-FeedForward-Norm[0][0] \n",
            "                                                                 Encoder-9-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-9-MultiHeadSelfAttentio (None, 384, 768)     1536        Encoder-9-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-9-FeedForward (FeedForw (None, 384, 768)     4722432     Encoder-9-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-9-FeedForward-Dropout ( (None, 384, 768)     0           Encoder-9-FeedForward[0][0]      \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-9-FeedForward-Add (Add) (None, 384, 768)     0           Encoder-9-MultiHeadSelfAttention-\n",
            "                                                                 Encoder-9-FeedForward-Dropout[0][\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-9-FeedForward-Norm (Lay (None, 384, 768)     1536        Encoder-9-FeedForward-Add[0][0]  \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-10-MultiHeadSelfAttenti (None, 384, 768)     2362368     Encoder-9-FeedForward-Norm[0][0] \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-10-MultiHeadSelfAttenti (None, 384, 768)     0           Encoder-10-MultiHeadSelfAttention\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-10-MultiHeadSelfAttenti (None, 384, 768)     0           Encoder-9-FeedForward-Norm[0][0] \n",
            "                                                                 Encoder-10-MultiHeadSelfAttention\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-10-MultiHeadSelfAttenti (None, 384, 768)     1536        Encoder-10-MultiHeadSelfAttention\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-10-FeedForward (FeedFor (None, 384, 768)     4722432     Encoder-10-MultiHeadSelfAttention\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-10-FeedForward-Dropout  (None, 384, 768)     0           Encoder-10-FeedForward[0][0]     \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-10-FeedForward-Add (Add (None, 384, 768)     0           Encoder-10-MultiHeadSelfAttention\n",
            "                                                                 Encoder-10-FeedForward-Dropout[0]\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-10-FeedForward-Norm (La (None, 384, 768)     1536        Encoder-10-FeedForward-Add[0][0] \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-11-MultiHeadSelfAttenti (None, 384, 768)     2362368     Encoder-10-FeedForward-Norm[0][0]\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-11-MultiHeadSelfAttenti (None, 384, 768)     0           Encoder-11-MultiHeadSelfAttention\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-11-MultiHeadSelfAttenti (None, 384, 768)     0           Encoder-10-FeedForward-Norm[0][0]\n",
            "                                                                 Encoder-11-MultiHeadSelfAttention\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-11-MultiHeadSelfAttenti (None, 384, 768)     1536        Encoder-11-MultiHeadSelfAttention\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-11-FeedForward (FeedFor (None, 384, 768)     4722432     Encoder-11-MultiHeadSelfAttention\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-11-FeedForward-Dropout  (None, 384, 768)     0           Encoder-11-FeedForward[0][0]     \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-11-FeedForward-Add (Add (None, 384, 768)     0           Encoder-11-MultiHeadSelfAttention\n",
            "                                                                 Encoder-11-FeedForward-Dropout[0]\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-11-FeedForward-Norm (La (None, 384, 768)     1536        Encoder-11-FeedForward-Add[0][0] \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-12-MultiHeadSelfAttenti (None, 384, 768)     2362368     Encoder-11-FeedForward-Norm[0][0]\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-12-MultiHeadSelfAttenti (None, 384, 768)     0           Encoder-12-MultiHeadSelfAttention\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-12-MultiHeadSelfAttenti (None, 384, 768)     0           Encoder-11-FeedForward-Norm[0][0]\n",
            "                                                                 Encoder-12-MultiHeadSelfAttention\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-12-MultiHeadSelfAttenti (None, 384, 768)     1536        Encoder-12-MultiHeadSelfAttention\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-12-FeedForward (FeedFor (None, 384, 768)     4722432     Encoder-12-MultiHeadSelfAttention\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-12-FeedForward-Dropout  (None, 384, 768)     0           Encoder-12-FeedForward[0][0]     \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-12-FeedForward-Add (Add (None, 384, 768)     0           Encoder-12-MultiHeadSelfAttention\n",
            "                                                                 Encoder-12-FeedForward-Dropout[0]\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-12-FeedForward-Norm (La (None, 384, 768)     1536        Encoder-12-FeedForward-Add[0][0] \n",
            "__________________________________________________________________________________________________\n",
            "non_masking_2 (NonMasking)      (None, 384, 768)     0           Encoder-12-FeedForward-Norm[0][0]\n",
            "__________________________________________________________________________________________________\n",
            "my_layer__start_2 (MyLayer_Star (None, 384)          1536        non_masking_2[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "my_layer__end_2 (MyLayer_End)   (None, 384)          1536        non_masking_2[0][0]              \n",
            "==================================================================================================\n",
            "Total params: 177,167,616\n",
            "Trainable params: 177,167,616\n",
            "Non-trainable params: 0\n",
            "__________________________________________________________________________________________________\n",
            "Model: \"model_4\"\n",
            "__________________________________________________________________________________________________\n",
            "Layer (type)                    Output Shape         Param #     Connected to                     \n",
            "==================================================================================================\n",
            "Input-Token (InputLayer)        (None, 384)          0                                            \n",
            "__________________________________________________________________________________________________\n",
            "Input-Segment (InputLayer)      (None, 384)          0                                            \n",
            "__________________________________________________________________________________________________\n",
            "Embedding-Token (TokenEmbedding [(None, 384, 768), ( 91812096    Input-Token[0][0]                \n",
            "__________________________________________________________________________________________________\n",
            "Embedding-Segment (Embedding)   (None, 384, 768)     1536        Input-Segment[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "Embedding-Token-Segment (Add)   (None, 384, 768)     0           Embedding-Token[0][0]            \n",
            "                                                                 Embedding-Segment[0][0]          \n",
            "__________________________________________________________________________________________________\n",
            "Embedding-Position (PositionEmb (None, 384, 768)     294912      Embedding-Token-Segment[0][0]    \n",
            "__________________________________________________________________________________________________\n",
            "Embedding-Dropout (Dropout)     (None, 384, 768)     0           Embedding-Position[0][0]         \n",
            "__________________________________________________________________________________________________\n",
            "Embedding-Norm (LayerNormalizat (None, 384, 768)     1536        Embedding-Dropout[0][0]          \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-1-MultiHeadSelfAttentio (None, 384, 768)     2362368     Embedding-Norm[0][0]             \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-1-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-1-MultiHeadSelfAttention[\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-1-MultiHeadSelfAttentio (None, 384, 768)     0           Embedding-Norm[0][0]             \n",
            "                                                                 Encoder-1-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-1-MultiHeadSelfAttentio (None, 384, 768)     1536        Encoder-1-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-1-FeedForward (FeedForw (None, 384, 768)     4722432     Encoder-1-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-1-FeedForward-Dropout ( (None, 384, 768)     0           Encoder-1-FeedForward[0][0]      \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-1-FeedForward-Add (Add) (None, 384, 768)     0           Encoder-1-MultiHeadSelfAttention-\n",
            "                                                                 Encoder-1-FeedForward-Dropout[0][\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-1-FeedForward-Norm (Lay (None, 384, 768)     1536        Encoder-1-FeedForward-Add[0][0]  \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-2-MultiHeadSelfAttentio (None, 384, 768)     2362368     Encoder-1-FeedForward-Norm[0][0] \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-2-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-2-MultiHeadSelfAttention[\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-2-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-1-FeedForward-Norm[0][0] \n",
            "                                                                 Encoder-2-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-2-MultiHeadSelfAttentio (None, 384, 768)     1536        Encoder-2-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-2-FeedForward (FeedForw (None, 384, 768)     4722432     Encoder-2-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-2-FeedForward-Dropout ( (None, 384, 768)     0           Encoder-2-FeedForward[0][0]      \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-2-FeedForward-Add (Add) (None, 384, 768)     0           Encoder-2-MultiHeadSelfAttention-\n",
            "                                                                 Encoder-2-FeedForward-Dropout[0][\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-2-FeedForward-Norm (Lay (None, 384, 768)     1536        Encoder-2-FeedForward-Add[0][0]  \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-3-MultiHeadSelfAttentio (None, 384, 768)     2362368     Encoder-2-FeedForward-Norm[0][0] \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-3-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-3-MultiHeadSelfAttention[\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-3-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-2-FeedForward-Norm[0][0] \n",
            "                                                                 Encoder-3-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-3-MultiHeadSelfAttentio (None, 384, 768)     1536        Encoder-3-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-3-FeedForward (FeedForw (None, 384, 768)     4722432     Encoder-3-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-3-FeedForward-Dropout ( (None, 384, 768)     0           Encoder-3-FeedForward[0][0]      \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-3-FeedForward-Add (Add) (None, 384, 768)     0           Encoder-3-MultiHeadSelfAttention-\n",
            "                                                                 Encoder-3-FeedForward-Dropout[0][\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-3-FeedForward-Norm (Lay (None, 384, 768)     1536        Encoder-3-FeedForward-Add[0][0]  \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-4-MultiHeadSelfAttentio (None, 384, 768)     2362368     Encoder-3-FeedForward-Norm[0][0] \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-4-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-4-MultiHeadSelfAttention[\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-4-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-3-FeedForward-Norm[0][0] \n",
            "                                                                 Encoder-4-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-4-MultiHeadSelfAttentio (None, 384, 768)     1536        Encoder-4-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-4-FeedForward (FeedForw (None, 384, 768)     4722432     Encoder-4-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-4-FeedForward-Dropout ( (None, 384, 768)     0           Encoder-4-FeedForward[0][0]      \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-4-FeedForward-Add (Add) (None, 384, 768)     0           Encoder-4-MultiHeadSelfAttention-\n",
            "                                                                 Encoder-4-FeedForward-Dropout[0][\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-4-FeedForward-Norm (Lay (None, 384, 768)     1536        Encoder-4-FeedForward-Add[0][0]  \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-5-MultiHeadSelfAttentio (None, 384, 768)     2362368     Encoder-4-FeedForward-Norm[0][0] \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-5-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-5-MultiHeadSelfAttention[\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-5-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-4-FeedForward-Norm[0][0] \n",
            "                                                                 Encoder-5-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-5-MultiHeadSelfAttentio (None, 384, 768)     1536        Encoder-5-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-5-FeedForward (FeedForw (None, 384, 768)     4722432     Encoder-5-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-5-FeedForward-Dropout ( (None, 384, 768)     0           Encoder-5-FeedForward[0][0]      \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-5-FeedForward-Add (Add) (None, 384, 768)     0           Encoder-5-MultiHeadSelfAttention-\n",
            "                                                                 Encoder-5-FeedForward-Dropout[0][\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-5-FeedForward-Norm (Lay (None, 384, 768)     1536        Encoder-5-FeedForward-Add[0][0]  \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-6-MultiHeadSelfAttentio (None, 384, 768)     2362368     Encoder-5-FeedForward-Norm[0][0] \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-6-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-6-MultiHeadSelfAttention[\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-6-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-5-FeedForward-Norm[0][0] \n",
            "                                                                 Encoder-6-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-6-MultiHeadSelfAttentio (None, 384, 768)     1536        Encoder-6-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-6-FeedForward (FeedForw (None, 384, 768)     4722432     Encoder-6-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-6-FeedForward-Dropout ( (None, 384, 768)     0           Encoder-6-FeedForward[0][0]      \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-6-FeedForward-Add (Add) (None, 384, 768)     0           Encoder-6-MultiHeadSelfAttention-\n",
            "                                                                 Encoder-6-FeedForward-Dropout[0][\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-6-FeedForward-Norm (Lay (None, 384, 768)     1536        Encoder-6-FeedForward-Add[0][0]  \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-7-MultiHeadSelfAttentio (None, 384, 768)     2362368     Encoder-6-FeedForward-Norm[0][0] \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-7-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-7-MultiHeadSelfAttention[\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-7-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-6-FeedForward-Norm[0][0] \n",
            "                                                                 Encoder-7-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-7-MultiHeadSelfAttentio (None, 384, 768)     1536        Encoder-7-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-7-FeedForward (FeedForw (None, 384, 768)     4722432     Encoder-7-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-7-FeedForward-Dropout ( (None, 384, 768)     0           Encoder-7-FeedForward[0][0]      \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-7-FeedForward-Add (Add) (None, 384, 768)     0           Encoder-7-MultiHeadSelfAttention-\n",
            "                                                                 Encoder-7-FeedForward-Dropout[0][\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-7-FeedForward-Norm (Lay (None, 384, 768)     1536        Encoder-7-FeedForward-Add[0][0]  \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-8-MultiHeadSelfAttentio (None, 384, 768)     2362368     Encoder-7-FeedForward-Norm[0][0] \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-8-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-8-MultiHeadSelfAttention[\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-8-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-7-FeedForward-Norm[0][0] \n",
            "                                                                 Encoder-8-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-8-MultiHeadSelfAttentio (None, 384, 768)     1536        Encoder-8-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-8-FeedForward (FeedForw (None, 384, 768)     4722432     Encoder-8-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-8-FeedForward-Dropout ( (None, 384, 768)     0           Encoder-8-FeedForward[0][0]      \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-8-FeedForward-Add (Add) (None, 384, 768)     0           Encoder-8-MultiHeadSelfAttention-\n",
            "                                                                 Encoder-8-FeedForward-Dropout[0][\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-8-FeedForward-Norm (Lay (None, 384, 768)     1536        Encoder-8-FeedForward-Add[0][0]  \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-9-MultiHeadSelfAttentio (None, 384, 768)     2362368     Encoder-8-FeedForward-Norm[0][0] \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-9-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-9-MultiHeadSelfAttention[\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-9-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-8-FeedForward-Norm[0][0] \n",
            "                                                                 Encoder-9-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-9-MultiHeadSelfAttentio (None, 384, 768)     1536        Encoder-9-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-9-FeedForward (FeedForw (None, 384, 768)     4722432     Encoder-9-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-9-FeedForward-Dropout ( (None, 384, 768)     0           Encoder-9-FeedForward[0][0]      \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-9-FeedForward-Add (Add) (None, 384, 768)     0           Encoder-9-MultiHeadSelfAttention-\n",
            "                                                                 Encoder-9-FeedForward-Dropout[0][\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-9-FeedForward-Norm (Lay (None, 384, 768)     1536        Encoder-9-FeedForward-Add[0][0]  \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-10-MultiHeadSelfAttenti (None, 384, 768)     2362368     Encoder-9-FeedForward-Norm[0][0] \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-10-MultiHeadSelfAttenti (None, 384, 768)     0           Encoder-10-MultiHeadSelfAttention\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-10-MultiHeadSelfAttenti (None, 384, 768)     0           Encoder-9-FeedForward-Norm[0][0] \n",
            "                                                                 Encoder-10-MultiHeadSelfAttention\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-10-MultiHeadSelfAttenti (None, 384, 768)     1536        Encoder-10-MultiHeadSelfAttention\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-10-FeedForward (FeedFor (None, 384, 768)     4722432     Encoder-10-MultiHeadSelfAttention\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-10-FeedForward-Dropout  (None, 384, 768)     0           Encoder-10-FeedForward[0][0]     \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-10-FeedForward-Add (Add (None, 384, 768)     0           Encoder-10-MultiHeadSelfAttention\n",
            "                                                                 Encoder-10-FeedForward-Dropout[0]\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-10-FeedForward-Norm (La (None, 384, 768)     1536        Encoder-10-FeedForward-Add[0][0] \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-11-MultiHeadSelfAttenti (None, 384, 768)     2362368     Encoder-10-FeedForward-Norm[0][0]\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-11-MultiHeadSelfAttenti (None, 384, 768)     0           Encoder-11-MultiHeadSelfAttention\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-11-MultiHeadSelfAttenti (None, 384, 768)     0           Encoder-10-FeedForward-Norm[0][0]\n",
            "                                                                 Encoder-11-MultiHeadSelfAttention\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-11-MultiHeadSelfAttenti (None, 384, 768)     1536        Encoder-11-MultiHeadSelfAttention\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-11-FeedForward (FeedFor (None, 384, 768)     4722432     Encoder-11-MultiHeadSelfAttention\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-11-FeedForward-Dropout  (None, 384, 768)     0           Encoder-11-FeedForward[0][0]     \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-11-FeedForward-Add (Add (None, 384, 768)     0           Encoder-11-MultiHeadSelfAttention\n",
            "                                                                 Encoder-11-FeedForward-Dropout[0]\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-11-FeedForward-Norm (La (None, 384, 768)     1536        Encoder-11-FeedForward-Add[0][0] \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-12-MultiHeadSelfAttenti (None, 384, 768)     2362368     Encoder-11-FeedForward-Norm[0][0]\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-12-MultiHeadSelfAttenti (None, 384, 768)     0           Encoder-12-MultiHeadSelfAttention\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-12-MultiHeadSelfAttenti (None, 384, 768)     0           Encoder-11-FeedForward-Norm[0][0]\n",
            "                                                                 Encoder-12-MultiHeadSelfAttention\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-12-MultiHeadSelfAttenti (None, 384, 768)     1536        Encoder-12-MultiHeadSelfAttention\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-12-FeedForward (FeedFor (None, 384, 768)     4722432     Encoder-12-MultiHeadSelfAttention\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-12-FeedForward-Dropout  (None, 384, 768)     0           Encoder-12-FeedForward[0][0]     \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-12-FeedForward-Add (Add (None, 384, 768)     0           Encoder-12-MultiHeadSelfAttention\n",
            "                                                                 Encoder-12-FeedForward-Dropout[0]\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-12-FeedForward-Norm (La (None, 384, 768)     1536        Encoder-12-FeedForward-Add[0][0] \n",
            "__________________________________________________________________________________________________\n",
            "non_masking_2 (NonMasking)      (None, 384, 768)     0           Encoder-12-FeedForward-Norm[0][0]\n",
            "__________________________________________________________________________________________________\n",
            "my_layer__start_2 (MyLayer_Star (None, 384)          1536        non_masking_2[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "my_layer__end_2 (MyLayer_End)   (None, 384)          1536        non_masking_2[0][0]              \n",
            "==================================================================================================\n",
            "Total params: 177,167,616\n",
            "Trainable params: 177,167,616\n",
            "Non-trainable params: 0\n",
            "__________________________________________________________________________________________________\n",
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:1033: The name tf.assign_add is deprecated. Please use tf.compat.v1.assign_add instead.\n",
            "\n",
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:1020: The name tf.assign is deprecated. Please use tf.compat.v1.assign instead.\n",
            "\n",
            "Train on 82919 samples, validate on 4365 samples\n",
            "Epoch 1/1\n",
            "82919/82919 [==============================] - 5002s 60ms/step - loss: 1.6119 - my_layer__start_2_loss: 0.8597 - my_layer__end_2_loss: 0.7523 - my_layer__start_2_acc: 0.7320 - my_layer__end_2_acc: 0.7735 - val_loss: 1.7590 - val_my_layer__start_2_loss: 0.9819 - val_my_layer__end_2_loss: 0.7771 - val_my_layer__start_2_acc: 0.6948 - val_my_layer__end_2_acc: 0.7677\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RSKZXlpFox6_",
        "colab_type": "text"
      },
      "source": [
        "g드라이브상에 squad 모델을 저장할 경로 지정"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qhW2hIzQoOtf",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "path = \"gdrive/My Drive/Colab Notebooks/squad\""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xC96iEmbhKZM",
        "colab_type": "text"
      },
      "source": [
        "BERT MODEL을 저장합니다."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "E3pc5zhjCet-",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "bert_model.save_weights(path+\"/squad_wordpiece.h5\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uHmMOi6bhPxm",
        "colab_type": "text"
      },
      "source": [
        "버트 모형을 다시 훈련합니다.  \n",
        "이번에는 validation_split을 입력하지 않아서 전체 데이터가 훈련 되도록 만들어 줍니다."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "98LafkcmWAc6",
        "colab_type": "code",
        "outputId": "fa66b7fd-f203-4c92-cc9b-67ee637c0628",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 336
        }
      },
      "source": [
        "bert_model.compile(optimizer=RAdam(learning_rate=0.00003, decay=0.0001), loss='categorical_crossentropy', metrics=['accuracy'])\n",
        "bert_model.fit(train_x, train_y, batch_size=10, shuffle=False, verbose=1)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/math_grad.py:1424: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use tf.where in 2.0, which has the same broadcast rule as np.where\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/math_grad.py:1424: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use tf.where in 2.0, which has the same broadcast rule as np.where\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:1033: The name tf.assign_add is deprecated. Please use tf.compat.v1.assign_add instead.\n",
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:1033: The name tf.assign_add is deprecated. Please use tf.compat.v1.assign_add instead.\n",
            "\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:1020: The name tf.assign is deprecated. Please use tf.compat.v1.assign instead.\n",
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:1020: The name tf.assign is deprecated. Please use tf.compat.v1.assign instead.\n",
            "\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/1\n",
            "87284/87284 [==============================] - 9501s 109ms/step - loss: 1.0188 - my_layer__start_1_loss: 0.5568 - my_layer__end_1_loss: 0.4620 - my_layer__start_1_acc: 0.8083 - my_layer__end_1_acc: 0.8504\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<keras.callbacks.History at 0x7f88b234eb70>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 97
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "h0xs7diprDw1",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "bert_model.save_weights(path+\"/squad_wordpiece_2.h5\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "154p7E_ihh2b",
        "colab_type": "text"
      },
      "source": [
        "버트 모형을 한번 더 훈련시켜 줍니다. learning_rate을 0.00001로 바꿔 줍니다."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qnZ0bdoXCJVr",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "bert_model.compile(optimizer=RAdam(learning_rate=0.00001, decay=0.0001), loss='categorical_crossentropy', metrics=['accuracy'])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WkGimQRChdMZ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "bert_model.save_weights(path+\"/squad_wordpiece_3.h5\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e-HySrkKNV_e",
        "colab_type": "text"
      },
      "source": [
        "재사용을 위해 bert_model을 지드라이브에 저장해줍니다."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f0ZgfR6MRIul",
        "colab_type": "text"
      },
      "source": [
        "버트 모형을 로드해줍니다. 이미 로드하였던 모델에 계수들만 살짝 얹혀 줍니다."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DwMLCYp0TzTv",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "bert_model = get_bert_finetuning_model(model)\n",
        "bert_model.load_weights(path+\"/squad_wordpiece_3.h5\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YeXe8E-0i1si",
        "colab_type": "text"
      },
      "source": [
        "Test data set에 대한 bert_input을 만들어 줍니다.  \n",
        "Train data set과는 다르게 label을 생성하지 않습니다."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "q_WdnjmMYZ-N",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def convert_pred_data(question, doc):\n",
        "    global tokenizer\n",
        "    indices, segments = [], []\n",
        "    ids, segment = tokenizer.encode(question, doc, max_len=SEQ_LEN)\n",
        "    indices.append(ids)\n",
        "    segments.append(segment)\n",
        "    indices_x = np.array(indices)\n",
        "    segments = np.array(segments)\n",
        "    return [indices_x, segments]\n",
        "\n",
        "def load_pred_data(question, doc):\n",
        "    data_x = convert_pred_data(question, doc)\n",
        "    return data_x"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "T4LmVp-5lOnR",
        "colab_type": "text"
      },
      "source": [
        "질문과 문장을 받아 답을 알려주는 함수를 정의합니다."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Engs2SIlW0SP",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def predict_letter(question, doc):\n",
        "  \n",
        "  test_input = load_pred_data(question, doc)\n",
        "  test_start, test_end = bert_model.predict(test_input)\n",
        "  \n",
        "  indexes = tokenizer.encode(question, doc, max_len=SEQ_LEN)[0]\n",
        "  start = np.argmax(test_start, axis=1).item()\n",
        "  end = np.argmax(test_end, axis=1).item()\n",
        "  start_tok = indexes[start]\n",
        "  end_tok = indexes[end]\n",
        "  print(\"Question : \", question)\n",
        "  \n",
        "  print(\"-\"*50)\n",
        "  print(\"Context : \", end = \" \")\n",
        "  \n",
        "  def split_text(text, n):\n",
        "    for line in text.splitlines():\n",
        "        while len(line) > n:\n",
        "           x, line = line[:n], line[n:]\n",
        "           yield x\n",
        "        yield line\n",
        "\n",
        "  \n",
        "\n",
        "  for line in split_text(doc, 150):\n",
        "    print(line)\n",
        "\n",
        "  print(\"-\"*50)\n",
        "  print(\"ANSWER : \", end = \" \")\n",
        "  print(\"\\n\")\n",
        "  sentences = []\n",
        "  \n",
        "  for i in range(start, end+1):\n",
        "    token_based_word = reverse_token_dict[indexes[i]]\n",
        "    sentences.append(token_based_word)\n",
        "    print(token_based_word, end= \" \")\n",
        "  \n",
        "  print(\"\\n\")\n",
        "  print(\"Untokenized Answer : \", end = \"\")\n",
        "  for w in sentences:\n",
        "    if w.startswith(\"##\"):\n",
        "      w = w.replace(\"##\", \"\")\n",
        "    else:\n",
        "      w = \" \" + w\n",
        "    \n",
        "    print(w, end=\"\")\n",
        "  print(\"\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YBRodQAqlXf7",
        "colab_type": "text"
      },
      "source": [
        "SQAUD 데이터 셋에서 test 용도로 쓰이는 dev 파일을 PANDAS DATAFRAME 형식으로 불러오는 함수를 정의합니다.  \n",
        "train 데이터와 모양이 약간 다르기 때문에, 함수를 새로 정의해야 합니다."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6FpWAQR1ZqGt",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def squad_json_to_dataframe_dev(input_file_path, record_path = ['data','paragraphs','qas','answers'],\n",
        "                           verbose = 1):\n",
        "    \"\"\"\n",
        "    input_file_path: path to the squad json file.\n",
        "    record_path: path to deepest level in json file default value is\n",
        "    ['data','paragraphs','qas','answers']\n",
        "    verbose: 0 to suppress it default is 1\n",
        "    \"\"\"\n",
        "    if verbose:\n",
        "        print(\"Reading the json file\")    \n",
        "    file = json.loads(open(input_file_path).read())\n",
        "    if verbose:\n",
        "        print(\"processing...\")\n",
        "    # parsing different level's in the json file\n",
        "    js = pd.io.json.json_normalize(file , record_path )\n",
        "    m = pd.io.json.json_normalize(file, record_path[:-1] )\n",
        "    r = pd.io.json.json_normalize(file,record_path[:-2])\n",
        "    \n",
        "    #combining it into single dataframe\n",
        "    idx = np.repeat(r['context'].values, r.qas.str.len())\n",
        "    m['context'] = idx\n",
        "    main = m[['id','question','context','answers']].set_index('id').reset_index()\n",
        "    main['c_id'] = main['context'].factorize()[0]\n",
        "    if verbose:\n",
        "        print(\"shape of the dataframe is {}\".format(main.shape))\n",
        "        print(\"Done\")\n",
        "    return main"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qMVFr75nZq9G",
        "colab_type": "code",
        "outputId": "98dd7db3-a3ee-4cc0-ae74-9a6d071ca41a",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 87
        }
      },
      "source": [
        "input_file_path ='dev-v1.1.json'\n",
        "record_path = ['data','paragraphs','qas','answers']\n",
        "verbose = 0\n",
        "dev = squad_json_to_dataframe_dev(input_file_path=input_file_path,record_path=record_path)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Reading the json file\n",
            "processing...\n",
            "shape of the dataframe is (10570, 5)\n",
            "Done\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1QSzFOnPmv4g",
        "colab_type": "text"
      },
      "source": [
        "TEST DATA가 잘 불려왔는지 확인해 보겠습니다."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tluhycV8ZxqF",
        "colab_type": "code",
        "outputId": "3b554d9d-82e6-4062-d762-f6b14e204169",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 402
        }
      },
      "source": [
        "dev"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>id</th>\n",
              "      <th>question</th>\n",
              "      <th>context</th>\n",
              "      <th>answers</th>\n",
              "      <th>c_id</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>56be4db0acb8001400a502ec</td>\n",
              "      <td>Which NFL team represented the AFC at Super Bo...</td>\n",
              "      <td>Super Bowl 50 was an American football game to...</td>\n",
              "      <td>[{'answer_start': 177, 'text': 'Denver Broncos...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>56be4db0acb8001400a502ed</td>\n",
              "      <td>Which NFL team represented the NFC at Super Bo...</td>\n",
              "      <td>Super Bowl 50 was an American football game to...</td>\n",
              "      <td>[{'answer_start': 249, 'text': 'Carolina Panth...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>56be4db0acb8001400a502ee</td>\n",
              "      <td>Where did Super Bowl 50 take place?</td>\n",
              "      <td>Super Bowl 50 was an American football game to...</td>\n",
              "      <td>[{'answer_start': 403, 'text': 'Santa Clara, C...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>56be4db0acb8001400a502ef</td>\n",
              "      <td>Which NFL team won Super Bowl 50?</td>\n",
              "      <td>Super Bowl 50 was an American football game to...</td>\n",
              "      <td>[{'answer_start': 177, 'text': 'Denver Broncos...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>56be4db0acb8001400a502f0</td>\n",
              "      <td>What color was used to emphasize the 50th anni...</td>\n",
              "      <td>Super Bowl 50 was an American football game to...</td>\n",
              "      <td>[{'answer_start': 488, 'text': 'gold'}, {'answ...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10565</th>\n",
              "      <td>5737aafd1c456719005744fb</td>\n",
              "      <td>What is the metric term less used than the New...</td>\n",
              "      <td>The pound-force has a metric counterpart, less...</td>\n",
              "      <td>[{'answer_start': 82, 'text': 'kilogram-force'...</td>\n",
              "      <td>2066</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10566</th>\n",
              "      <td>5737aafd1c456719005744fc</td>\n",
              "      <td>What is the kilogram-force sometimes reffered ...</td>\n",
              "      <td>The pound-force has a metric counterpart, less...</td>\n",
              "      <td>[{'answer_start': 114, 'text': 'kilopond'}, {'...</td>\n",
              "      <td>2066</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10567</th>\n",
              "      <td>5737aafd1c456719005744fd</td>\n",
              "      <td>What is a very seldom used unit of mass in the...</td>\n",
              "      <td>The pound-force has a metric counterpart, less...</td>\n",
              "      <td>[{'answer_start': 274, 'text': 'slug'}, {'answ...</td>\n",
              "      <td>2066</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10568</th>\n",
              "      <td>5737aafd1c456719005744fe</td>\n",
              "      <td>What seldom used term of a unit of force equal...</td>\n",
              "      <td>The pound-force has a metric counterpart, less...</td>\n",
              "      <td>[{'answer_start': 712, 'text': 'kip'}, {'answe...</td>\n",
              "      <td>2066</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10569</th>\n",
              "      <td>5737aafd1c456719005744ff</td>\n",
              "      <td>What is the seldom used force unit equal to on...</td>\n",
              "      <td>The pound-force has a metric counterpart, less...</td>\n",
              "      <td>[{'answer_start': 665, 'text': 'sthène'}, {'an...</td>\n",
              "      <td>2066</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>10570 rows × 5 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "                             id  ...  c_id\n",
              "0      56be4db0acb8001400a502ec  ...     0\n",
              "1      56be4db0acb8001400a502ed  ...     0\n",
              "2      56be4db0acb8001400a502ee  ...     0\n",
              "3      56be4db0acb8001400a502ef  ...     0\n",
              "4      56be4db0acb8001400a502f0  ...     0\n",
              "...                         ...  ...   ...\n",
              "10565  5737aafd1c456719005744fb  ...  2066\n",
              "10566  5737aafd1c456719005744fc  ...  2066\n",
              "10567  5737aafd1c456719005744fd  ...  2066\n",
              "10568  5737aafd1c456719005744fe  ...  2066\n",
              "10569  5737aafd1c456719005744ff  ...  2066\n",
              "\n",
              "[10570 rows x 5 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 45
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "I9614q2jmcq-",
        "colab_type": "text"
      },
      "source": [
        "테스트 데이터에 대해서 결과를 확인합니다.  \n",
        "훈련에 사용하지 않은 테스트 데이터에 대한 예측을 제법 잘 수행하는 것을 보실 수 있겠습니다."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IBmiDNInax0z",
        "colab_type": "code",
        "outputId": "8231fcf5-1cee-44a0-94f7-3b114288d3f0",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "import random\n",
        "for i in random.sample(range(100),100):\n",
        "  doc = dev['context'][i]\n",
        "  question = dev['question'][i]\n",
        "  answers = dev['answers'][i]\n",
        "  predict_letter(question, doc)\n",
        "  print(\"\")\n",
        "  print(\"real answer : \", answers)\n",
        "  print(\"\")"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Question :  How many tackles did Von Miller accomlish by himself in the game?\n",
            "--------------------------------------------------\n",
            "Context :  The Broncos took an early lead in Super Bowl 50 and never trailed. Newton was limited by Denver's defense, which sacked him seven times and forced him\n",
            " into three turnovers, including a fumble which they recovered for a touchdown. Denver linebacker Von Miller was named Super Bowl MVP, recording five \n",
            "solo tackles, 2½ sacks, and two forced fumbles.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "five \n",
            "\n",
            "Untokenized Answer :  five\n",
            "\n",
            "real answer :  [{'answer_start': 295, 'text': 'five'}, {'answer_start': 295, 'text': 'five'}, {'answer_start': 295, 'text': 'five'}]\n",
            "\n",
            "Question :  What year did the Carolina Panthers form?\n",
            "--------------------------------------------------\n",
            "Context :  The Panthers finished the regular season with a 15–1 record, and quarterback Cam Newton was named the NFL Most Valuable Player (MVP). They defeated th\n",
            "e Arizona Cardinals 49–15 in the NFC Championship Game and advanced to their second Super Bowl appearance since the franchise was founded in 1995. The\n",
            " Broncos finished the regular season with a 12–4 record, and denied the New England Patriots a chance to defend their title from Super Bowl XLIX by de\n",
            "feating them 20–18 in the AFC Championship Game. They joined the Patriots, Dallas Cowboys, and Pittsburgh Steelers as one of four teams that have made\n",
            " eight appearances in the Super Bowl.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "1995 \n",
            "\n",
            "Untokenized Answer :  1995\n",
            "\n",
            "real answer :  [{'answer_start': 291, 'text': '1995.'}, {'answer_start': 291, 'text': '1995'}, {'answer_start': 291, 'text': '1995'}]\n",
            "\n",
            "Question :  Who won Super Bowl 50?\n",
            "--------------------------------------------------\n",
            "Context :  Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football\n",
            " Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24–10 to earn their third Super B\n",
            "owl title. The game was played on February 7, 2016, at Levi's Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th S\n",
            "uper Bowl, the league emphasized the \"golden anniversary\" with various gold-themed initiatives, as well as temporarily suspending the tradition of nam\n",
            "ing each Super Bowl game with Roman numerals (under which the game would have been known as \"Super Bowl L\"), so that the logo could prominently featur\n",
            "e the Arabic numerals 50.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "den ##ver bron ##cos \n",
            "\n",
            "Untokenized Answer :  denver broncos\n",
            "\n",
            "real answer :  [{'answer_start': 177, 'text': 'Denver Broncos'}, {'answer_start': 177, 'text': 'Denver Broncos'}, {'answer_start': 177, 'text': 'Denver Broncos'}]\n",
            "\n",
            "Question :  Which team won Super Bowl 50.\n",
            "--------------------------------------------------\n",
            "Context :  Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football\n",
            " Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24–10 to earn their third Super B\n",
            "owl title. The game was played on February 7, 2016, at Levi's Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th S\n",
            "uper Bowl, the league emphasized the \"golden anniversary\" with various gold-themed initiatives, as well as temporarily suspending the tradition of nam\n",
            "ing each Super Bowl game with Roman numerals (under which the game would have been known as \"Super Bowl L\"), so that the logo could prominently featur\n",
            "e the Arabic numerals 50.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "den ##ver bron ##cos defeated the national football conference ( n ##f ##c ) champion car ##olina pan ##thers \n",
            "\n",
            "Untokenized Answer :  denver broncos defeated the national football conference ( nfc ) champion carolina panthers\n",
            "\n",
            "real answer :  [{'answer_start': 177, 'text': 'Denver Broncos'}, {'answer_start': 177, 'text': 'Denver Broncos'}, {'answer_start': 177, 'text': 'Denver Broncos'}]\n",
            "\n",
            "Question :  Who did the Panthers beat in the NFC Championship Game?\n",
            "--------------------------------------------------\n",
            "Context :  The Panthers finished the regular season with a 15–1 record, and quarterback Cam Newton was named the NFL Most Valuable Player (MVP). They defeated th\n",
            "e Arizona Cardinals 49–15 in the NFC Championship Game and advanced to their second Super Bowl appearance since the franchise was founded in 1995. The\n",
            " Broncos finished the regular season with a 12–4 record, and denied the New England Patriots a chance to defend their title from Super Bowl XLIX by de\n",
            "feating them 20–18 in the AFC Championship Game. They joined the Patriots, Dallas Cowboys, and Pittsburgh Steelers as one of four teams that have made\n",
            " eight appearances in the Super Bowl.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "ari ##zona cardinal ##s \n",
            "\n",
            "Untokenized Answer :  arizona cardinals\n",
            "\n",
            "real answer :  [{'answer_start': 152, 'text': 'Arizona Cardinals'}, {'answer_start': 148, 'text': 'the Arizona Cardinals'}, {'answer_start': 152, 'text': 'Arizona Cardinals'}]\n",
            "\n",
            "Question :  Who were the defending Super Bowl champions?\n",
            "--------------------------------------------------\n",
            "Context :  The Panthers finished the regular season with a 15–1 record, and quarterback Cam Newton was named the NFL Most Valuable Player (MVP). They defeated th\n",
            "e Arizona Cardinals 49–15 in the NFC Championship Game and advanced to their second Super Bowl appearance since the franchise was founded in 1995. The\n",
            " Broncos finished the regular season with a 12–4 record, and denied the New England Patriots a chance to defend their title from Super Bowl XLIX by de\n",
            "feating them 20–18 in the AFC Championship Game. They joined the Patriots, Dallas Cowboys, and Pittsburgh Steelers as one of four teams that have made\n",
            " eight appearances in the Super Bowl.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "new engl ##and pat ##riot ##s \n",
            "\n",
            "Untokenized Answer :  new england patriots\n",
            "\n",
            "real answer :  [{'answer_start': 372, 'text': 'New England Patriots'}, {'answer_start': 368, 'text': 'the New England Patriots'}, {'answer_start': 372, 'text': 'New England Patriots'}]\n",
            "\n",
            "Question :  How many times did the Denver defense force Newton into turnovers?\n",
            "--------------------------------------------------\n",
            "Context :  The Broncos took an early lead in Super Bowl 50 and never trailed. Newton was limited by Denver's defense, which sacked him seven times and forced him\n",
            " into three turnovers, including a fumble which they recovered for a touchdown. Denver linebacker Von Miller was named Super Bowl MVP, recording five \n",
            "solo tackles, 2½ sacks, and two forced fumbles.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "three \n",
            "\n",
            "Untokenized Answer :  three\n",
            "\n",
            "real answer :  [{'answer_start': 156, 'text': 'three'}, {'answer_start': 156, 'text': 'three'}, {'answer_start': 156, 'text': 'three'}]\n",
            "\n",
            "Question :  How many tackles did Von Miller get during the game?\n",
            "--------------------------------------------------\n",
            "Context :  The Broncos took an early lead in Super Bowl 50 and never trailed. Newton was limited by Denver's defense, which sacked him seven times and forced him\n",
            " into three turnovers, including a fumble which they recovered for a touchdown. Denver linebacker Von Miller was named Super Bowl MVP, recording five \n",
            "solo tackles, 2½ sacks, and two forced fumbles.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "five \n",
            "\n",
            "Untokenized Answer :  five\n",
            "\n",
            "real answer :  [{'answer_start': 45, 'text': '5'}, {'answer_start': 295, 'text': 'five'}, {'answer_start': 295, 'text': 'five'}]\n",
            "\n",
            "Question :  How many teams have played in the Super Bowl eight times?\n",
            "--------------------------------------------------\n",
            "Context :  The Panthers finished the regular season with a 15–1 record, and quarterback Cam Newton was named the NFL Most Valuable Player (MVP). They defeated th\n",
            "e Arizona Cardinals 49–15 in the NFC Championship Game and advanced to their second Super Bowl appearance since the franchise was founded in 1995. The\n",
            " Broncos finished the regular season with a 12–4 record, and denied the New England Patriots a chance to defend their title from Super Bowl XLIX by de\n",
            "feating them 20–18 in the AFC Championship Game. They joined the Patriots, Dallas Cowboys, and Pittsburgh Steelers as one of four teams that have made\n",
            " eight appearances in the Super Bowl.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "four \n",
            "\n",
            "Untokenized Answer :  four\n",
            "\n",
            "real answer :  [{'answer_start': 170, 'text': '4'}, {'answer_start': 575, 'text': 'four'}, {'answer_start': 575, 'text': 'four'}]\n",
            "\n",
            "Question :  What position does Von Miller play for the Denver Broncos?\n",
            "--------------------------------------------------\n",
            "Context :  The Broncos took an early lead in Super Bowl 50 and never trailed. Newton was limited by Denver's defense, which sacked him seven times and forced him\n",
            " into three turnovers, including a fumble which they recovered for a touchdown. Denver linebacker Von Miller was named Super Bowl MVP, recording five \n",
            "solo tackles, 2½ sacks, and two forced fumbles.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "line ##backer \n",
            "\n",
            "Untokenized Answer :  linebacker\n",
            "\n",
            "real answer :  [{'answer_start': 237, 'text': 'linebacker'}, {'answer_start': 237, 'text': 'linebacker'}, {'answer_start': 237, 'text': 'linebacker'}]\n",
            "\n",
            "Question :  Who did the Panthers beat to become the NFC champs?\n",
            "--------------------------------------------------\n",
            "Context :  The Panthers finished the regular season with a 15–1 record, and quarterback Cam Newton was named the NFL Most Valuable Player (MVP). They defeated th\n",
            "e Arizona Cardinals 49–15 in the NFC Championship Game and advanced to their second Super Bowl appearance since the franchise was founded in 1995. The\n",
            " Broncos finished the regular season with a 12–4 record, and denied the New England Patriots a chance to defend their title from Super Bowl XLIX by de\n",
            "feating them 20–18 in the AFC Championship Game. They joined the Patriots, Dallas Cowboys, and Pittsburgh Steelers as one of four teams that have made\n",
            " eight appearances in the Super Bowl.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "ari ##zona cardinal ##s \n",
            "\n",
            "Untokenized Answer :  arizona cardinals\n",
            "\n",
            "real answer :  [{'answer_start': 152, 'text': 'Arizona Cardinals'}, {'answer_start': 148, 'text': 'the Arizona Cardinals'}, {'answer_start': 152, 'text': 'Arizona Cardinals'}]\n",
            "\n",
            "Question :  Which team did not get a chance to defend their Super Bowl XLIX win in Super Bowl 50?\n",
            "--------------------------------------------------\n",
            "Context :  The Panthers finished the regular season with a 15–1 record, and quarterback Cam Newton was named the NFL Most Valuable Player (MVP). They defeated th\n",
            "e Arizona Cardinals 49–15 in the NFC Championship Game and advanced to their second Super Bowl appearance since the franchise was founded in 1995. The\n",
            " Broncos finished the regular season with a 12–4 record, and denied the New England Patriots a chance to defend their title from Super Bowl XLIX by de\n",
            "feating them 20–18 in the AFC Championship Game. They joined the Patriots, Dallas Cowboys, and Pittsburgh Steelers as one of four teams that have made\n",
            " eight appearances in the Super Bowl.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "new engl ##and pat ##riot ##s \n",
            "\n",
            "Untokenized Answer :  new england patriots\n",
            "\n",
            "real answer :  [{'answer_start': 372, 'text': 'New England Patriots'}, {'answer_start': 368, 'text': 'the New England Patriots'}, {'answer_start': 372, 'text': 'New England Patriots'}]\n",
            "\n",
            "Question :  The name of the NFL championship game is?\n",
            "--------------------------------------------------\n",
            "Context :  Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football\n",
            " Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24–10 to earn their third Super B\n",
            "owl title. The game was played on February 7, 2016, at Levi's Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th S\n",
            "uper Bowl, the league emphasized the \"golden anniversary\" with various gold-themed initiatives, as well as temporarily suspending the tradition of nam\n",
            "ing each Super Bowl game with Roman numerals (under which the game would have been known as \"Super Bowl L\"), so that the logo could prominently featur\n",
            "e the Arabic numerals 50.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "super bowl 50 \n",
            "\n",
            "Untokenized Answer :  super bowl 50\n",
            "\n",
            "real answer :  [{'answer_start': 0, 'text': 'Super Bowl'}, {'answer_start': 0, 'text': 'Super Bowl'}, {'answer_start': 0, 'text': 'Super Bowl'}]\n",
            "\n",
            "Question :  Who was the male singer who performed as a special guest during Super Bowl 50?\n",
            "--------------------------------------------------\n",
            "Context :  CBS broadcast Super Bowl 50 in the U.S., and charged an average of $5 million for a 30-second commercial during the game. The Super Bowl 50 halftime s\n",
            "how was headlined by the British rock group Coldplay with special guest performers Beyoncé and Bruno Mars, who headlined the Super Bowl XLVII and Supe\n",
            "r Bowl XLVIII halftime shows, respectively. It was the third-most watched U.S. broadcast ever.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "be ##yon ##ce and brun ##o mars \n",
            "\n",
            "Untokenized Answer :  beyonce and bruno mars\n",
            "\n",
            "real answer :  [{'answer_start': 245, 'text': 'Bruno Mars'}, {'answer_start': 245, 'text': 'Bruno Mars'}, {'answer_start': 245, 'text': 'Bruno Mars,'}]\n",
            "\n",
            "Question :  Who lost to the Broncos in the AFC Championship?\n",
            "--------------------------------------------------\n",
            "Context :  The Panthers finished the regular season with a 15–1 record, and quarterback Cam Newton was named the NFL Most Valuable Player (MVP). They defeated th\n",
            "e Arizona Cardinals 49–15 in the NFC Championship Game and advanced to their second Super Bowl appearance since the franchise was founded in 1995. The\n",
            " Broncos finished the regular season with a 12–4 record, and denied the New England Patriots a chance to defend their title from Super Bowl XLIX by de\n",
            "feating them 20–18 in the AFC Championship Game. They joined the Patriots, Dallas Cowboys, and Pittsburgh Steelers as one of four teams that have made\n",
            " eight appearances in the Super Bowl.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "new engl ##and pat ##riot ##s \n",
            "\n",
            "Untokenized Answer :  new england patriots\n",
            "\n",
            "real answer :  [{'answer_start': 372, 'text': 'New England Patriots'}, {'answer_start': 368, 'text': 'the New England Patriots'}, {'answer_start': 372, 'text': 'New England Patriots'}]\n",
            "\n",
            "Question :  What team was the NFC champion?\n",
            "--------------------------------------------------\n",
            "Context :  Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football\n",
            " Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24–10 to earn their third Super B\n",
            "owl title. The game was played on February 7, 2016, at Levi's Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th S\n",
            "uper Bowl, the league emphasized the \"golden anniversary\" with various gold-themed initiatives, as well as temporarily suspending the tradition of nam\n",
            "ing each Super Bowl game with Roman numerals (under which the game would have been known as \"Super Bowl L\"), so that the logo could prominently featur\n",
            "e the Arabic numerals 50.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "car ##olina pan ##thers \n",
            "\n",
            "Untokenized Answer :  carolina panthers\n",
            "\n",
            "real answer :  [{'answer_start': 249, 'text': 'Carolina Panthers'}, {'answer_start': 249, 'text': 'Carolina Panthers'}, {'answer_start': 249, 'text': 'Carolina Panthers'}]\n",
            "\n",
            "Question :  What year did the Denver Broncos secure a Super Bowl title for the third time?\n",
            "--------------------------------------------------\n",
            "Context :  Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football\n",
            " Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24–10 to earn their third Super B\n",
            "owl title. The game was played on February 7, 2016, at Levi's Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th S\n",
            "uper Bowl, the league emphasized the \"golden anniversary\" with various gold-themed initiatives, as well as temporarily suspending the tradition of nam\n",
            "ing each Super Bowl game with Roman numerals (under which the game would have been known as \"Super Bowl L\"), so that the logo could prominently featur\n",
            "e the Arabic numerals 50.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "2015 \n",
            "\n",
            "Untokenized Answer :  2015\n",
            "\n",
            "real answer :  [{'answer_start': 116, 'text': '2015'}, {'answer_start': 346, 'text': '2016'}, {'answer_start': 116, 'text': '2015'}]\n",
            "\n",
            "Question :  Who did Carolina beat in the NFC championship game?\n",
            "--------------------------------------------------\n",
            "Context :  The Panthers finished the regular season with a 15–1 record, and quarterback Cam Newton was named the NFL Most Valuable Player (MVP). They defeated th\n",
            "e Arizona Cardinals 49–15 in the NFC Championship Game and advanced to their second Super Bowl appearance since the franchise was founded in 1995. The\n",
            " Broncos finished the regular season with a 12–4 record, and denied the New England Patriots a chance to defend their title from Super Bowl XLIX by de\n",
            "feating them 20–18 in the AFC Championship Game. They joined the Patriots, Dallas Cowboys, and Pittsburgh Steelers as one of four teams that have made\n",
            " eight appearances in the Super Bowl.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "ari ##zona cardinal ##s \n",
            "\n",
            "Untokenized Answer :  arizona cardinals\n",
            "\n",
            "real answer :  [{'answer_start': 152, 'text': 'Arizona Cardinals'}, {'answer_start': 148, 'text': 'the Arizona Cardinals'}, {'answer_start': 152, 'text': 'Arizona Cardinals'}]\n",
            "\n",
            "Question :  What team was the AFC champion?\n",
            "--------------------------------------------------\n",
            "Context :  Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football\n",
            " Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24–10 to earn their third Super B\n",
            "owl title. The game was played on February 7, 2016, at Levi's Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th S\n",
            "uper Bowl, the league emphasized the \"golden anniversary\" with various gold-themed initiatives, as well as temporarily suspending the tradition of nam\n",
            "ing each Super Bowl game with Roman numerals (under which the game would have been known as \"Super Bowl L\"), so that the logo could prominently featur\n",
            "e the Arabic numerals 50.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "den ##ver bron ##cos \n",
            "\n",
            "Untokenized Answer :  denver broncos\n",
            "\n",
            "real answer :  [{'answer_start': 177, 'text': 'Denver Broncos'}, {'answer_start': 177, 'text': 'Denver Broncos'}, {'answer_start': 177, 'text': 'Denver Broncos'}]\n",
            "\n",
            "Question :  What was the theme of Super Bowl 50?\n",
            "--------------------------------------------------\n",
            "Context :  Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football\n",
            " Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24–10 to earn their third Super B\n",
            "owl title. The game was played on February 7, 2016, at Levi's Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th S\n",
            "uper Bowl, the league emphasized the \"golden anniversary\" with various gold-themed initiatives, as well as temporarily suspending the tradition of nam\n",
            "ing each Super Bowl game with Roman numerals (under which the game would have been known as \"Super Bowl L\"), so that the logo could prominently featur\n",
            "e the Arabic numerals 50.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "gold \n",
            "\n",
            "Untokenized Answer :  gold\n",
            "\n",
            "real answer :  [{'answer_start': 487, 'text': '\"golden anniversary\"'}, {'answer_start': 521, 'text': 'gold-themed'}, {'answer_start': 521, 'text': 'gold'}]\n",
            "\n",
            "Question :  Who was this season's NFL MVP?\n",
            "--------------------------------------------------\n",
            "Context :  The Panthers finished the regular season with a 15–1 record, and quarterback Cam Newton was named the NFL Most Valuable Player (MVP). They defeated th\n",
            "e Arizona Cardinals 49–15 in the NFC Championship Game and advanced to their second Super Bowl appearance since the franchise was founded in 1995. The\n",
            " Broncos finished the regular season with a 12–4 record, and denied the New England Patriots a chance to defend their title from Super Bowl XLIX by de\n",
            "feating them 20–18 in the AFC Championship Game. They joined the Patriots, Dallas Cowboys, and Pittsburgh Steelers as one of four teams that have made\n",
            " eight appearances in the Super Bowl.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "cam new ##ton \n",
            "\n",
            "Untokenized Answer :  cam newton\n",
            "\n",
            "real answer :  [{'answer_start': 77, 'text': 'Cam Newton'}, {'answer_start': 77, 'text': 'Cam Newton'}, {'answer_start': 77, 'text': 'Cam Newton'}]\n",
            "\n",
            "Question :  How many times have the Panthers been in the Super Bowl?\n",
            "--------------------------------------------------\n",
            "Context :  The Panthers finished the regular season with a 15–1 record, and quarterback Cam Newton was named the NFL Most Valuable Player (MVP). They defeated th\n",
            "e Arizona Cardinals 49–15 in the NFC Championship Game and advanced to their second Super Bowl appearance since the franchise was founded in 1995. The\n",
            " Broncos finished the regular season with a 12–4 record, and denied the New England Patriots a chance to defend their title from Super Bowl XLIX by de\n",
            "feating them 20–18 in the AFC Championship Game. They joined the Patriots, Dallas Cowboys, and Pittsburgh Steelers as one of four teams that have made\n",
            " eight appearances in the Super Bowl.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "eight \n",
            "\n",
            "Untokenized Answer :  eight\n",
            "\n",
            "real answer :  [{'answer_start': 345, 'text': '2'}, {'answer_start': 227, 'text': 'second'}, {'answer_start': 227, 'text': 'second'}]\n",
            "\n",
            "Question :  Who was the main performer at this year's halftime show?\n",
            "--------------------------------------------------\n",
            "Context :  CBS broadcast Super Bowl 50 in the U.S., and charged an average of $5 million for a 30-second commercial during the game. The Super Bowl 50 halftime s\n",
            "how was headlined by the British rock group Coldplay with special guest performers Beyoncé and Bruno Mars, who headlined the Super Bowl XLVII and Supe\n",
            "r Bowl XLVIII halftime shows, respectively. It was the third-most watched U.S. broadcast ever.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "be ##yon ##ce \n",
            "\n",
            "Untokenized Answer :  beyonce\n",
            "\n",
            "real answer :  [{'answer_start': 194, 'text': 'Coldplay'}, {'answer_start': 194, 'text': 'Coldplay'}, {'answer_start': 194, 'text': 'Coldplay'}]\n",
            "\n",
            "Question :  Who headlined the halftime show for Super Bowl 50?\n",
            "--------------------------------------------------\n",
            "Context :  CBS broadcast Super Bowl 50 in the U.S., and charged an average of $5 million for a 30-second commercial during the game. The Super Bowl 50 halftime s\n",
            "how was headlined by the British rock group Coldplay with special guest performers Beyoncé and Bruno Mars, who headlined the Super Bowl XLVII and Supe\n",
            "r Bowl XLVIII halftime shows, respectively. It was the third-most watched U.S. broadcast ever.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "cold ##play \n",
            "\n",
            "Untokenized Answer :  coldplay\n",
            "\n",
            "real answer :  [{'answer_start': 194, 'text': 'Coldplay'}, {'answer_start': 194, 'text': 'Coldplay'}, {'answer_start': 194, 'text': 'Coldplay'}]\n",
            "\n",
            "Question :  What Denver player caused two fumbles for the Panthers?\n",
            "--------------------------------------------------\n",
            "Context :  The Broncos took an early lead in Super Bowl 50 and never trailed. Newton was limited by Denver's defense, which sacked him seven times and forced him\n",
            " into three turnovers, including a fumble which they recovered for a touchdown. Denver linebacker Von Miller was named Super Bowl MVP, recording five \n",
            "solo tackles, 2½ sacks, and two forced fumbles.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "von mille ##r \n",
            "\n",
            "Untokenized Answer :  von miller\n",
            "\n",
            "real answer :  [{'answer_start': 248, 'text': 'Von Miller'}, {'answer_start': 248, 'text': 'Von Miller'}, {'answer_start': 248, 'text': 'Von Miller'}]\n",
            "\n",
            "Question :  What was the average cost for a TV ad lasting 30 seconds during Super Bowl 50?\n",
            "--------------------------------------------------\n",
            "Context :  CBS broadcast Super Bowl 50 in the U.S., and charged an average of $5 million for a 30-second commercial during the game. The Super Bowl 50 halftime s\n",
            "how was headlined by the British rock group Coldplay with special guest performers Beyoncé and Bruno Mars, who headlined the Super Bowl XLVII and Supe\n",
            "r Bowl XLVIII halftime shows, respectively. It was the third-most watched U.S. broadcast ever.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "$ 5 million \n",
            "\n",
            "Untokenized Answer :  $ 5 million\n",
            "\n",
            "real answer :  [{'answer_start': 67, 'text': '$5 million'}, {'answer_start': 67, 'text': '$5 million'}, {'answer_start': 67, 'text': '$5 million'}]\n",
            "\n",
            "Question :  What year was Super Bowl 50?\n",
            "--------------------------------------------------\n",
            "Context :  Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football\n",
            " Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24–10 to earn their third Super B\n",
            "owl title. The game was played on February 7, 2016, at Levi's Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th S\n",
            "uper Bowl, the league emphasized the \"golden anniversary\" with various gold-themed initiatives, as well as temporarily suspending the tradition of nam\n",
            "ing each Super Bowl game with Roman numerals (under which the game would have been known as \"Super Bowl L\"), so that the logo could prominently featur\n",
            "e the Arabic numerals 50.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "2015 \n",
            "\n",
            "Untokenized Answer :  2015\n",
            "\n",
            "real answer :  [{'answer_start': 116, 'text': '2015'}, {'answer_start': 346, 'text': '2016'}, {'answer_start': 346, 'text': '2016'}]\n",
            "\n",
            "Question :  Who broadcast the Super Bowl on TV?\n",
            "--------------------------------------------------\n",
            "Context :  CBS broadcast Super Bowl 50 in the U.S., and charged an average of $5 million for a 30-second commercial during the game. The Super Bowl 50 halftime s\n",
            "how was headlined by the British rock group Coldplay with special guest performers Beyoncé and Bruno Mars, who headlined the Super Bowl XLVII and Supe\n",
            "r Bowl XLVIII halftime shows, respectively. It was the third-most watched U.S. broadcast ever.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "c ##bs \n",
            "\n",
            "Untokenized Answer :  cbs\n",
            "\n",
            "real answer :  [{'answer_start': 0, 'text': 'CBS'}, {'answer_start': 0, 'text': 'CBS'}, {'answer_start': 0, 'text': 'CBS'}]\n",
            "\n",
            "Question :  What color was used to emphasize the 50th anniversary of the Super Bowl?\n",
            "--------------------------------------------------\n",
            "Context :  Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football\n",
            " Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24–10 to earn their third Super B\n",
            "owl title. The game was played on February 7, 2016, at Levi's Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th S\n",
            "uper Bowl, the league emphasized the \"golden anniversary\" with various gold-themed initiatives, as well as temporarily suspending the tradition of nam\n",
            "ing each Super Bowl game with Roman numerals (under which the game would have been known as \"Super Bowl L\"), so that the logo could prominently featur\n",
            "e the Arabic numerals 50.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "golden \n",
            "\n",
            "Untokenized Answer :  golden\n",
            "\n",
            "real answer :  [{'answer_start': 488, 'text': 'gold'}, {'answer_start': 488, 'text': 'gold'}, {'answer_start': 521, 'text': 'gold'}]\n",
            "\n",
            "Question :  Who won the Super Bowl MVP?\n",
            "--------------------------------------------------\n",
            "Context :  The Broncos took an early lead in Super Bowl 50 and never trailed. Newton was limited by Denver's defense, which sacked him seven times and forced him\n",
            " into three turnovers, including a fumble which they recovered for a touchdown. Denver linebacker Von Miller was named Super Bowl MVP, recording five \n",
            "solo tackles, 2½ sacks, and two forced fumbles.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "the bron ##cos \n",
            "\n",
            "Untokenized Answer :  the broncos\n",
            "\n",
            "real answer :  [{'answer_start': 248, 'text': 'Von Miller'}, {'answer_start': 0, 'text': 'The Broncos'}, {'answer_start': 252, 'text': 'Miller'}]\n",
            "\n",
            "Question :  How many solo tackles did Von Miller make at Super Bowl 50?\n",
            "--------------------------------------------------\n",
            "Context :  The Broncos took an early lead in Super Bowl 50 and never trailed. Newton was limited by Denver's defense, which sacked him seven times and forced him\n",
            " into three turnovers, including a fumble which they recovered for a touchdown. Denver linebacker Von Miller was named Super Bowl MVP, recording five \n",
            "solo tackles, 2½ sacks, and two forced fumbles.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "five \n",
            "\n",
            "Untokenized Answer :  five\n",
            "\n",
            "real answer :  [{'answer_start': 295, 'text': 'five solo tackles'}, {'answer_start': 295, 'text': 'five'}, {'answer_start': 295, 'text': 'five'}]\n",
            "\n",
            "Question :  What halftime performer previously headlined Super Bowl XLVIII?\n",
            "--------------------------------------------------\n",
            "Context :  CBS broadcast Super Bowl 50 in the U.S., and charged an average of $5 million for a 30-second commercial during the game. The Super Bowl 50 halftime s\n",
            "how was headlined by the British rock group Coldplay with special guest performers Beyoncé and Bruno Mars, who headlined the Super Bowl XLVII and Supe\n",
            "r Bowl XLVIII halftime shows, respectively. It was the third-most watched U.S. broadcast ever.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "brun ##o mars \n",
            "\n",
            "Untokenized Answer :  bruno mars\n",
            "\n",
            "real answer :  [{'answer_start': 245, 'text': 'Bruno Mars'}, {'answer_start': 245, 'text': 'Bruno Mars'}, {'answer_start': 251, 'text': 'Mars'}]\n",
            "\n",
            "Question :  Who was given the esteemed status of MVP for Super Bowl 50?\n",
            "--------------------------------------------------\n",
            "Context :  The Broncos took an early lead in Super Bowl 50 and never trailed. Newton was limited by Denver's defense, which sacked him seven times and forced him\n",
            " into three turnovers, including a fumble which they recovered for a touchdown. Denver linebacker Von Miller was named Super Bowl MVP, recording five \n",
            "solo tackles, 2½ sacks, and two forced fumbles.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "von mille ##r \n",
            "\n",
            "Untokenized Answer :  von miller\n",
            "\n",
            "real answer :  [{'answer_start': 248, 'text': 'Von Miller'}, {'answer_start': 248, 'text': 'Von Miller'}, {'answer_start': 252, 'text': 'Miller'}]\n",
            "\n",
            "Question :  Which Carolina Panthers player was named Most Valuable Player?\n",
            "--------------------------------------------------\n",
            "Context :  The Panthers finished the regular season with a 15–1 record, and quarterback Cam Newton was named the NFL Most Valuable Player (MVP). They defeated th\n",
            "e Arizona Cardinals 49–15 in the NFC Championship Game and advanced to their second Super Bowl appearance since the franchise was founded in 1995. The\n",
            " Broncos finished the regular season with a 12–4 record, and denied the New England Patriots a chance to defend their title from Super Bowl XLIX by de\n",
            "feating them 20–18 in the AFC Championship Game. They joined the Patriots, Dallas Cowboys, and Pittsburgh Steelers as one of four teams that have made\n",
            " eight appearances in the Super Bowl.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "cam new ##ton \n",
            "\n",
            "Untokenized Answer :  cam newton\n",
            "\n",
            "real answer :  [{'answer_start': 77, 'text': 'Cam Newton'}, {'answer_start': 77, 'text': 'Cam Newton'}, {'answer_start': 77, 'text': 'Cam Newton'}]\n",
            "\n",
            "Question :  Who did the Carolina Panthers beat in the 2015 NFC Championship game?\n",
            "--------------------------------------------------\n",
            "Context :  The Panthers finished the regular season with a 15–1 record, and quarterback Cam Newton was named the NFL Most Valuable Player (MVP). They defeated th\n",
            "e Arizona Cardinals 49–15 in the NFC Championship Game and advanced to their second Super Bowl appearance since the franchise was founded in 1995. The\n",
            " Broncos finished the regular season with a 12–4 record, and denied the New England Patriots a chance to defend their title from Super Bowl XLIX by de\n",
            "feating them 20–18 in the AFC Championship Game. They joined the Patriots, Dallas Cowboys, and Pittsburgh Steelers as one of four teams that have made\n",
            " eight appearances in the Super Bowl.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "ari ##zona cardinal ##s \n",
            "\n",
            "Untokenized Answer :  arizona cardinals\n",
            "\n",
            "real answer :  [{'answer_start': 152, 'text': 'Arizona Cardinals'}, {'answer_start': 148, 'text': 'the Arizona Cardinals'}, {'answer_start': 152, 'text': 'Arizona Cardinals'}]\n",
            "\n",
            "Question :  How much money did a 1/2 minute commercial cost?\n",
            "--------------------------------------------------\n",
            "Context :  CBS broadcast Super Bowl 50 in the U.S., and charged an average of $5 million for a 30-second commercial during the game. The Super Bowl 50 halftime s\n",
            "how was headlined by the British rock group Coldplay with special guest performers Beyoncé and Bruno Mars, who headlined the Super Bowl XLVII and Supe\n",
            "r Bowl XLVIII halftime shows, respectively. It was the third-most watched U.S. broadcast ever.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "$ 5 million \n",
            "\n",
            "Untokenized Answer :  $ 5 million\n",
            "\n",
            "real answer :  [{'answer_start': 67, 'text': '$5 million'}, {'answer_start': 67, 'text': '$5 million'}, {'answer_start': 67, 'text': '$5 million'}]\n",
            "\n",
            "Question :  Which NFL team represented the AFC at Super Bowl 50?\n",
            "--------------------------------------------------\n",
            "Context :  Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football\n",
            " Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24–10 to earn their third Super B\n",
            "owl title. The game was played on February 7, 2016, at Levi's Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th S\n",
            "uper Bowl, the league emphasized the \"golden anniversary\" with various gold-themed initiatives, as well as temporarily suspending the tradition of nam\n",
            "ing each Super Bowl game with Roman numerals (under which the game would have been known as \"Super Bowl L\"), so that the logo could prominently featur\n",
            "e the Arabic numerals 50.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "car ##olina pan ##thers \n",
            "\n",
            "Untokenized Answer :  carolina panthers\n",
            "\n",
            "real answer :  [{'answer_start': 177, 'text': 'Denver Broncos'}, {'answer_start': 177, 'text': 'Denver Broncos'}, {'answer_start': 177, 'text': 'Denver Broncos'}]\n",
            "\n",
            "Question :  Which team held the scoring lead throughout the entire game?\n",
            "--------------------------------------------------\n",
            "Context :  The Broncos took an early lead in Super Bowl 50 and never trailed. Newton was limited by Denver's defense, which sacked him seven times and forced him\n",
            " into three turnovers, including a fumble which they recovered for a touchdown. Denver linebacker Von Miller was named Super Bowl MVP, recording five \n",
            "solo tackles, 2½ sacks, and two forced fumbles.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "bron ##cos \n",
            "\n",
            "Untokenized Answer :  broncos\n",
            "\n",
            "real answer :  [{'answer_start': 4, 'text': 'Broncos'}, {'answer_start': 0, 'text': 'The Broncos'}, {'answer_start': 4, 'text': 'Broncos'}]\n",
            "\n",
            "Question :  Super Bowl 50 determined the NFL champion for what season?\n",
            "--------------------------------------------------\n",
            "Context :  Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football\n",
            " Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24–10 to earn their third Super B\n",
            "owl title. The game was played on February 7, 2016, at Levi's Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th S\n",
            "uper Bowl, the league emphasized the \"golden anniversary\" with various gold-themed initiatives, as well as temporarily suspending the tradition of nam\n",
            "ing each Super Bowl game with Roman numerals (under which the game would have been known as \"Super Bowl L\"), so that the logo could prominently featur\n",
            "e the Arabic numerals 50.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "2015 \n",
            "\n",
            "Untokenized Answer :  2015\n",
            "\n",
            "real answer :  [{'answer_start': 116, 'text': '2015'}, {'answer_start': 112, 'text': 'the 2015 season'}, {'answer_start': 116, 'text': '2015'}]\n",
            "\n",
            "Question :  What team did the Panthers defeat?\n",
            "--------------------------------------------------\n",
            "Context :  The Panthers finished the regular season with a 15–1 record, and quarterback Cam Newton was named the NFL Most Valuable Player (MVP). They defeated th\n",
            "e Arizona Cardinals 49–15 in the NFC Championship Game and advanced to their second Super Bowl appearance since the franchise was founded in 1995. The\n",
            " Broncos finished the regular season with a 12–4 record, and denied the New England Patriots a chance to defend their title from Super Bowl XLIX by de\n",
            "feating them 20–18 in the AFC Championship Game. They joined the Patriots, Dallas Cowboys, and Pittsburgh Steelers as one of four teams that have made\n",
            " eight appearances in the Super Bowl.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "ari ##zona cardinal ##s \n",
            "\n",
            "Untokenized Answer :  arizona cardinals\n",
            "\n",
            "real answer :  [{'answer_start': 152, 'text': 'Arizona Cardinals'}, {'answer_start': 148, 'text': 'the Arizona Cardinals'}, {'answer_start': 152, 'text': 'Arizona Cardinals'}]\n",
            "\n",
            "Question :  Which Denver linebacker was named Super Bowl MVP?\n",
            "--------------------------------------------------\n",
            "Context :  The Broncos took an early lead in Super Bowl 50 and never trailed. Newton was limited by Denver's defense, which sacked him seven times and forced him\n",
            " into three turnovers, including a fumble which they recovered for a touchdown. Denver linebacker Von Miller was named Super Bowl MVP, recording five \n",
            "solo tackles, 2½ sacks, and two forced fumbles.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "von mille ##r \n",
            "\n",
            "Untokenized Answer :  von miller\n",
            "\n",
            "real answer :  [{'answer_start': 237, 'text': 'linebacker Von Miller'}, {'answer_start': 248, 'text': 'Von Miller'}, {'answer_start': 252, 'text': 'Miller'}]\n",
            "\n",
            "Question :  What day was the game played on?\n",
            "--------------------------------------------------\n",
            "Context :  Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football\n",
            " Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24–10 to earn their third Super B\n",
            "owl title. The game was played on February 7, 2016, at Levi's Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th S\n",
            "uper Bowl, the league emphasized the \"golden anniversary\" with various gold-themed initiatives, as well as temporarily suspending the tradition of nam\n",
            "ing each Super Bowl game with Roman numerals (under which the game would have been known as \"Super Bowl L\"), so that the logo could prominently featur\n",
            "e the Arabic numerals 50.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "februar ##y 7 , 2016 \n",
            "\n",
            "Untokenized Answer :  february 7 , 2016\n",
            "\n",
            "real answer :  [{'answer_start': 334, 'text': 'February 7, 2016'}, {'answer_start': 334, 'text': 'February 7'}, {'answer_start': 334, 'text': 'February 7, 2016'}]\n",
            "\n",
            "Question :  What was the theme of Super Bowl 50?\n",
            "--------------------------------------------------\n",
            "Context :  Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football\n",
            " Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24–10 to earn their third Super B\n",
            "owl title. The game was played on February 7, 2016, at Levi's Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th S\n",
            "uper Bowl, the league emphasized the \"golden anniversary\" with various gold-themed initiatives, as well as temporarily suspending the tradition of nam\n",
            "ing each Super Bowl game with Roman numerals (under which the game would have been known as \"Super Bowl L\"), so that the logo could prominently featur\n",
            "e the Arabic numerals 50.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "gold \n",
            "\n",
            "Untokenized Answer :  gold\n",
            "\n",
            "real answer :  [{'answer_start': 487, 'text': '\"golden anniversary\"'}, {'answer_start': 521, 'text': 'gold-themed'}, {'answer_start': 487, 'text': '\"golden anniversary'}]\n",
            "\n",
            "Question :  What station aired the Super Bowl?\n",
            "--------------------------------------------------\n",
            "Context :  CBS broadcast Super Bowl 50 in the U.S., and charged an average of $5 million for a 30-second commercial during the game. The Super Bowl 50 halftime s\n",
            "how was headlined by the British rock group Coldplay with special guest performers Beyoncé and Bruno Mars, who headlined the Super Bowl XLVII and Supe\n",
            "r Bowl XLVIII halftime shows, respectively. It was the third-most watched U.S. broadcast ever.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "c ##bs \n",
            "\n",
            "Untokenized Answer :  cbs\n",
            "\n",
            "real answer :  [{'answer_start': 0, 'text': 'CBS'}, {'answer_start': 0, 'text': 'CBS'}, {'answer_start': 0, 'text': 'CBS'}]\n",
            "\n",
            "Question :  Who is the quarterback for the Panthers?\n",
            "--------------------------------------------------\n",
            "Context :  The Panthers finished the regular season with a 15–1 record, and quarterback Cam Newton was named the NFL Most Valuable Player (MVP). They defeated th\n",
            "e Arizona Cardinals 49–15 in the NFC Championship Game and advanced to their second Super Bowl appearance since the franchise was founded in 1995. The\n",
            " Broncos finished the regular season with a 12–4 record, and denied the New England Patriots a chance to defend their title from Super Bowl XLIX by de\n",
            "feating them 20–18 in the AFC Championship Game. They joined the Patriots, Dallas Cowboys, and Pittsburgh Steelers as one of four teams that have made\n",
            " eight appearances in the Super Bowl.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "cam new ##ton \n",
            "\n",
            "Untokenized Answer :  cam newton\n",
            "\n",
            "real answer :  [{'answer_start': 77, 'text': 'Cam Newton'}, {'answer_start': 77, 'text': 'Cam Newton'}, {'answer_start': 77, 'text': 'Cam Newton'}]\n",
            "\n",
            "Question :  Which NFL team represented the NFC at Super Bowl 50?\n",
            "--------------------------------------------------\n",
            "Context :  Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football\n",
            " Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24–10 to earn their third Super B\n",
            "owl title. The game was played on February 7, 2016, at Levi's Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th S\n",
            "uper Bowl, the league emphasized the \"golden anniversary\" with various gold-themed initiatives, as well as temporarily suspending the tradition of nam\n",
            "ing each Super Bowl game with Roman numerals (under which the game would have been known as \"Super Bowl L\"), so that the logo could prominently featur\n",
            "e the Arabic numerals 50.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "car ##olina pan ##thers \n",
            "\n",
            "Untokenized Answer :  carolina panthers\n",
            "\n",
            "real answer :  [{'answer_start': 249, 'text': 'Carolina Panthers'}, {'answer_start': 249, 'text': 'Carolina Panthers'}, {'answer_start': 249, 'text': 'Carolina Panthers'}]\n",
            "\n",
            "Question :  What was the average cost for a 30 second commercial during Super Bowl 50?\n",
            "--------------------------------------------------\n",
            "Context :  CBS broadcast Super Bowl 50 in the U.S., and charged an average of $5 million for a 30-second commercial during the game. The Super Bowl 50 halftime s\n",
            "how was headlined by the British rock group Coldplay with special guest performers Beyoncé and Bruno Mars, who headlined the Super Bowl XLVII and Supe\n",
            "r Bowl XLVIII halftime shows, respectively. It was the third-most watched U.S. broadcast ever.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "$ 5 million \n",
            "\n",
            "Untokenized Answer :  $ 5 million\n",
            "\n",
            "real answer :  [{'answer_start': 67, 'text': '$5 million'}, {'answer_start': 67, 'text': '$5 million'}, {'answer_start': 67, 'text': '$5 million'}]\n",
            "\n",
            "Question :  Which network broadcasted the 50th Super Bowl game? \n",
            "--------------------------------------------------\n",
            "Context :  CBS broadcast Super Bowl 50 in the U.S., and charged an average of $5 million for a 30-second commercial during the game. The Super Bowl 50 halftime s\n",
            "how was headlined by the British rock group Coldplay with special guest performers Beyoncé and Bruno Mars, who headlined the Super Bowl XLVII and Supe\n",
            "r Bowl XLVIII halftime shows, respectively. It was the third-most watched U.S. broadcast ever.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "c ##bs \n",
            "\n",
            "Untokenized Answer :  cbs\n",
            "\n",
            "real answer :  [{'answer_start': 0, 'text': 'CBS'}, {'answer_start': 0, 'text': 'CBS'}, {'answer_start': 0, 'text': 'CBS'}]\n",
            "\n",
            "Question :  What was the final score of Super Bowl 50? \n",
            "--------------------------------------------------\n",
            "Context :  Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football\n",
            " Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24–10 to earn their third Super B\n",
            "owl title. The game was played on February 7, 2016, at Levi's Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th S\n",
            "uper Bowl, the league emphasized the \"golden anniversary\" with various gold-themed initiatives, as well as temporarily suspending the tradition of nam\n",
            "ing each Super Bowl game with Roman numerals (under which the game would have been known as \"Super Bowl L\"), so that the logo could prominently featur\n",
            "e the Arabic numerals 50.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "24 [UNK] 10 \n",
            "\n",
            "Untokenized Answer :  24 [UNK] 10\n",
            "\n",
            "real answer :  [{'answer_start': 267, 'text': '24–10'}, {'answer_start': 267, 'text': '24–10'}, {'answer_start': 267, 'text': '24–10'}]\n",
            "\n",
            "Question :  Which Newton turnover resulted in seven points for Denver?\n",
            "--------------------------------------------------\n",
            "Context :  The Broncos took an early lead in Super Bowl 50 and never trailed. Newton was limited by Denver's defense, which sacked him seven times and forced him\n",
            " into three turnovers, including a fumble which they recovered for a touchdown. Denver linebacker Von Miller was named Super Bowl MVP, recording five \n",
            "solo tackles, 2½ sacks, and two forced fumbles.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "fu ##mble \n",
            "\n",
            "Untokenized Answer :  fumble\n",
            "\n",
            "real answer :  [{'answer_start': 183, 'text': 'a fumble'}, {'answer_start': 183, 'text': 'a fumble'}, {'answer_start': 185, 'text': 'fumble'}]\n",
            "\n",
            "Question :  Who was limited by Denver's defense?\n",
            "--------------------------------------------------\n",
            "Context :  The Broncos took an early lead in Super Bowl 50 and never trailed. Newton was limited by Denver's defense, which sacked him seven times and forced him\n",
            " into three turnovers, including a fumble which they recovered for a touchdown. Denver linebacker Von Miller was named Super Bowl MVP, recording five \n",
            "solo tackles, 2½ sacks, and two forced fumbles.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "new ##ton \n",
            "\n",
            "Untokenized Answer :  newton\n",
            "\n",
            "real answer :  [{'answer_start': 67, 'text': \"Newton was limited by Denver's defense\"}, {'answer_start': 67, 'text': 'Newton'}, {'answer_start': 67, 'text': 'Newton'}]\n",
            "\n",
            "Question :  Where did Super Bowl 50 take place?\n",
            "--------------------------------------------------\n",
            "Context :  Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football\n",
            " Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24–10 to earn their third Super B\n",
            "owl title. The game was played on February 7, 2016, at Levi's Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th S\n",
            "uper Bowl, the league emphasized the \"golden anniversary\" with various gold-themed initiatives, as well as temporarily suspending the tradition of nam\n",
            "ing each Super Bowl game with Roman numerals (under which the game would have been known as \"Super Bowl L\"), so that the logo could prominently featur\n",
            "e the Arabic numerals 50.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "le ##vi ' s stadium in the san franc ##is ##co bay area at santa clara , cal ##if ##orn ##ia \n",
            "\n",
            "Untokenized Answer :  levi ' s stadium in the san francisco bay area at santa clara , california\n",
            "\n",
            "real answer :  [{'answer_start': 403, 'text': 'Santa Clara, California'}, {'answer_start': 355, 'text': \"Levi's Stadium\"}, {'answer_start': 355, 'text': \"Levi's Stadium in the San Francisco Bay Area at Santa Clara, California.\"}]\n",
            "\n",
            "Question :  Who was the Most Valuable Player for the 2015 NFL season?\n",
            "--------------------------------------------------\n",
            "Context :  The Panthers finished the regular season with a 15–1 record, and quarterback Cam Newton was named the NFL Most Valuable Player (MVP). They defeated th\n",
            "e Arizona Cardinals 49–15 in the NFC Championship Game and advanced to their second Super Bowl appearance since the franchise was founded in 1995. The\n",
            " Broncos finished the regular season with a 12–4 record, and denied the New England Patriots a chance to defend their title from Super Bowl XLIX by de\n",
            "feating them 20–18 in the AFC Championship Game. They joined the Patriots, Dallas Cowboys, and Pittsburgh Steelers as one of four teams that have made\n",
            " eight appearances in the Super Bowl.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "cam new ##ton \n",
            "\n",
            "Untokenized Answer :  cam newton\n",
            "\n",
            "real answer :  [{'answer_start': 77, 'text': 'Cam Newton'}, {'answer_start': 77, 'text': 'Cam Newton'}, {'answer_start': 77, 'text': 'Cam Newton'}]\n",
            "\n",
            "Question :  How many times did the Broncos cause turnovers in the game?\n",
            "--------------------------------------------------\n",
            "Context :  The Broncos took an early lead in Super Bowl 50 and never trailed. Newton was limited by Denver's defense, which sacked him seven times and forced him\n",
            " into three turnovers, including a fumble which they recovered for a touchdown. Denver linebacker Von Miller was named Super Bowl MVP, recording five \n",
            "solo tackles, 2½ sacks, and two forced fumbles.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "three \n",
            "\n",
            "Untokenized Answer :  three\n",
            "\n",
            "real answer :  [{'answer_start': 156, 'text': 'three'}, {'answer_start': 156, 'text': 'three'}, {'answer_start': 156, 'text': 'three'}]\n",
            "\n",
            "Question :  Which Carolina Panthers team member was picked as the team's MVP in 2015? \n",
            "--------------------------------------------------\n",
            "Context :  The Panthers finished the regular season with a 15–1 record, and quarterback Cam Newton was named the NFL Most Valuable Player (MVP). They defeated th\n",
            "e Arizona Cardinals 49–15 in the NFC Championship Game and advanced to their second Super Bowl appearance since the franchise was founded in 1995. The\n",
            " Broncos finished the regular season with a 12–4 record, and denied the New England Patriots a chance to defend their title from Super Bowl XLIX by de\n",
            "feating them 20–18 in the AFC Championship Game. They joined the Patriots, Dallas Cowboys, and Pittsburgh Steelers as one of four teams that have made\n",
            " eight appearances in the Super Bowl.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "cam new ##ton \n",
            "\n",
            "Untokenized Answer :  cam newton\n",
            "\n",
            "real answer :  [{'answer_start': 77, 'text': 'Cam Newton'}, {'answer_start': 77, 'text': 'Cam Newton'}, {'answer_start': 77, 'text': 'Cam Newton'}]\n",
            "\n",
            "Question :  What city did Super Bowl 50 take place in?\n",
            "--------------------------------------------------\n",
            "Context :  Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football\n",
            " Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24–10 to earn their third Super B\n",
            "owl title. The game was played on February 7, 2016, at Levi's Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th S\n",
            "uper Bowl, the league emphasized the \"golden anniversary\" with various gold-themed initiatives, as well as temporarily suspending the tradition of nam\n",
            "ing each Super Bowl game with Roman numerals (under which the game would have been known as \"Super Bowl L\"), so that the logo could prominently featur\n",
            "e the Arabic numerals 50.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "santa clara , cal ##if ##orn ##ia \n",
            "\n",
            "Untokenized Answer :  santa clara , california\n",
            "\n",
            "real answer :  [{'answer_start': 403, 'text': 'Santa Clara'}, {'answer_start': 403, 'text': 'Santa Clara'}, {'answer_start': 403, 'text': 'Santa Clara'}]\n",
            "\n",
            "Question :  How many times was Cam Newton sacked?\n",
            "--------------------------------------------------\n",
            "Context :  The Broncos took an early lead in Super Bowl 50 and never trailed. Newton was limited by Denver's defense, which sacked him seven times and forced him\n",
            " into three turnovers, including a fumble which they recovered for a touchdown. Denver linebacker Von Miller was named Super Bowl MVP, recording five \n",
            "solo tackles, 2½ sacks, and two forced fumbles.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "seven \n",
            "\n",
            "Untokenized Answer :  seven\n",
            "\n",
            "real answer :  [{'answer_start': 124, 'text': 'seven'}, {'answer_start': 124, 'text': 'seven'}, {'answer_start': 124, 'text': 'seven'}]\n",
            "\n",
            "Question :  What year was the Carolina Panthers franchise founded?\n",
            "--------------------------------------------------\n",
            "Context :  The Panthers finished the regular season with a 15–1 record, and quarterback Cam Newton was named the NFL Most Valuable Player (MVP). They defeated th\n",
            "e Arizona Cardinals 49–15 in the NFC Championship Game and advanced to their second Super Bowl appearance since the franchise was founded in 1995. The\n",
            " Broncos finished the regular season with a 12–4 record, and denied the New England Patriots a chance to defend their title from Super Bowl XLIX by de\n",
            "feating them 20–18 in the AFC Championship Game. They joined the Patriots, Dallas Cowboys, and Pittsburgh Steelers as one of four teams that have made\n",
            " eight appearances in the Super Bowl.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "1995 \n",
            "\n",
            "Untokenized Answer :  1995\n",
            "\n",
            "real answer :  [{'answer_start': 291, 'text': '1995'}, {'answer_start': 291, 'text': '1995'}, {'answer_start': 291, 'text': '1995'}]\n",
            "\n",
            "Question :  If Roman numerals were used, what would Super Bowl 50 have been called?\n",
            "--------------------------------------------------\n",
            "Context :  Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football\n",
            " Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24–10 to earn their third Super B\n",
            "owl title. The game was played on February 7, 2016, at Levi's Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th S\n",
            "uper Bowl, the league emphasized the \"golden anniversary\" with various gold-themed initiatives, as well as temporarily suspending the tradition of nam\n",
            "ing each Super Bowl game with Roman numerals (under which the game would have been known as \"Super Bowl L\"), so that the logo could prominently featur\n",
            "e the Arabic numerals 50.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "super bowl l \n",
            "\n",
            "Untokenized Answer :  super bowl l\n",
            "\n",
            "real answer :  [{'answer_start': 693, 'text': 'Super Bowl L'}, {'answer_start': 704, 'text': 'L'}, {'answer_start': 693, 'text': 'Super Bowl L'}]\n",
            "\n",
            "Question :  Who did the Broncos prevent from going to the Super Bowl?\n",
            "--------------------------------------------------\n",
            "Context :  The Panthers finished the regular season with a 15–1 record, and quarterback Cam Newton was named the NFL Most Valuable Player (MVP). They defeated th\n",
            "e Arizona Cardinals 49–15 in the NFC Championship Game and advanced to their second Super Bowl appearance since the franchise was founded in 1995. The\n",
            " Broncos finished the regular season with a 12–4 record, and denied the New England Patriots a chance to defend their title from Super Bowl XLIX by de\n",
            "feating them 20–18 in the AFC Championship Game. They joined the Patriots, Dallas Cowboys, and Pittsburgh Steelers as one of four teams that have made\n",
            " eight appearances in the Super Bowl.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "new engl ##and pat ##riot ##s \n",
            "\n",
            "Untokenized Answer :  new england patriots\n",
            "\n",
            "real answer :  [{'answer_start': 372, 'text': 'New England Patriots'}, {'answer_start': 368, 'text': 'the New England Patriots'}, {'answer_start': 372, 'text': 'New England Patriots'}]\n",
            "\n",
            "Question :  Who won Super Bowl 50?\n",
            "--------------------------------------------------\n",
            "Context :  Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football\n",
            " Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24–10 to earn their third Super B\n",
            "owl title. The game was played on February 7, 2016, at Levi's Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th S\n",
            "uper Bowl, the league emphasized the \"golden anniversary\" with various gold-themed initiatives, as well as temporarily suspending the tradition of nam\n",
            "ing each Super Bowl game with Roman numerals (under which the game would have been known as \"Super Bowl L\"), so that the logo could prominently featur\n",
            "e the Arabic numerals 50.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "den ##ver bron ##cos \n",
            "\n",
            "Untokenized Answer :  denver broncos\n",
            "\n",
            "real answer :  [{'answer_start': 177, 'text': 'Denver Broncos'}, {'answer_start': 177, 'text': 'Denver Broncos'}, {'answer_start': 177, 'text': 'Denver Broncos'}]\n",
            "\n",
            "Question :  What is the AFC short for?\n",
            "--------------------------------------------------\n",
            "Context :  Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football\n",
            " Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24–10 to earn their third Super B\n",
            "owl title. The game was played on February 7, 2016, at Levi's Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th S\n",
            "uper Bowl, the league emphasized the \"golden anniversary\" with various gold-themed initiatives, as well as temporarily suspending the tradition of nam\n",
            "ing each Super Bowl game with Roman numerals (under which the game would have been known as \"Super Bowl L\"), so that the logo could prominently featur\n",
            "e the Arabic numerals 50.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "american football conference \n",
            "\n",
            "Untokenized Answer :  american football conference\n",
            "\n",
            "real answer :  [{'answer_start': 133, 'text': 'American Football Conference'}, {'answer_start': 133, 'text': 'American Football Conference'}, {'answer_start': 133, 'text': 'American Football Conference'}]\n",
            "\n",
            "Question :  Who did Denver beat in the AFC championship?\n",
            "--------------------------------------------------\n",
            "Context :  The Panthers finished the regular season with a 15–1 record, and quarterback Cam Newton was named the NFL Most Valuable Player (MVP). They defeated th\n",
            "e Arizona Cardinals 49–15 in the NFC Championship Game and advanced to their second Super Bowl appearance since the franchise was founded in 1995. The\n",
            " Broncos finished the regular season with a 12–4 record, and denied the New England Patriots a chance to defend their title from Super Bowl XLIX by de\n",
            "feating them 20–18 in the AFC Championship Game. They joined the Patriots, Dallas Cowboys, and Pittsburgh Steelers as one of four teams that have made\n",
            " eight appearances in the Super Bowl.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "ari ##zona cardinal ##s \n",
            "\n",
            "Untokenized Answer :  arizona cardinals\n",
            "\n",
            "real answer :  [{'answer_start': 372, 'text': 'New England Patriots'}, {'answer_start': 368, 'text': 'the New England Patriots'}, {'answer_start': 372, 'text': 'New England Patriots'}]\n",
            "\n",
            "Question :  What band headlined half-time during Super Bowl 50?\n",
            "--------------------------------------------------\n",
            "Context :  CBS broadcast Super Bowl 50 in the U.S., and charged an average of $5 million for a 30-second commercial during the game. The Super Bowl 50 halftime s\n",
            "how was headlined by the British rock group Coldplay with special guest performers Beyoncé and Bruno Mars, who headlined the Super Bowl XLVII and Supe\n",
            "r Bowl XLVIII halftime shows, respectively. It was the third-most watched U.S. broadcast ever.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "cold ##play \n",
            "\n",
            "Untokenized Answer :  coldplay\n",
            "\n",
            "real answer :  [{'answer_start': 194, 'text': 'Coldplay'}, {'answer_start': 194, 'text': 'Coldplay'}, {'answer_start': 194, 'text': 'Coldplay'}]\n",
            "\n",
            "Question :  What was the average cost of a 30-second commercial?\n",
            "--------------------------------------------------\n",
            "Context :  CBS broadcast Super Bowl 50 in the U.S., and charged an average of $5 million for a 30-second commercial during the game. The Super Bowl 50 halftime s\n",
            "how was headlined by the British rock group Coldplay with special guest performers Beyoncé and Bruno Mars, who headlined the Super Bowl XLVII and Supe\n",
            "r Bowl XLVIII halftime shows, respectively. It was the third-most watched U.S. broadcast ever.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "$ 5 million \n",
            "\n",
            "Untokenized Answer :  $ 5 million\n",
            "\n",
            "real answer :  [{'answer_start': 67, 'text': '$5 million'}, {'answer_start': 67, 'text': '$5 million'}, {'answer_start': 67, 'text': '$5 million'}]\n",
            "\n",
            "Question :  How many teams have been in the Super Bowl eight times?\n",
            "--------------------------------------------------\n",
            "Context :  The Panthers finished the regular season with a 15–1 record, and quarterback Cam Newton was named the NFL Most Valuable Player (MVP). They defeated th\n",
            "e Arizona Cardinals 49–15 in the NFC Championship Game and advanced to their second Super Bowl appearance since the franchise was founded in 1995. The\n",
            " Broncos finished the regular season with a 12–4 record, and denied the New England Patriots a chance to defend their title from Super Bowl XLIX by de\n",
            "feating them 20–18 in the AFC Championship Game. They joined the Patriots, Dallas Cowboys, and Pittsburgh Steelers as one of four teams that have made\n",
            " eight appearances in the Super Bowl.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "four \n",
            "\n",
            "Untokenized Answer :  four\n",
            "\n",
            "real answer :  [{'answer_start': 575, 'text': 'four'}, {'answer_start': 575, 'text': 'four'}, {'answer_start': 575, 'text': 'four'}]\n",
            "\n",
            "Question :  Who was the 2015 NFL MVP?\n",
            "--------------------------------------------------\n",
            "Context :  The Panthers finished the regular season with a 15–1 record, and quarterback Cam Newton was named the NFL Most Valuable Player (MVP). They defeated th\n",
            "e Arizona Cardinals 49–15 in the NFC Championship Game and advanced to their second Super Bowl appearance since the franchise was founded in 1995. The\n",
            " Broncos finished the regular season with a 12–4 record, and denied the New England Patriots a chance to defend their title from Super Bowl XLIX by de\n",
            "feating them 20–18 in the AFC Championship Game. They joined the Patriots, Dallas Cowboys, and Pittsburgh Steelers as one of four teams that have made\n",
            " eight appearances in the Super Bowl.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "cam new ##ton \n",
            "\n",
            "Untokenized Answer :  cam newton\n",
            "\n",
            "real answer :  [{'answer_start': 77, 'text': 'Cam Newton'}, {'answer_start': 77, 'text': 'Cam Newton'}, {'answer_start': 77, 'text': 'Cam Newton'}]\n",
            "\n",
            "Question :  Super Bowl 50 decided the NFL champion for what season?\n",
            "--------------------------------------------------\n",
            "Context :  Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football\n",
            " Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24–10 to earn their third Super B\n",
            "owl title. The game was played on February 7, 2016, at Levi's Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th S\n",
            "uper Bowl, the league emphasized the \"golden anniversary\" with various gold-themed initiatives, as well as temporarily suspending the tradition of nam\n",
            "ing each Super Bowl game with Roman numerals (under which the game would have been known as \"Super Bowl L\"), so that the logo could prominently featur\n",
            "e the Arabic numerals 50.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "2015 \n",
            "\n",
            "Untokenized Answer :  2015\n",
            "\n",
            "real answer :  [{'answer_start': 116, 'text': '2015'}, {'answer_start': 112, 'text': 'the 2015 season'}, {'answer_start': 116, 'text': '2015'}]\n",
            "\n",
            "Question :  How many appearances have the Denver Broncos made in the Super Bowl?\n",
            "--------------------------------------------------\n",
            "Context :  The Panthers finished the regular season with a 15–1 record, and quarterback Cam Newton was named the NFL Most Valuable Player (MVP). They defeated th\n",
            "e Arizona Cardinals 49–15 in the NFC Championship Game and advanced to their second Super Bowl appearance since the franchise was founded in 1995. The\n",
            " Broncos finished the regular season with a 12–4 record, and denied the New England Patriots a chance to defend their title from Super Bowl XLIX by de\n",
            "feating them 20–18 in the AFC Championship Game. They joined the Patriots, Dallas Cowboys, and Pittsburgh Steelers as one of four teams that have made\n",
            " eight appearances in the Super Bowl.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "eight \n",
            "\n",
            "Untokenized Answer :  eight\n",
            "\n",
            "real answer :  [{'answer_start': 467, 'text': '8'}, {'answer_start': 601, 'text': 'eight'}, {'answer_start': 601, 'text': 'eight'}]\n",
            "\n",
            "Question :  What city did Super Bowl 50 take place in?\n",
            "--------------------------------------------------\n",
            "Context :  Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football\n",
            " Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24–10 to earn their third Super B\n",
            "owl title. The game was played on February 7, 2016, at Levi's Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th S\n",
            "uper Bowl, the league emphasized the \"golden anniversary\" with various gold-themed initiatives, as well as temporarily suspending the tradition of nam\n",
            "ing each Super Bowl game with Roman numerals (under which the game would have been known as \"Super Bowl L\"), so that the logo could prominently featur\n",
            "e the Arabic numerals 50.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "santa clara , cal ##if ##orn ##ia \n",
            "\n",
            "Untokenized Answer :  santa clara , california\n",
            "\n",
            "real answer :  [{'answer_start': 403, 'text': 'Santa Clara'}, {'answer_start': 403, 'text': 'Santa Clara'}, {'answer_start': 403, 'text': 'Santa Clara'}]\n",
            "\n",
            "Question :  How many fumbles did Von Miller force in Super Bowl 50?\n",
            "--------------------------------------------------\n",
            "Context :  The Broncos took an early lead in Super Bowl 50 and never trailed. Newton was limited by Denver's defense, which sacked him seven times and forced him\n",
            " into three turnovers, including a fumble which they recovered for a touchdown. Denver linebacker Von Miller was named Super Bowl MVP, recording five \n",
            "solo tackles, 2½ sacks, and two forced fumbles.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "two \n",
            "\n",
            "Untokenized Answer :  two\n",
            "\n",
            "real answer :  [{'answer_start': 314, 'text': '2'}, {'answer_start': 328, 'text': 'two'}, {'answer_start': 328, 'text': 'two'}]\n",
            "\n",
            "Question :  How many turnovers did Cam Newton have?\n",
            "--------------------------------------------------\n",
            "Context :  The Broncos took an early lead in Super Bowl 50 and never trailed. Newton was limited by Denver's defense, which sacked him seven times and forced him\n",
            " into three turnovers, including a fumble which they recovered for a touchdown. Denver linebacker Von Miller was named Super Bowl MVP, recording five \n",
            "solo tackles, 2½ sacks, and two forced fumbles.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "three \n",
            "\n",
            "Untokenized Answer :  three\n",
            "\n",
            "real answer :  [{'answer_start': 156, 'text': 'three'}, {'answer_start': 156, 'text': 'three'}, {'answer_start': 156, 'text': 'three'}]\n",
            "\n",
            "Question :  How many fumbles did Von Miller force?\n",
            "--------------------------------------------------\n",
            "Context :  The Broncos took an early lead in Super Bowl 50 and never trailed. Newton was limited by Denver's defense, which sacked him seven times and forced him\n",
            " into three turnovers, including a fumble which they recovered for a touchdown. Denver linebacker Von Miller was named Super Bowl MVP, recording five \n",
            "solo tackles, 2½ sacks, and two forced fumbles.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "two \n",
            "\n",
            "Untokenized Answer :  two\n",
            "\n",
            "real answer :  [{'answer_start': 328, 'text': 'two'}, {'answer_start': 328, 'text': 'two'}, {'answer_start': 328, 'text': 'two'}]\n",
            "\n",
            "Question :  What position does Von Miller play?\n",
            "--------------------------------------------------\n",
            "Context :  The Broncos took an early lead in Super Bowl 50 and never trailed. Newton was limited by Denver's defense, which sacked him seven times and forced him\n",
            " into three turnovers, including a fumble which they recovered for a touchdown. Denver linebacker Von Miller was named Super Bowl MVP, recording five \n",
            "solo tackles, 2½ sacks, and two forced fumbles.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "line ##backer \n",
            "\n",
            "Untokenized Answer :  linebacker\n",
            "\n",
            "real answer :  [{'answer_start': 237, 'text': 'linebacker'}, {'answer_start': 237, 'text': 'linebacker'}, {'answer_start': 237, 'text': 'linebacker'}]\n",
            "\n",
            "Question :  How many times was Cam Newton sacked in Super Bowl 50?\n",
            "--------------------------------------------------\n",
            "Context :  The Broncos took an early lead in Super Bowl 50 and never trailed. Newton was limited by Denver's defense, which sacked him seven times and forced him\n",
            " into three turnovers, including a fumble which they recovered for a touchdown. Denver linebacker Von Miller was named Super Bowl MVP, recording five \n",
            "solo tackles, 2½ sacks, and two forced fumbles.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "seven \n",
            "\n",
            "Untokenized Answer :  seven\n",
            "\n",
            "real answer :  [{'answer_start': 124, 'text': 'seven'}, {'answer_start': 124, 'text': 'seven'}, {'answer_start': 124, 'text': 'seven'}]\n",
            "\n",
            "Question :  Which network broadcasted Super Bowl 50 in the U.S.?\n",
            "--------------------------------------------------\n",
            "Context :  CBS broadcast Super Bowl 50 in the U.S., and charged an average of $5 million for a 30-second commercial during the game. The Super Bowl 50 halftime s\n",
            "how was headlined by the British rock group Coldplay with special guest performers Beyoncé and Bruno Mars, who headlined the Super Bowl XLVII and Supe\n",
            "r Bowl XLVIII halftime shows, respectively. It was the third-most watched U.S. broadcast ever.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "c ##bs \n",
            "\n",
            "Untokenized Answer :  cbs\n",
            "\n",
            "real answer :  [{'answer_start': 0, 'text': 'CBS'}, {'answer_start': 0, 'text': 'CBS'}, {'answer_start': 0, 'text': 'CBS'}]\n",
            "\n",
            "Question :  What month, day and year did Super Bowl 50 take place? \n",
            "--------------------------------------------------\n",
            "Context :  Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football\n",
            " Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24–10 to earn their third Super B\n",
            "owl title. The game was played on February 7, 2016, at Levi's Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th S\n",
            "uper Bowl, the league emphasized the \"golden anniversary\" with various gold-themed initiatives, as well as temporarily suspending the tradition of nam\n",
            "ing each Super Bowl game with Roman numerals (under which the game would have been known as \"Super Bowl L\"), so that the logo could prominently featur\n",
            "e the Arabic numerals 50.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "februar ##y 7 \n",
            "\n",
            "Untokenized Answer :  february 7\n",
            "\n",
            "real answer :  [{'answer_start': 334, 'text': 'February 7, 2016'}, {'answer_start': 334, 'text': 'February 7, 2016'}, {'answer_start': 334, 'text': 'February 7, 2016'}]\n",
            "\n",
            "Question :  What 2015 NFL team one the AFC playoff?\n",
            "--------------------------------------------------\n",
            "Context :  Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football\n",
            " Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24–10 to earn their third Super B\n",
            "owl title. The game was played on February 7, 2016, at Levi's Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th S\n",
            "uper Bowl, the league emphasized the \"golden anniversary\" with various gold-themed initiatives, as well as temporarily suspending the tradition of nam\n",
            "ing each Super Bowl game with Roman numerals (under which the game would have been known as \"Super Bowl L\"), so that the logo could prominently featur\n",
            "e the Arabic numerals 50.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "car ##olina pan ##thers \n",
            "\n",
            "Untokenized Answer :  carolina panthers\n",
            "\n",
            "real answer :  [{'answer_start': 177, 'text': 'Denver Broncos'}, {'answer_start': 177, 'text': 'Denver Broncos'}, {'answer_start': 177, 'text': 'Denver Broncos'}]\n",
            "\n",
            "Question :  How many forced fumbles did Von Miller have during the Super Bowl 50 game?\n",
            "--------------------------------------------------\n",
            "Context :  The Broncos took an early lead in Super Bowl 50 and never trailed. Newton was limited by Denver's defense, which sacked him seven times and forced him\n",
            " into three turnovers, including a fumble which they recovered for a touchdown. Denver linebacker Von Miller was named Super Bowl MVP, recording five \n",
            "solo tackles, 2½ sacks, and two forced fumbles.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "two \n",
            "\n",
            "Untokenized Answer :  two\n",
            "\n",
            "real answer :  [{'answer_start': 314, 'text': '2'}, {'answer_start': 328, 'text': 'two'}, {'answer_start': 328, 'text': 'two'}]\n",
            "\n",
            "Question :  Who won the MVP for the Super Bowl?\n",
            "--------------------------------------------------\n",
            "Context :  The Broncos took an early lead in Super Bowl 50 and never trailed. Newton was limited by Denver's defense, which sacked him seven times and forced him\n",
            " into three turnovers, including a fumble which they recovered for a touchdown. Denver linebacker Von Miller was named Super Bowl MVP, recording five \n",
            "solo tackles, 2½ sacks, and two forced fumbles.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "the bron ##cos \n",
            "\n",
            "Untokenized Answer :  the broncos\n",
            "\n",
            "real answer :  [{'answer_start': 248, 'text': 'Von Miller'}, {'answer_start': 248, 'text': 'Von Miller'}, {'answer_start': 248, 'text': 'Von Miller'}]\n",
            "\n",
            "Question :  Who did Denver beat in the 2015 AFC Championship game?\n",
            "--------------------------------------------------\n",
            "Context :  The Panthers finished the regular season with a 15–1 record, and quarterback Cam Newton was named the NFL Most Valuable Player (MVP). They defeated th\n",
            "e Arizona Cardinals 49–15 in the NFC Championship Game and advanced to their second Super Bowl appearance since the franchise was founded in 1995. The\n",
            " Broncos finished the regular season with a 12–4 record, and denied the New England Patriots a chance to defend their title from Super Bowl XLIX by de\n",
            "feating them 20–18 in the AFC Championship Game. They joined the Patriots, Dallas Cowboys, and Pittsburgh Steelers as one of four teams that have made\n",
            " eight appearances in the Super Bowl.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "ari ##zona cardinal ##s \n",
            "\n",
            "Untokenized Answer :  arizona cardinals\n",
            "\n",
            "real answer :  [{'answer_start': 372, 'text': 'New England Patriots'}, {'answer_start': 368, 'text': 'the New England Patriots'}, {'answer_start': 372, 'text': 'New England Patriots'}]\n",
            "\n",
            "Question :  What venue did Super Bowl 50 take place in?\n",
            "--------------------------------------------------\n",
            "Context :  Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football\n",
            " Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24–10 to earn their third Super B\n",
            "owl title. The game was played on February 7, 2016, at Levi's Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th S\n",
            "uper Bowl, the league emphasized the \"golden anniversary\" with various gold-themed initiatives, as well as temporarily suspending the tradition of nam\n",
            "ing each Super Bowl game with Roman numerals (under which the game would have been known as \"Super Bowl L\"), so that the logo could prominently featur\n",
            "e the Arabic numerals 50.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "le ##vi ' s stadium \n",
            "\n",
            "Untokenized Answer :  levi ' s stadium\n",
            "\n",
            "real answer :  [{'answer_start': 355, 'text': \"Levi's Stadium\"}, {'answer_start': 355, 'text': \"Levi's Stadium\"}, {'answer_start': 355, 'text': \"Levi's Stadium in the San Francisco Bay Area at Santa Clara\"}]\n",
            "\n",
            "Question :  What ranking does the Super Bowl 50 halftime show have on the list of most watched TV broadcasts?\n",
            "--------------------------------------------------\n",
            "Context :  CBS broadcast Super Bowl 50 in the U.S., and charged an average of $5 million for a 30-second commercial during the game. The Super Bowl 50 halftime s\n",
            "how was headlined by the British rock group Coldplay with special guest performers Beyoncé and Bruno Mars, who headlined the Super Bowl XLVII and Supe\n",
            "r Bowl XLVIII halftime shows, respectively. It was the third-most watched U.S. broadcast ever.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "third \n",
            "\n",
            "Untokenized Answer :  third\n",
            "\n",
            "real answer :  [{'answer_start': 355, 'text': 'third'}, {'answer_start': 355, 'text': 'third'}, {'answer_start': 355, 'text': 'third'}]\n",
            "\n",
            "Question :  What stadium did Super Bowl 50 take place in?\n",
            "--------------------------------------------------\n",
            "Context :  Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football\n",
            " Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24–10 to earn their third Super B\n",
            "owl title. The game was played on February 7, 2016, at Levi's Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th S\n",
            "uper Bowl, the league emphasized the \"golden anniversary\" with various gold-themed initiatives, as well as temporarily suspending the tradition of nam\n",
            "ing each Super Bowl game with Roman numerals (under which the game would have been known as \"Super Bowl L\"), so that the logo could prominently featur\n",
            "e the Arabic numerals 50.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "le ##vi ' s stadium \n",
            "\n",
            "Untokenized Answer :  levi ' s stadium\n",
            "\n",
            "real answer :  [{'answer_start': 355, 'text': \"Levi's Stadium\"}, {'answer_start': 355, 'text': \"Levi's Stadium\"}, {'answer_start': 355, 'text': \"Levi's Stadium\"}]\n",
            "\n",
            "Question :  How many times was the Panthers' quarterback sacked?\n",
            "--------------------------------------------------\n",
            "Context :  The Broncos took an early lead in Super Bowl 50 and never trailed. Newton was limited by Denver's defense, which sacked him seven times and forced him\n",
            " into three turnovers, including a fumble which they recovered for a touchdown. Denver linebacker Von Miller was named Super Bowl MVP, recording five \n",
            "solo tackles, 2½ sacks, and two forced fumbles.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "seven \n",
            "\n",
            "Untokenized Answer :  seven\n",
            "\n",
            "real answer :  [{'answer_start': 124, 'text': 'seven'}, {'answer_start': 124, 'text': 'seven'}, {'answer_start': 124, 'text': 'seven'}]\n",
            "\n",
            "Question :  What two artists came out with Coldplay during the half-time show?\n",
            "--------------------------------------------------\n",
            "Context :  CBS broadcast Super Bowl 50 in the U.S., and charged an average of $5 million for a 30-second commercial during the game. The Super Bowl 50 halftime s\n",
            "how was headlined by the British rock group Coldplay with special guest performers Beyoncé and Bruno Mars, who headlined the Super Bowl XLVII and Supe\n",
            "r Bowl XLVIII halftime shows, respectively. It was the third-most watched U.S. broadcast ever.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "be ##yon ##ce and brun ##o mars \n",
            "\n",
            "Untokenized Answer :  beyonce and bruno mars\n",
            "\n",
            "real answer :  [{'answer_start': 233, 'text': 'Beyoncé and Bruno Mars'}, {'answer_start': 233, 'text': 'Beyoncé and Bruno Mars'}, {'answer_start': 233, 'text': 'Beyoncé and Bruno Mars'}]\n",
            "\n",
            "Question :  What day was the Super Bowl played on?\n",
            "--------------------------------------------------\n",
            "Context :  Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football\n",
            " Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24–10 to earn their third Super B\n",
            "owl title. The game was played on February 7, 2016, at Levi's Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th S\n",
            "uper Bowl, the league emphasized the \"golden anniversary\" with various gold-themed initiatives, as well as temporarily suspending the tradition of nam\n",
            "ing each Super Bowl game with Roman numerals (under which the game would have been known as \"Super Bowl L\"), so that the logo could prominently featur\n",
            "e the Arabic numerals 50.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "februar ##y 7 , 2016 \n",
            "\n",
            "Untokenized Answer :  february 7 , 2016\n",
            "\n",
            "real answer :  [{'answer_start': 334, 'text': 'February 7, 2016'}, {'answer_start': 334, 'text': 'February 7'}, {'answer_start': 334, 'text': 'February 7, 2016'}]\n",
            "\n",
            "Question :  Which performers joined the headliner during the Super Bowl 50 halftime show?\n",
            "--------------------------------------------------\n",
            "Context :  CBS broadcast Super Bowl 50 in the U.S., and charged an average of $5 million for a 30-second commercial during the game. The Super Bowl 50 halftime s\n",
            "how was headlined by the British rock group Coldplay with special guest performers Beyoncé and Bruno Mars, who headlined the Super Bowl XLVII and Supe\n",
            "r Bowl XLVIII halftime shows, respectively. It was the third-most watched U.S. broadcast ever.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "be ##yon ##ce and brun ##o mars \n",
            "\n",
            "Untokenized Answer :  beyonce and bruno mars\n",
            "\n",
            "real answer :  [{'answer_start': 233, 'text': 'Beyoncé and Bruno Mars'}, {'answer_start': 233, 'text': 'Beyoncé and Bruno Mars'}, {'answer_start': 233, 'text': 'Beyoncé and Bruno Mars'}]\n",
            "\n",
            "Question :  Where was Super Bowl 50 held?\n",
            "--------------------------------------------------\n",
            "Context :  Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football\n",
            " Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24–10 to earn their third Super B\n",
            "owl title. The game was played on February 7, 2016, at Levi's Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th S\n",
            "uper Bowl, the league emphasized the \"golden anniversary\" with various gold-themed initiatives, as well as temporarily suspending the tradition of nam\n",
            "ing each Super Bowl game with Roman numerals (under which the game would have been known as \"Super Bowl L\"), so that the logo could prominently featur\n",
            "e the Arabic numerals 50.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "le ##vi ' s stadium in the san franc ##is ##co bay area at santa clara , cal ##if ##orn ##ia \n",
            "\n",
            "Untokenized Answer :  levi ' s stadium in the san francisco bay area at santa clara , california\n",
            "\n",
            "real answer :  [{'answer_start': 403, 'text': 'Santa Clara, California.'}, {'answer_start': 355, 'text': \"Levi's Stadium\"}, {'answer_start': 355, 'text': \"Levi's Stadium\"}]\n",
            "\n",
            "Question :  Who was the Super Bowl 50 MVP?\n",
            "--------------------------------------------------\n",
            "Context :  The Broncos took an early lead in Super Bowl 50 and never trailed. Newton was limited by Denver's defense, which sacked him seven times and forced him\n",
            " into three turnovers, including a fumble which they recovered for a touchdown. Denver linebacker Von Miller was named Super Bowl MVP, recording five \n",
            "solo tackles, 2½ sacks, and two forced fumbles.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "von mille ##r \n",
            "\n",
            "Untokenized Answer :  von miller\n",
            "\n",
            "real answer :  [{'answer_start': 248, 'text': 'Von Miller'}, {'answer_start': 248, 'text': 'Von Miller'}, {'answer_start': 252, 'text': 'Miller'}]\n",
            "\n",
            "Question :  What was the win/loss ratio in 2015 for the Carolina Panthers during their regular season?\n",
            "--------------------------------------------------\n",
            "Context :  The Panthers finished the regular season with a 15–1 record, and quarterback Cam Newton was named the NFL Most Valuable Player (MVP). They defeated th\n",
            "e Arizona Cardinals 49–15 in the NFC Championship Game and advanced to their second Super Bowl appearance since the franchise was founded in 1995. The\n",
            " Broncos finished the regular season with a 12–4 record, and denied the New England Patriots a chance to defend their title from Super Bowl XLIX by de\n",
            "feating them 20–18 in the AFC Championship Game. They joined the Patriots, Dallas Cowboys, and Pittsburgh Steelers as one of four teams that have made\n",
            " eight appearances in the Super Bowl.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "15 [UNK] 1 \n",
            "\n",
            "Untokenized Answer :  15 [UNK] 1\n",
            "\n",
            "real answer :  [{'answer_start': 48, 'text': '15–1'}, {'answer_start': 48, 'text': '15–1'}, {'answer_start': 48, 'text': '15–1'}]\n",
            "\n",
            "Question :  Which NFL team won Super Bowl 50?\n",
            "--------------------------------------------------\n",
            "Context :  Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football\n",
            " Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24–10 to earn their third Super B\n",
            "owl title. The game was played on February 7, 2016, at Levi's Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th S\n",
            "uper Bowl, the league emphasized the \"golden anniversary\" with various gold-themed initiatives, as well as temporarily suspending the tradition of nam\n",
            "ing each Super Bowl game with Roman numerals (under which the game would have been known as \"Super Bowl L\"), so that the logo could prominently featur\n",
            "e the Arabic numerals 50.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "car ##olina pan ##thers \n",
            "\n",
            "Untokenized Answer :  carolina panthers\n",
            "\n",
            "real answer :  [{'answer_start': 177, 'text': 'Denver Broncos'}, {'answer_start': 177, 'text': 'Denver Broncos'}, {'answer_start': 177, 'text': 'Denver Broncos'}]\n",
            "\n",
            "Question :  At which Super Bowl did Beyonce headline the halftime show?\n",
            "--------------------------------------------------\n",
            "Context :  CBS broadcast Super Bowl 50 in the U.S., and charged an average of $5 million for a 30-second commercial during the game. The Super Bowl 50 halftime s\n",
            "how was headlined by the British rock group Coldplay with special guest performers Beyoncé and Bruno Mars, who headlined the Super Bowl XLVII and Supe\n",
            "r Bowl XLVIII halftime shows, respectively. It was the third-most watched U.S. broadcast ever.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "x ##lv ##ii \n",
            "\n",
            "Untokenized Answer :  xlvii\n",
            "\n",
            "real answer :  [{'answer_start': 275, 'text': 'Super Bowl XLVII'}, {'answer_start': 275, 'text': 'Super Bowl XLVII'}, {'answer_start': 286, 'text': 'XLVII'}]\n",
            "\n",
            "Question :  What was the number of solo tackles that Von Miller had in Super Bowl 50?\n",
            "--------------------------------------------------\n",
            "Context :  The Broncos took an early lead in Super Bowl 50 and never trailed. Newton was limited by Denver's defense, which sacked him seven times and forced him\n",
            " into three turnovers, including a fumble which they recovered for a touchdown. Denver linebacker Von Miller was named Super Bowl MVP, recording five \n",
            "solo tackles, 2½ sacks, and two forced fumbles.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "five \n",
            "\n",
            "Untokenized Answer :  five\n",
            "\n",
            "real answer :  [{'answer_start': 45, 'text': '5'}, {'answer_start': 295, 'text': 'five'}, {'answer_start': 295, 'text': 'five'}]\n",
            "\n",
            "Question :  Who was the broadcaster for Super Bowl 50 in the United States?\n",
            "--------------------------------------------------\n",
            "Context :  CBS broadcast Super Bowl 50 in the U.S., and charged an average of $5 million for a 30-second commercial during the game. The Super Bowl 50 halftime s\n",
            "how was headlined by the British rock group Coldplay with special guest performers Beyoncé and Bruno Mars, who headlined the Super Bowl XLVII and Supe\n",
            "r Bowl XLVIII halftime shows, respectively. It was the third-most watched U.S. broadcast ever.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "c ##bs \n",
            "\n",
            "Untokenized Answer :  cbs\n",
            "\n",
            "real answer :  [{'answer_start': 0, 'text': 'CBS'}, {'answer_start': 0, 'text': 'CBS'}, {'answer_start': 0, 'text': 'CBS'}]\n",
            "\n",
            "Question :  What does AFC stand for?\n",
            "--------------------------------------------------\n",
            "Context :  Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football\n",
            " Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24–10 to earn their third Super B\n",
            "owl title. The game was played on February 7, 2016, at Levi's Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th S\n",
            "uper Bowl, the league emphasized the \"golden anniversary\" with various gold-themed initiatives, as well as temporarily suspending the tradition of nam\n",
            "ing each Super Bowl game with Roman numerals (under which the game would have been known as \"Super Bowl L\"), so that the logo could prominently featur\n",
            "e the Arabic numerals 50.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "american football conference \n",
            "\n",
            "Untokenized Answer :  american football conference\n",
            "\n",
            "real answer :  [{'answer_start': 133, 'text': 'American Football Conference'}, {'answer_start': 133, 'text': 'American Football Conference'}, {'answer_start': 133, 'text': 'American Football Conference'}]\n",
            "\n",
            "Question :  Who was the Most Valuable Player of Super Bowl 50?\n",
            "--------------------------------------------------\n",
            "Context :  The Broncos took an early lead in Super Bowl 50 and never trailed. Newton was limited by Denver's defense, which sacked him seven times and forced him\n",
            " into three turnovers, including a fumble which they recovered for a touchdown. Denver linebacker Von Miller was named Super Bowl MVP, recording five \n",
            "solo tackles, 2½ sacks, and two forced fumbles.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "von mille ##r \n",
            "\n",
            "Untokenized Answer :  von miller\n",
            "\n",
            "real answer :  [{'answer_start': 248, 'text': 'Von Miller'}, {'answer_start': 248, 'text': 'Von Miller'}, {'answer_start': 248, 'text': 'Von Miller'}]\n",
            "\n",
            "Question :  What halftime performer previously headlined Super Bowl XLVII?\n",
            "--------------------------------------------------\n",
            "Context :  CBS broadcast Super Bowl 50 in the U.S., and charged an average of $5 million for a 30-second commercial during the game. The Super Bowl 50 halftime s\n",
            "how was headlined by the British rock group Coldplay with special guest performers Beyoncé and Bruno Mars, who headlined the Super Bowl XLVII and Supe\n",
            "r Bowl XLVIII halftime shows, respectively. It was the third-most watched U.S. broadcast ever.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "brun ##o mars \n",
            "\n",
            "Untokenized Answer :  bruno mars\n",
            "\n",
            "real answer :  [{'answer_start': 233, 'text': 'Beyoncé'}, {'answer_start': 233, 'text': 'Beyoncé'}, {'answer_start': 233, 'text': 'Beyoncé'}]\n",
            "\n",
            "Question :  Which group headlined the Super Bowl 50 halftime show?\n",
            "--------------------------------------------------\n",
            "Context :  CBS broadcast Super Bowl 50 in the U.S., and charged an average of $5 million for a 30-second commercial during the game. The Super Bowl 50 halftime s\n",
            "how was headlined by the British rock group Coldplay with special guest performers Beyoncé and Bruno Mars, who headlined the Super Bowl XLVII and Supe\n",
            "r Bowl XLVIII halftime shows, respectively. It was the third-most watched U.S. broadcast ever.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "cold ##play \n",
            "\n",
            "Untokenized Answer :  coldplay\n",
            "\n",
            "real answer :  [{'answer_start': 194, 'text': 'Coldplay'}, {'answer_start': 194, 'text': 'Coldplay'}, {'answer_start': 194, 'text': 'Coldplay'}]\n",
            "\n",
            "Question :  What were the win/loss game stats for the Denver Bronco's regular season in 2015?\n",
            "--------------------------------------------------\n",
            "Context :  The Panthers finished the regular season with a 15–1 record, and quarterback Cam Newton was named the NFL Most Valuable Player (MVP). They defeated th\n",
            "e Arizona Cardinals 49–15 in the NFC Championship Game and advanced to their second Super Bowl appearance since the franchise was founded in 1995. The\n",
            " Broncos finished the regular season with a 12–4 record, and denied the New England Patriots a chance to defend their title from Super Bowl XLIX by de\n",
            "feating them 20–18 in the AFC Championship Game. They joined the Patriots, Dallas Cowboys, and Pittsburgh Steelers as one of four teams that have made\n",
            " eight appearances in the Super Bowl.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "12 [UNK] 4 \n",
            "\n",
            "Untokenized Answer :  12 [UNK] 4\n",
            "\n",
            "real answer :  [{'answer_start': 344, 'text': '12–4'}, {'answer_start': 344, 'text': '12–4'}, {'answer_start': 344, 'text': '12–4'}]\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    }
  ]
}