冒險25_打造自己的Tokenizer(文字型資料的處理).ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "冒險25_打造自己的 Tokenizer (文字型資料的處理)",
      "provenance": [],
      "collapsed_sections": [],
      "authorship_tag": "ABX9TyNhDWCRoTgbhL10SeFpXK6f",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/yenlung/Python-AI-Book/blob/main/%E5%86%92%E9%9A%AA25_%E6%89%93%E9%80%A0%E8%87%AA%E5%B7%B1%E7%9A%84Tokenizer(%E6%96%87%E5%AD%97%E5%9E%8B%E8%B3%87%E6%96%99%E7%9A%84%E8%99%95%E7%90%86).ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "1Zwoi-DYDR-g"
      },
      "outputs": [],
      "source": [
        "%matplotlib inline\n",
        "\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "import matplotlib.pyplot as plt"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oxc6kSls4X4t"
      },
      "source": [
        "### 2. 讀入套件\n",
        "\n",
        "我們最主要是需要 `tf.keras` 中的 `Tokenizer`。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xyV0Igeq5-OE"
      },
      "source": [
        "from tensorflow.keras.preprocessing.text import Tokenizer\n",
        "import pickle"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "!wget --no-check-certificate \\\n",
        "    https://raw.githubusercontent.com/yenlung/Python-AI-Book/main/dream.txt \\\n",
        "    -O /content/dream.txt"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "OSJ-KgLazFVW",
        "outputId": "1d1ba482-0aed-461d-c796-23c82ddd7a4c"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "--2022-04-05 03:53:10--  https://raw.githubusercontent.com/yenlung/Python-AI-Book/main/dream.txt\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.108.133, 185.199.109.133, 185.199.110.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 2656053 (2.5M) [text/plain]\n",
            "Saving to: ‘/content/dream.txt’\n",
            "\n",
            "/content/dream.txt  100%[===================>]   2.53M  --.-KB/s    in 0.04s   \n",
            "\n",
            "2022-04-05 03:53:11 (60.0 MB/s) - ‘/content/dream.txt’ saved [2656053/2656053]\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 3. 讀入蒐集到的所有文本"
      ],
      "metadata": {
        "id": "Fo0KP2wz1c2N"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "f = open('dream.txt', 'r')\n",
        "lines = f.readlines()\n",
        "f.close()"
      ],
      "metadata": {
        "id": "myC1Bfdx1crV"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "text_lines = [x.lstrip('\\u3000\\u3000') for x in lines]"
      ],
      "metadata": {
        "id": "AmHGTXJE1akG"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "text = ''.join(text_lines)"
      ],
      "metadata": {
        "id": "tOjCQNQcz7Ia"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 4. 打造自己的 Tokenizer"
      ],
      "metadata": {
        "id": "_vFZp5Z411AU"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "tokenizer = Tokenizer(char_level=True)"
      ],
      "metadata": {
        "id": "3LCDyAFP1zSW"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "tokenizer.fit_on_texts([text])"
      ],
      "metadata": {
        "id": "2b1yPEb415SH"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "tokenizer.texts_to_sequences([\"我打造了一個函數學習機。\"])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "DZ4CtB752IZO",
        "outputId": "e3c13651-6a3e-4d9f-b5d5-dc0209c4c40d"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[[15, 99, 721, 3, 6, 26, 597, 362, 1061, 912, 2]]"
            ]
          },
          "metadata": {},
          "execution_count": 20
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "tokenizer.sequences_to_texts([[15, 99, 721, 3, 6, 26, 597, 362, 1061, 912, 2]])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "RgFlpIhg2Jmo",
        "outputId": "e26c3584-2efb-4116-8d1e-ac8b7f5cdb30"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['我 打 造 了 一 個 數 學 習 機 。']"
            ]
          },
          "metadata": {},
          "execution_count": 21
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "###5. 把Tokenizer存起來"
      ],
      "metadata": {
        "id": "Z5Uq6JYD3X2C"
      }
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "bWUh9zFEFi2F",
        "outputId": "a6ed39e0-e02d-4037-cb11-e7d8606479cc"
      },
      "source": [
        "from google.colab import drive\n",
        "\n",
        "drive.mount('/content/drive')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Mounted at /content/drive\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "e_4lNrxlF_M_",
        "outputId": "85174214-73b4-48f9-940b-ce975a675aef"
      },
      "source": [
        "%cd \"/content/drive/MyDrive/Colab Notebooks/\""
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/content/drive/MyDrive/Colab Notebooks\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "igw9qiUSGM77"
      },
      "source": [
        "f = open('MyTokenizer.pkl', 'wb')\n",
        "pickle.dump(tokenizer, f)\n",
        "f.close()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "之後要用 `pickle` 讀回我們訓練好的 tokenizer 是這樣:\n",
        "\n",
        "```python\n",
        "f = open('tokenizer.pkl', 'rb')\n",
        "tokenizer = pickle.load(f)\n",
        "f.close()\n",
        "```"
      ],
      "metadata": {
        "id": "4pkAc8tD3m3j"
      }
    }
  ]
}