GPT2_IndiaSpeaks.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "GPT2 IndiaSpeaks.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "mount_file_id": "1TPgsHyavUQ-wNqDDtncu-mKjAekT-1PD",
      "authorship_tag": "ABX9TyPkZBmdfDP4rUoQ9TXuGerc",
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/masala-man/gpt2-indiaspeaks/blob/main/GPT2_IndiaSpeaks.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QZY8Nuik1DxX"
      },
      "source": [
        "\n",
        "#**DIY IndiaSpeaks**\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "J7-s2uUyFwqD"
      },
      "source": [
        "## 0. Setup Environment\n",
        "\n",
        "Install all the packages we need and mount Google Drive and display the GPU allocated to us\n",
        "\n",
        "If you're using Colab for the first time, make sure GPU acceleration is turned on and that you've pressed the Google Drive button in the Files section of the sidebar to provide access to your Drive.\n",
        "\n",
        "(`Runtime > Change runtime type > GPU` to enable acceleration)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "cellView": "code",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "OUqZad1r16Nc",
        "outputId": "0a419ddd-182f-4b4f-b0f4-6ae730bd74a3"
      },
      "source": [
        "# Install necessary packages\n",
        "!pip install git+https://github.com/huggingface/transformers\n",
        "!pip install datasets\n",
        "!pip install pushshift.py\n",
        "!pip install numpy\n",
        "!pip install pandas\n",
        "!pip install redditcleaner\n",
        "\n",
        "import tensorflow as tf\n",
        "from google.colab import drive\n",
        "\n",
        "# Mount Google Drive\n",
        "drive.mount('drive')\n",
        "\n",
        "# Fetch training script\n",
        "!wget wget https://raw.githubusercontent.com/huggingface/transformers/master/examples/language-modeling/run_clm.py\n",
        "\n",
        "# Display graphics card details\n",
        "!nvidia-smi"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting git+https://github.com/huggingface/transformers\n",
            "  Cloning https://github.com/huggingface/transformers to /tmp/pip-req-build-sw1u2tho\n",
            "  Running command git clone -q https://github.com/huggingface/transformers /tmp/pip-req-build-sw1u2tho\n",
            "  Installing build dependencies ... \u001b[?25l\u001b[?25hdone\n",
            "  Getting requirements to build wheel ... \u001b[?25l\u001b[?25hdone\n",
            "    Preparing wheel metadata ... \u001b[?25l\u001b[?25hdone\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from transformers==4.6.0.dev0) (2.23.0)\n",
            "Collecting sacremoses\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/08/cd/342e584ee544d044fb573ae697404ce22ede086c9e87ce5960772084cad0/sacremoses-0.0.44.tar.gz (862kB)\n",
            "\u001b[K     |████████████████████████████████| 870kB 5.8MB/s \n",
            "\u001b[?25hRequirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.7/dist-packages (from transformers==4.6.0.dev0) (2019.12.20)\n",
            "Requirement already satisfied: importlib-metadata; python_version < \"3.8\" in /usr/local/lib/python3.7/dist-packages (from transformers==4.6.0.dev0) (3.8.1)\n",
            "Collecting tokenizers<0.11,>=0.10.1\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/ae/04/5b870f26a858552025a62f1649c20d29d2672c02ff3c3fb4c688ca46467a/tokenizers-0.10.2-cp37-cp37m-manylinux2010_x86_64.whl (3.3MB)\n",
            "\u001b[K     |████████████████████████████████| 3.3MB 34.5MB/s \n",
            "\u001b[?25hRequirement already satisfied: packaging in /usr/local/lib/python3.7/dist-packages (from transformers==4.6.0.dev0) (20.9)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.7/dist-packages (from transformers==4.6.0.dev0) (3.0.12)\n",
            "Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.7/dist-packages (from transformers==4.6.0.dev0) (4.41.1)\n",
            "Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.7/dist-packages (from transformers==4.6.0.dev0) (1.19.5)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests->transformers==4.6.0.dev0) (1.24.3)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests->transformers==4.6.0.dev0) (3.0.4)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests->transformers==4.6.0.dev0) (2.10)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests->transformers==4.6.0.dev0) (2020.12.5)\n",
            "Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from sacremoses->transformers==4.6.0.dev0) (1.15.0)\n",
            "Requirement already satisfied: click in /usr/local/lib/python3.7/dist-packages (from sacremoses->transformers==4.6.0.dev0) (7.1.2)\n",
            "Requirement already satisfied: joblib in /usr/local/lib/python3.7/dist-packages (from sacremoses->transformers==4.6.0.dev0) (1.0.1)\n",
            "Requirement already satisfied: typing-extensions>=3.6.4; python_version < \"3.8\" in /usr/local/lib/python3.7/dist-packages (from importlib-metadata; python_version < \"3.8\"->transformers==4.6.0.dev0) (3.7.4.3)\n",
            "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata; python_version < \"3.8\"->transformers==4.6.0.dev0) (3.4.1)\n",
            "Requirement already satisfied: pyparsing>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from packaging->transformers==4.6.0.dev0) (2.4.7)\n",
            "Building wheels for collected packages: transformers\n",
            "  Building wheel for transformers (PEP 517) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for transformers: filename=transformers-4.6.0.dev0-cp37-none-any.whl size=2090005 sha256=b29a41386c1cba216a210567c85e8b99fb59a352d1cc7283b2f5a226934bfc0b\n",
            "  Stored in directory: /tmp/pip-ephem-wheel-cache-hmxifg4p/wheels/70/d3/52/b3fa4f8b8ef04167ac62e5bb2accb62ae764db2a378247490e\n",
            "Successfully built transformers\n",
            "Building wheels for collected packages: sacremoses\n",
            "  Building wheel for sacremoses (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for sacremoses: filename=sacremoses-0.0.44-cp37-none-any.whl size=886084 sha256=d1d1ca6aef1bc31bb17e7055420800d001fca3d19bbb372616c1ca0a15d1914b\n",
            "  Stored in directory: /root/.cache/pip/wheels/3e/fb/c0/13ab4d63d537658f448366744654323077c4d90069b6512f3c\n",
            "Successfully built sacremoses\n",
            "Installing collected packages: sacremoses, tokenizers, transformers\n",
            "Successfully installed sacremoses-0.0.44 tokenizers-0.10.2 transformers-4.6.0.dev0\n",
            "Collecting datasets\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/54/90/43b396481a8298c6010afb93b3c1e71d4ba6f8c10797a7da8eb005e45081/datasets-1.5.0-py3-none-any.whl (192kB)\n",
            "\u001b[K     |████████████████████████████████| 194kB 3.9MB/s \n",
            "\u001b[?25hRequirement already satisfied: dill in /usr/local/lib/python3.7/dist-packages (from datasets) (0.3.3)\n",
            "Collecting huggingface-hub<0.1.0\n",
            "  Downloading https://files.pythonhosted.org/packages/a1/88/7b1e45720ecf59c6c6737ff332f41c955963090a18e72acbcbeac6b25e86/huggingface_hub-0.0.8-py3-none-any.whl\n",
            "Requirement already satisfied: importlib-metadata; python_version < \"3.8\" in /usr/local/lib/python3.7/dist-packages (from datasets) (3.8.1)\n",
            "Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.7/dist-packages (from datasets) (1.19.5)\n",
            "Collecting fsspec\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/62/11/f7689b996f85e45f718745c899f6747ee5edb4878cadac0a41ab146828fa/fsspec-0.9.0-py3-none-any.whl (107kB)\n",
            "\u001b[K     |████████████████████████████████| 112kB 6.5MB/s \n",
            "\u001b[?25hRequirement already satisfied: requests>=2.19.0 in /usr/local/lib/python3.7/dist-packages (from datasets) (2.23.0)\n",
            "Requirement already satisfied: pyarrow>=0.17.1 in /usr/local/lib/python3.7/dist-packages (from datasets) (3.0.0)\n",
            "Collecting xxhash\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/e7/27/1c0b37c53a7852f1c190ba5039404d27b3ae96a55f48203a74259f8213c9/xxhash-2.0.0-cp37-cp37m-manylinux2010_x86_64.whl (243kB)\n",
            "\u001b[K     |████████████████████████████████| 245kB 6.8MB/s \n",
            "\u001b[?25hRequirement already satisfied: pandas in /usr/local/lib/python3.7/dist-packages (from datasets) (1.1.5)\n",
            "Requirement already satisfied: multiprocess in /usr/local/lib/python3.7/dist-packages (from datasets) (0.70.11.1)\n",
            "Requirement already satisfied: tqdm<4.50.0,>=4.27 in /usr/local/lib/python3.7/dist-packages (from datasets) (4.41.1)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.7/dist-packages (from huggingface-hub<0.1.0->datasets) (3.0.12)\n",
            "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata; python_version < \"3.8\"->datasets) (3.4.1)\n",
            "Requirement already satisfied: typing-extensions>=3.6.4; python_version < \"3.8\" in /usr/local/lib/python3.7/dist-packages (from importlib-metadata; python_version < \"3.8\"->datasets) (3.7.4.3)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests>=2.19.0->datasets) (1.24.3)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests>=2.19.0->datasets) (3.0.4)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests>=2.19.0->datasets) (2.10)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests>=2.19.0->datasets) (2020.12.5)\n",
            "Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.7/dist-packages (from pandas->datasets) (2.8.1)\n",
            "Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.7/dist-packages (from pandas->datasets) (2018.9)\n",
            "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.7/dist-packages (from python-dateutil>=2.7.3->pandas->datasets) (1.15.0)\n",
            "Installing collected packages: huggingface-hub, fsspec, xxhash, datasets\n",
            "Successfully installed datasets-1.5.0 fsspec-0.9.0 huggingface-hub-0.0.8 xxhash-2.0.0\n",
            "Collecting pushshift.py\n",
            "  Downloading https://files.pythonhosted.org/packages/3c/ae/81c3ba16c944c3ffaa11cc1d7aed7304dc84b6440627e64034557aa4c9e0/pushshift.py-0.1.2-py3-none-any.whl\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from pushshift.py) (2.23.0)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests->pushshift.py) (3.0.4)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests->pushshift.py) (2020.12.5)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests->pushshift.py) (1.24.3)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests->pushshift.py) (2.10)\n",
            "Installing collected packages: pushshift.py\n",
            "Successfully installed pushshift.py-0.1.2\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (1.19.5)\n",
            "Requirement already satisfied: pandas in /usr/local/lib/python3.7/dist-packages (1.1.5)\n",
            "Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.7/dist-packages (from pandas) (2018.9)\n",
            "Requirement already satisfied: numpy>=1.15.4 in /usr/local/lib/python3.7/dist-packages (from pandas) (1.19.5)\n",
            "Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.7/dist-packages (from pandas) (2.8.1)\n",
            "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.7/dist-packages (from python-dateutil>=2.7.3->pandas) (1.15.0)\n",
            "Collecting redditcleaner\n",
            "  Downloading https://files.pythonhosted.org/packages/f9/8a/7491757daaf8f3381f736473018880c9e89defd44b9ebbf48a83c172e5ff/redditcleaner-1.1.2-py3-none-any.whl\n",
            "Installing collected packages: redditcleaner\n",
            "Successfully installed redditcleaner-1.1.2\n",
            "Drive already mounted at drive; to attempt to forcibly remount, call drive.mount(\"drive\", force_remount=True).\n",
            "--2021-04-10 06:56:42--  http://wget/\n",
            "Resolving wget (wget)... failed: Name or service not known.\n",
            "wget: unable to resolve host address ‘wget’\n",
            "--2021-04-10 06:56:42--  https://raw.githubusercontent.com/huggingface/transformers/master/examples/language-modeling/run_clm.py\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.109.133, 185.199.110.133, 185.199.111.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.109.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 19385 (19K) [text/plain]\n",
            "Saving to: ‘run_clm.py’\n",
            "\n",
            "run_clm.py          100%[===================>]  18.93K  --.-KB/s    in 0s      \n",
            "\n",
            "2021-04-10 06:56:42 (42.1 MB/s) - ‘run_clm.py’ saved [19385/19385]\n",
            "\n",
            "FINISHED --2021-04-10 06:56:42--\n",
            "Total wall clock time: 0.4s\n",
            "Downloaded: 1 files, 19K in 0s (42.1 MB/s)\n",
            "Sat Apr 10 06:56:43 2021       \n",
            "+-----------------------------------------------------------------------------+\n",
            "| NVIDIA-SMI 460.67       Driver Version: 460.32.03    CUDA Version: 11.2     |\n",
            "|-------------------------------+----------------------+----------------------+\n",
            "| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\n",
            "| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\n",
            "|                               |                      |               MIG M. |\n",
            "|===============================+======================+======================|\n",
            "|   0  Tesla K80           Off  | 00000000:00:04.0 Off |                    0 |\n",
            "| N/A   51C    P8    31W / 149W |      0MiB / 11441MiB |      0%      Default |\n",
            "|                               |                      |                  N/A |\n",
            "+-------------------------------+----------------------+----------------------+\n",
            "                                                                               \n",
            "+-----------------------------------------------------------------------------+\n",
            "| Processes:                                                                  |\n",
            "|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |\n",
            "|        ID   ID                                                   Usage      |\n",
            "|=============================================================================|\n",
            "|  No running processes found                                                 |\n",
            "+-----------------------------------------------------------------------------+\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1rsM5cFQKAAk"
      },
      "source": [
        "## 1. Gather Data\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2XZuKsvzHLL5"
      },
      "source": [
        "### Scraping\n",
        "\n",
        "Get posts from reddit. Use pushshift because the reddit API sucks."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0x6gGr_Fp3rS"
      },
      "source": [
        "from pushshift_py import PushshiftAPI\n",
        "import csv\n",
        "import os\n",
        "import datetime as dt\n",
        "import pandas as pd\n",
        "import numpy as np\n",
        "import json \n",
        "\n",
        "########CONFIG########\n",
        "sub_to_scrape = 'IndiaSpeaks'\n",
        "data_to_fetch = ['title', 'selftext', 'url', 'author', 'created_utc']\n",
        "output_path = \"drive/MyDrive\"\n",
        "# end_epoch=int(dt.datetime(2020, 4, 1).timestamp())\n",
        "######################\n",
        "\n",
        "# Show all columns when displaying dataframe\n",
        "pd.set_option('display.max_rows', 500)\n",
        "pd.set_option('display.max_columns', 500)\n",
        "pd.set_option('display.width', 1000)\n",
        "\n",
        "# Scraping\n",
        "api = PushshiftAPI()\n",
        "scraped_data = list(api.search_submissions(\n",
        "                            subreddit=sub_to_scrape,\n",
        "                            filter=data_to_fetch,\n",
        "                            is_self=True))\n",
        "\n",
        "posts = {\n",
        "    \"title\": [],\n",
        "    \"author\": [],\n",
        "    \"selftext\": [],\n",
        "    \"url\": [],\n",
        "    \"date\": []\n",
        "}\n",
        "\n",
        "# Form dataframe\n",
        "for n,x in enumerate(scraped_data):\n",
        "  posts['title'].append(scraped_data[n].d_['title'])\n",
        "  posts['author'].append(scraped_data[n].d_['author'])\n",
        "  posts['selftext'].append(scraped_data[n][2])\n",
        "  posts['url'].append(scraped_data[n].d_['url'])\n",
        "  posts['date'].append(dt.datetime.fromtimestamp(scraped_data[n].d_['created_utc']).strftime('%d-%m-%Y'))\n",
        "\n",
        "df = pd.DataFrame(data=posts)\n",
        "df['textlength'] = df['selftext'].str.len()\n",
        "df['date'] = pd.to_datetime(df['date'])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PJloAreFG2lT"
      },
      "source": [
        "### Show Data\n",
        "\n",
        "Take a look at the collected data put in a dataframe"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "QMvUISMdFEHs",
        "outputId": "1f363b54-c36e-4cd6-94e8-7d559058df7b"
      },
      "source": [
        "print(df)\n",
        "df.info()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "                                                   title             author                                           selftext                                                url       date  textlength\n",
            "0      Anyone suffering from Crohn’s disease in this ...        Mutant12123  Would like to get in touch with Indian brother...  https://www.reddit.com/r/IndiaSpeaks/comments/... 2021-09-04          99\n",
            "1      Hi I am AnnyArun. An avid motorcycle tourer, a...           annyarun                                          [removed]  https://www.reddit.com/r/IndiaSpeaks/comments/... 2021-09-04           9\n",
            "2      Is an MBA from IGNOU worth it? What are pros /...         any-mystic  I am a software engineer with 5.5 years of exp...  https://www.reddit.com/r/IndiaSpeaks/comments/... 2021-09-04          54\n",
            "3               help me fund a mushroom farming business         maitakeblr                                          [removed]  https://www.reddit.com/r/IndiaSpeaks/comments/... 2021-09-04           9\n",
            "4                               *Belongs to SC community  West-Database1544  &amp;#x200B;\\n\\nhttps://preview.redd.it/hunq8u...  https://www.reddit.com/r/IndiaSpeaks/comments/... 2021-09-04         142\n",
            "...                                                  ...                ...                                                ...                                                ...        ...         ...\n",
            "28355             Automod: Comment removal control panel      Blackbird-007                                          [removed]  https://www.reddit.com/r/IndiaSpeaks/comments/... 2016-04-22           9\n",
            "28356                    Removed Posts Log [iteration 1]      Blackbird-007                                          [removed]  https://www.reddit.com/r/IndiaSpeaks/comments/... 2016-04-22           9\n",
            "28357                    Please read wiki before posting          [deleted]                                          [deleted]  https://www.reddit.com/r/IndiaSpeaks/comments/... 2016-03-15           9\n",
            "28358                                  Under development          [deleted]                                          [deleted]  https://www.reddit.com/r/IndiaSpeaks/comments/... 2016-03-15           9\n",
            "28359                             Sub under development.          [deleted]                                          [deleted]  https://www.reddit.com/r/IndiaSpeaks/comments/... 2016-03-13           9\n",
            "\n",
            "[28360 rows x 6 columns]\n",
            "<class 'pandas.core.frame.DataFrame'>\n",
            "RangeIndex: 28360 entries, 0 to 28359\n",
            "Data columns (total 6 columns):\n",
            " #   Column      Non-Null Count  Dtype         \n",
            "---  ------      --------------  -----         \n",
            " 0   title       28360 non-null  object        \n",
            " 1   author      28360 non-null  object        \n",
            " 2   selftext    28360 non-null  object        \n",
            " 3   url         28360 non-null  object        \n",
            " 4   date        28360 non-null  datetime64[ns]\n",
            " 5   textlength  28360 non-null  int64         \n",
            "dtypes: datetime64[ns](1), int64(1), object(4)\n",
            "memory usage: 1.3+ MB\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_hiHXIahHc_y"
      },
      "source": [
        "###Clean Data\n",
        "\n",
        "Clean data of any removed or deleted posts"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "jxFVR73mHktd",
        "outputId": "d543cfb8-d292-4890-834d-7ae43875b27a"
      },
      "source": [
        "import html\n",
        "import redditcleaner\n",
        "\n",
        "# Discard removed and deleted Posts from data\n",
        "df = df[~df['selftext'].isin(['[deleted]', '[removed]'])]\n",
        "df = df[~df['author'].isin(['[deleted]', '[removed]'])]\n",
        "df = df.drop_duplicates(subset='selftext')\n",
        "\n",
        "# Remove non-ASCII\n",
        "def is_ascii(body):\n",
        "  return all(ord(char) < 128 for char in body)\n",
        "df = df[~df['selftext'].map(is_ascii) != True]\n",
        "\n",
        "# Remove misencoded text\n",
        "for misenc_word in [f'a {x*\"?\"} a' for x in range(2, 10)]:\n",
        "  df = df[~df['selftext'].str.contains(misenc_word, regex=False)]\n",
        "\n",
        "# Clean HTML encoding\n",
        "df['title'] = df['title'].apply(lambda x: html.unescape(x))\n",
        "df['selftext'] = df['selftext'].apply(lambda x: html.unescape(x))\n",
        "\n",
        "# Clean encoding issues\n",
        "df['selftext'] = df['selftext'].apply(lambda x: x.encode('cp1252','replace').decode('cp1252'))\n",
        "\n",
        "# Remove markdown\n",
        "df['selftext'] = df['selftext'].map(redditcleaner.clean)\n",
        "\n",
        "# Reset Index\n",
        "df = df.reset_index(drop=True)\n",
        "\n",
        "# Save as CSV file for review\n",
        "if os.path.exists('data.csv'):\n",
        "  os.remove('data.csv')\n",
        "df.to_csv(r'data.csv', index=False, encoding=\"utf8\")\n",
        "\n",
        "# Display cleaned data\n",
        "print(df)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "                                                   title              author                                           selftext                                                url       date  textlength\n",
            "0      Anyone suffering from Crohn’s disease in this ...         Mutant12123  Would like to get in touch with Indian brother...  https://www.reddit.com/r/IndiaSpeaks/comments/... 2021-09-04          99\n",
            "1      Is an MBA from IGNOU worth it? What are pros /...          any-mystic  I am a software engineer with 5.5 years of exp...  https://www.reddit.com/r/IndiaSpeaks/comments/... 2021-09-04          54\n",
            "2                               *Belongs to SC community   West-Database1544  &x200B; https://preview.redd.it/hunq8upg06s61....  https://www.reddit.com/r/IndiaSpeaks/comments/... 2021-09-04         142\n",
            "3                                           India in UPA           insane-67  I was in a debate today wherein we were discus...  https://www.reddit.com/r/IndiaSpeaks/comments/... 2021-09-04        2920\n",
            "4      Uttarakhand CM takes huge decision on his birt...             drd_rdx  http://www.opindia.com/2021/04/uttarakhand-cm-...  https://www.reddit.com/r/IndiaSpeaks/comments/... 2021-09-04         115\n",
            "...                                                  ...                 ...                                                ...                                                ...        ...         ...\n",
            "15702                 Sub etiquette and other ramblings.  pm_me_ur_nightmare  So when we made that post on bakchodi we weren...  https://www.reddit.com/r/IndiaSpeaks/comments/... 2016-05-15        3709\n",
            "15703                Let's discuss political ideologies.      imadiscodancer  We always criticise and support someone's ideo...  https://www.reddit.com/r/IndiaSpeaks/comments/... 2016-05-13         479\n",
            "15704                            Modlogs are now public!       Blackbird-007  We initially wanted to implement /r/publicmodl...  https://www.reddit.com/r/IndiaSpeaks/comments/... 2016-11-05         605\n",
            "15705  Those of you who believe in ET, what is your r...       Blackbird-007  As some of you knowing me from /r/bakchodi mig...  https://www.reddit.com/r/IndiaSpeaks/comments/... 2016-10-05         232\n",
            "15706      Accepting ideas/feedbacks on our direction...       Blackbird-007  I and another person have been discussing abou...  https://www.reddit.com/r/IndiaSpeaks/comments/... 2016-04-23        1612\n",
            "\n",
            "[15707 rows x 6 columns]\n",
            "Empty DataFrame\n",
            "Columns: [title, author, selftext, url, date, textlength]\n",
            "Index: []\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "p7WqdunWM16g"
      },
      "source": [
        "### Visualize Data\n",
        "\n",
        "Check the distribution of post lengths"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-GWuqe4pM1Xo",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 295
        },
        "outputId": "626ba7fc-251c-461f-b4cb-8230e79eaa02"
      },
      "source": [
        "import matplotlib.pyplot as plt\n",
        "\n",
        "plt.hist(df['textlength'], bins=30, edgecolor = \"black\")\n",
        "plt.title(r'Post Length Distribution')\n",
        "plt.xlabel('Length of Post')\n",
        "plt.ylabel('Frequency')\n",
        "plt.axis([0, 15000, 0 , 20000])\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZEAAAEWCAYAAACnlKo3AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3debhWdbn/8fdHcB4ClAgBFQ31oEdRtkOpZZmKpKKdfoo/UzKTSu3kr0FRu5IGO9qgxalUKo5Q5jyRoYZmZecclI0i4kBsEHQjgoKKqano/ftjfR9YbPfwsHiGvd2f13U9117rXtP9rGfv597r+12DIgIzM7MiNqp3AmZm1nW5iJiZWWEuImZmVpiLiJmZFeYiYmZmhbmImJlZYS4iZjUgaSdJIalnBdd5sqQ/VnB9j0k6NA2Pl/TbCq77Akm/qtT6rPNwEbGakrRI0uuS/iFpmaSrJW21Aevr8MsubfMTRbdRxIZuM+2XNyW9kl5zJf2HpPeV5omIayLiiDLX9b2O5ouIPSLiz0Vzzm3vUEnNLdb9/Yj4/Iau2zofFxGrh2MiYitgX6AB+Gad8+msfhARWwN9gdOAA4H/lrRlJTdSyaMj635cRKxuImIJcCewJ4CkY1OTykuS/izpX0rzSjpP0pL0X/k8SYdJGgFcAJyYjmweWZ/tS9pI0jhJCyStkHSDpD5pWqn5aYykpyW9IOnC3LKbS5os6UVJT0g6t/Tft6TfADsAv095nZvb7Mmtra+D/fTPiJgJHAtsS1ZQkPRZSX9Lw5J0uaTlklZJelTSnpLGAicD56Zcfp/mX5T26RzgVUk9Wzl62kzS9WmfPyRp79z7D0kfzI1fLel7qcDdCWyftvcPSdu3PGLs4LNeJOnrkuZIejnlsFk5+8pqz0XE6kbSIGAk8LCkXYFrgXPI/vOeRvYlvImk3YCzgf3Sf+ZHAosi4i7g+8D1EbFVROzd6oba9mXgOOCjwPbAi8DPW8xzMLAbcBjwrdyX3UXATsDOwOHAZ0oLRMQpwNOkI66I+EEZ6+tQRLwCTAcOaWXyEcBHgF2B9wEnACsiYiJwDdlRzVYRcUxumZOATwK9ImJ1K+scBdwI9AF+B9wmaeMOcnwVOAp4Nm1vq4h4Nj9Pe591brYTgBHAYGAv4LPtbdfqx0XE6uE2SS8BfwP+QlYITgT+EBHTI+It4EfA5sCHgbeBTYGhkjaOiEURsaACeXwRuDAimiPiDWA88OkWzTvfjojXI+IR4BGgVKhOAL4fES9GRDMwocxttrW+cj1L9qXe0lvA1sDugCLiiYhY2sG6JkTEMxHxehvTZ0XETenzuAzYjKxJbUO191nnc3s2IlYCvweGVWC7VgUuIlYPx0VEr4jYMSLOTF9i2wOLSzNExDvAM8CAiGgi+691PLBc0nWStq9AHjsCt6YmlZeAJ8gKVr/cPM/lhl8DSicBbJ/yK8kPt6et9ZVrALCyZTAi/gT8jOxIarmkiZK26WBdHeW8Znr6PJrJ3veGavOzzs2zofvJasRFxDqLZ8m+1IGsjR8YBCwBiIjfRcTBaZ4ALk2zbshtqJ8BjkoFrfTaLPXVdGQpMDA3PqjF9IrfHjudxfYJ4P7WpkfEhIgYDgwla9b6Rge5dJTjmvckaSOy91tqmnoN2CI37wfWY73tftbWtbiIWGdxA/DJ1GG+MfA14A3gfyTtJunjkjYF/gm8DryTllsG7JS+5NqzsaTNcq+ewJXAxZJ2BJDUV9Ko9cj3fEm9JQ0g67PJW0bWX7LBJG0qaThwG1m/zX+1Ms9+kg5I++5Vsv2U30dFchku6VNpX51D9nnMSNNmA/9XUo90gsNHc8stA7ZV7nTkFtr8rAvkaHXmImKdQkTMI+uc/k/gBeAYso7pN8n6Qy5J8eeA9wPnp0VvTD9XSHqonU1MIys+pdd44KfAVOCPkl4h+4I8oMyUv0PWvPMUcA9wE9kXYcl/AN9MTWVfL3OdLZ2b8loBTAFmAR9OndctbQP8kqzILE7L/DBN+zVZf9JLkm5bj+3fTtZ/8SJwCvCp1IcB8BWyz+glsrO/1qw3Ip4k6zhfmLa5ThNYB5+1dTHyQ6nMNpykLwGjI+KjHc5s9h7iIxGzAiT1l3SQsmtNdiNrkrm13nmZ1VrVioikQZLuk/R4uqjoKyneR9J0SfPTz94pLkkTJDWli4z2za1rTJp/vqQxufjwdFFVU1pW1Xo/Zi1sAlwFvAL8iazp5xd1zcisDqrWnCWpP9A/Ih6StDVZe+5xZBcNrYyISySNA3pHxHmSRpJd/DWSrF36pxFxgLIriBvJbo8RaT3DI+JFSQ8C/w48QNbmPSEi7qzKGzIzs3ep2pFIRCyNiIfS8Ctk5+APILsKdnKabTJZYSHFp0RmBtArFaIjgekRsTIiXiS7YndEmrZNRMyIrBJOya3LzMxqoCY3XpO0E7AP2RFDv9yVtM+x9sKuAax78VNzirUXb24l3tr2xwJjAbbccsvhu+++e/E3Y2bWDc2aNeuFiOjbMl71IpIukLoZOCciVuW7LSIiJFX99LB0/6CJAA0NDdHY2FjtTZqZvadIWtxavKpnZ6ULiW4GromIW1J4WWqKKvWbLE/xJax71e/AFGsvPrCVuJmZ1Ug1z84S2UVOT0TEZblJU4HSGVZjyM5qKcVPTWdpHQi8nJq97gaOSFcG9ya7W+ndadoqSQembZ2aW5eZmdVANZuzDiK7yvVRSbNT7AKyK49vkHQ62ZW1J6Rp08jOzGoiuy/PaQARsVLSd4GZab7vpDt7ApwJXE12B9A708vMzGqk212x7j4RM7P1J2lWRDS0jPuKdTMzK8xFxMzMCnMRMTOzwlxEzMysMBcRMzMrzEXEzMwKcxExM7PCXETMzKwwFxEzMyvMRcTMzApzETEzs8JcRMzMrDAXETMzK8xFxMzMCnMRMTOzwlxEzMysMBcRMzMrzEXEzMwKq1oRkTRJ0nJJc3Ox6yXNTq9FpWevS9pJ0uu5aVfmlhku6VFJTZImSFKK95E0XdL89LN3td6LmZm1rppHIlcDI/KBiDgxIoZFxDDgZuCW3OQFpWkR8cVc/ArgDGBIepXWOQ64NyKGAPemcTMzq6GqFZGI+CuwsrVp6WjiBODa9tYhqT+wTUTMiIgApgDHpcmjgMlpeHIubmZmNVKvPpFDgGURMT8XGyzpYUl/kXRIig0AmnPzNKcYQL+IWJqGnwP6VTVjMzN7l5512u5JrHsUshTYISJWSBoO3CZpj3JXFhEhKdqaLmksMBZghx12KJiymZm1VPMjEUk9gU8B15diEfFGRKxIw7OABcCuwBJgYG7xgSkGsCw1d5WavZa3tc2ImBgRDRHR0Ldv30q+HTOzbq0ezVmfAJ6MiDXNVJL6SuqRhncm60BfmJqrVkk6MPWjnArcnhabCoxJw2NycTMzq5FqnuJ7LfC/wG6SmiWdniaN5t0d6h8B5qRTfm8CvhgRpU75M4FfAU1kRyh3pvglwOGS5pMVpkuq9V7MzKx1yk566j4aGhqisbGx3mmYmXUpkmZFREPLuK9YNzOzwlxEzMysMBcRMzMrzEXEzMwKcxExM7PCXETMzKwwFxEzMyvMRcTMzApzETEzs8JcRMzMrDAXETMzK8xFxMzMCnMRMTOzwlxEzMysMBcRMzMrzEXEzMwKcxExM7PCXETMzKwwFxEzMyusakVE0iRJyyXNzcXGS1oiaXZ6jcxNO19Sk6R5ko7MxUekWJOkcbn4YEkPpPj1kjap1nsxM7PWVfNI5GpgRCvxyyNiWHpNA5A0FBgN7JGW+YWkHpJ6AD8HjgKGAieleQEuTev6IPAicHoV34uZmbWiakUkIv4KrCxz9lHAdRHxRkQ8BTQB+6dXU0QsjIg3geuAUZIEfBy4KS0/GTiuom/AzMw6VI8+kbMlzUnNXb1TbADwTG6e5hRrK74t8FJErG4Rb5WksZIaJTU+//zzlXofZmbdXq2LyBXALsAwYCnw41psNCImRkRDRDT07du3Fps0M+sWetZyYxGxrDQs6ZfAHWl0CTAoN+vAFKON+Aqgl6Se6WgkP7+ZmdVITY9EJPXPjR4PlM7cmgqMlrSppMHAEOBBYCYwJJ2JtQlZ5/vUiAjgPuDTafkxwO21eA9mZrZW1Y5EJF0LHApsJ6kZuAg4VNIwIIBFwBcAIuIxSTcAjwOrgbMi4u20nrOBu4EewKSIeCxt4jzgOknfAx4Gfl2t92JmZq1T9k9999HQ0BCNjY31TsPMrEuRNCsiGlrGfcW6mZkV5iJiZmaFuYiYmVlhLiJmZlaYi4iZmRXmImJmZoW5iJiZWWEuImZmVpiLiJmZFeYiYmZmhbmImJlZYTW9FXxn8MicOWQPRqydDwwYxNLmp2u6TTOzWuh2RWT1W2+x43l3dDxjBS2+9Oiabs/MrFbcnGVmZoW5iJiZWWEuImZmVpiLiJmZFeYiYmZmhVWtiEiaJGm5pLm52A8lPSlpjqRbJfVK8Z0kvS5pdnpdmVtmuKRHJTVJmqB0fq6kPpKmS5qffvau1nsxM7PWVfNI5GpgRIvYdGDPiNgL+Dtwfm7agogYll5fzMWvAM4AhqRXaZ3jgHsjYghwbxo3M7MaqloRiYi/AitbxP4YEavT6AxgYHvrkNQf2CYiZkREAFOA49LkUcDkNDw5FzczsxqpZ5/I54A7c+ODJT0s6S+SDkmxAUBzbp7mFAPoFxFL0/BzQL+2NiRprKRGSY0Vyt3MzKjTFeuSLgRWA9ek0FJgh4hYIWk4cJukPcpdX0SEpGhn+kRgYtp2m/OZmdn6qXkRkfRZ4GjgsNRERUS8AbyRhmdJWgDsCixh3SavgSkGsExS/4hYmpq9ltfoLZiZWVLT5ixJI4BzgWMj4rVcvK+kHml4Z7IO9IWpuWqVpAPTWVmnArenxaYCY9LwmFzczMxqpKwiIulf13fFkq4F/hfYTVKzpNOBnwFbA9NbnMr7EWCOpNnATcAXI6LUKX8m8CugCVjA2n6US4DDJc0HPpHGzcyshpRalNqfSbof2JTstN1rIuLlKudVNZKiHnfxLWc/m5l1VpJmRURDy3hZRyIRcQhwMjAImCXpd5IOr3COZmbWxZTdJxIR84FvAucBHwUmpKvPP1Wt5MzMrHMrt09kL0mXA08AHweOiYh/ScOXVzE/MzPrxMo9xfc/yTq3L4iI10vBiHhW0jerkpmZmXV65RaRTwKvR8TbAJI2AjaLiNci4jdVy87MzDq1cvtE7gE2z41vkWJmZtaNlVtENouIf5RG0vAW1UnJzMy6inKLyKuS9i2NpPtbvd7O/GZm1g2U2ydyDnCjpGcBAR8ATqxaVmZm1iWUVUQiYqak3YHdUmheRLxVvbTMzKwrWJ+7+O4H7JSW2VcSETGlKlmZmVmXUFYRkfQbYBdgNvB2CpeeNGhmZt1UuUciDcDQ8F0Ezcwsp9yzs+aSdaabmZmtUe6RyHbA45IeJD2BECAijq1KVmZm1iWUW0TGVzMJMzPrmso9xfcvknYEhkTEPZK2AHpUNzUzM+vsyr0V/Blkj629KoUGALdVKykzM+sayu1YPws4CFgFax5Q9f6OFpI0SdJySXNzsT6Spkuan372TnFJmiCpSdKcFrdZGZPmny9pTC4+XNKjaZkJklTm+zEzswoot4i8ERFvlkYk9SS7TqQjVwMjWsTGAfdGxBDg3jQOcBQwJL3GAlekbfUBLgIOAPYHLioVnjTPGbnlWm7LzMyqqNwi8hdJFwCbp2er3wj8vqOFIuKvwMoW4VHA5DQ8GTguF58SmRlAL0n9gSOB6RGxMiJeBKYDI9K0bSJiRrp+ZUpuXWZmVgPlFpFxwPPAo8AXgGlkz1svol9ELE3DzwH90vAA4JncfM0p1l68uZX4u0gaK6lRUmPBnM3MrBXlnp31DvDL9KqYiAhJVb8KPiImAhMBarE9M7Puotx7Zz1FK30gEbFzgW0uk9Q/IpamJqnlKb4EGJSbb2CKLQEObRH/c4oPbGV+MzOrkXKbsxrI7uK7H3AIMAH4bcFtTgVKZ1iNAW7PxU9NZ2kdCLycmr3uBo6Q1Dt1qB8B3J2mrZJ0YDor69TcuszMrAbKbc5a0SL0E0mzgG+1t5yka8mOIraT1Ex2ltUlwA2STgcWAyek2acBI4Em4DXgtLTtlZK+C8xM830nIkqd9WeSnQG2OXBnepmZWY2U25y1b250I7Ijkw6XjYiT2ph0WCvzBtn1KK2tZxIwqZV4I7BnR3mYmVl1lHvvrB/nhlcDi1h7BGFmZt1Uuc1ZH6t2ImZm1vWU25z11famR8RllUnHzMy6kvV5suF+ZGdQARwDPAjMr0ZSZmbWNZRbRAYC+0bEKwCSxgN/iIjPVCsxMzPr/Mq9TqQf8GZu/E3W3q7EzMy6qXKPRKYAD0q6NY0fx9qbKJqZWTdV7tlZF0u6k+xqdYDTIuLh6qVlZmZdQbnNWQBbAKsi4qdAs6TBVcrJzMy6iHIfj3sRcB5wfgptTPF7Z5mZ2XtEuUcixwPHAq8CRMSzwNbVSsrMzLqGcovIm+neVgEgacvqpWRmZl1FuUXkBklXkT2y9gzgHir8gCozM+t6Ojw7Kz2r43pgd2AVsBvwrYiYXuXczMyskyvndu4haVpE/CvgwmFmZmuU25z1kKT9qpqJmZl1OeVesX4A8BlJi8jO0BLZQcpe1UrMzMw6v3aLiKQdIuJp4Mga5WNmZl1IR81ZtwFExGLgsohYnH8V2aCk3STNzr1WSTpH0nhJS3LxkbllzpfUJGmepCNz8REp1iRpXJF8zMysuI6as5Qb3rkSG4yIecAwAEk9gCXArcBpwOUR8aN1EpCGAqOBPYDtgXsk7Zom/xw4HGgGZkqaGhGPVyJPMzPrWEdFJNoYrpTDgAURsTg7k7hVo4DrIuIN4ClJTcD+aVpTRCwEkHRdmtdFxMysRjpqzto7NTe9AuyVhldJekXSqgpsfzRwbW78bElzJE2S1DvFBgDP5OZpTrG24u8iaaykRkmNFcjZzMySdotIRPSIiG0iYuuI6JmGS+PbbMiGJW1Cdj+uG1PoCmAXsqaupcCPN2T9eRExMSIaIqKhUus0M7PyT/GthqOAhyJiGUDpJ4CkXwJ3pNElwKDccgNTjHbiZmZWA+vzPJFKO4lcU5ak/rlpxwNz0/BUYLSkTdMzTIYADwIzgSGSBqejmtFpXjMzq5G6HImkuwAfDnwhF/6BpGFkHfiLStMi4jFJN5B1mK8GzoqIt9N6zgbuBnoAkyLisZq9CTMzq08RiYhXgW1bxE5pZ/6LgYtbiU8DplU8QTMzK0s9m7PMzKyLcxExM7PCXETMzKwwFxEzMyvMRcTMzApzETEzs8JcRMzMrDAXETMzK8xFxMzMCnMRMTOzwlxEzMysMBcRMzMrzEXEzMwKcxExM7PCXETMzKwwFxEzMyvMRcTMzApzETEzs8LqVkQkLZL0qKTZkhpTrI+k6ZLmp5+9U1ySJkhqkjRH0r659YxJ88+XNKZe78fMrDuq95HIxyJiWEQ0pPFxwL0RMQS4N40DHAUMSa+xwBWQFR3gIuAAYH/golLhMTOz6qt3EWlpFDA5DU8GjsvFp0RmBtBLUn/gSGB6RKyMiBeB6cCIWidtZtZd1bOIBPBHSbMkjU2xfhGxNA0/B/RLwwOAZ3LLNqdYW/F1SBorqbHUbGZmZpXRs47bPjgilkh6PzBd0pP5iRERkqISG4qIicBEgEqt08zM6ngkEhFL0s/lwK1kfRrLUjMV6efyNPsSYFBu8YEp1lbczMxqoC5FRNKWkrYuDQNHAHOBqUDpDKsxwO1peCpwajpL60Dg5dTsdTdwhKTeqUP9iBQzM7MaqFdzVj/gVkmlHH4XEXdJmgncIOl0YDFwQpp/GjASaAJeA04DiIiVkr4LzEzzfSciVtbubZiZdW91KSIRsRDYu5X4CuCwVuIBnNXGuiYBkyqdo5mZdayzneJrZmZdiIuImZkV5iJiZmaFuYiYmVlhLiJmZlaYi4iZmRXmImJmZoW5iJiZWWEuImZmVpiLiJmZFeYiYmZmhbmImJlZYS4iZmZWmIuImZkV5iJiZmaFuYiYmVlhLiJmZlaYi4iZmRVW8yIiaZCk+yQ9LukxSV9J8fGSlkianV4jc8ucL6lJ0jxJR+biI1KsSdK4Wr8XM7Purh7PWF8NfC0iHpK0NTBL0vQ07fKI+FF+ZklDgdHAHsD2wD2Sdk2Tfw4cDjQDMyVNjYjHa/IuzMys9kUkIpYCS9PwK5KeAAa0s8go4LqIeAN4SlITsH+a1hQRCwEkXZfmdRExM6uRuvaJSNoJ2Ad4IIXOljRH0iRJvVNsAPBMbrHmFGsrbmZmNVK3IiJpK+Bm4JyIWAVcAewCDCM7UvlxBbc1VlKjpMZKrdPMzOrTJ4KkjckKyDURcQtARCzLTf8lcEcaXQIMyi0+MMVoJ76OiJgITEzrjgq8BTMzoz5nZwn4NfBERFyWi/fPzXY8MDcNTwVGS9pU0mBgCPAgMBMYImmwpE3IOt+n1uI9mJlZph5HIgcBpwCPSpqdYhcAJ0kaBgSwCPgCQEQ8JukGsg7z1cBZEfE2gKSzgbuBHsCkiHislm/EzKy7q8fZWX8D1Mqkae0sczFwcSvxae0tZ2Zm1eUr1s3MrDAXETMzK8xFxMzMCnMRMTOzwlxEzMysMBcRMzMrrC5XrHc7PTYmu8aytj4wYBBLm5+u+XbNrPtwEamFt99ix/Pu6Hi+Clt86dE136aZdS9uzjIzs8JcRMzMrDAXETMzK8xFxMzMCnMRMTOzwlxEzMysMBcRMzMrzEXEzMwKcxExM7PCfMX6e5lvt2JmVeYi8l7m262YWZV1+eYsSSMkzZPUJGlcvfMxM+tOunQRkdQD+DlwFDAUOEnS0PpmZaVmtFq++g/cod7v2qxb6urNWfsDTRGxEEDSdcAo4PG6ZtXd1aEZbfGPjq9L/0+PTTbj7Tf/WdNtus/JOhNFRL1zKEzSp4EREfH5NH4KcEBEnN1ivrHA2DS6JzC3polumO2AF+qdxHpyztXX1fIF51wL1cx3x4jo2zLY1Y9EyhIRE4GJAJIaI6KhzimVravlC865FrpavuCca6Ee+XbpPhFgCTAoNz4wxczMrAa6ehGZCQyRNFjSJsBoYGqdczIz6za6dHNWRKyWdDZwN9ADmBQRj3Ww2MTqZ1ZRXS1fcM610NXyBedcCzXPt0t3rJuZWX119eYsMzOrIxcRMzMrrNsUkc50exRJgyTdJ+lxSY9J+kqK95E0XdL89LN3ikvShJT7HEn75tY1Js0/X9KYKufdQ9LDku5I44MlPZDyuj6d3ICkTdN4U5q+U24d56f4PElHVjnfXpJukvSkpCckfagL7OP/l34n5kq6VtJmnWk/S5okabmkublYxfappOGSHk3LTFAFriBtI+cfpt+LOZJuldQrN63VfdfWd0hbn0+lc85N+5qkkLRdGq/vfo6I9/yLrNN9AbAzsAnwCDC0jvn0B/ZNw1sDfye7bcsPgHEpPg64NA2PBO4EBBwIPJDifYCF6WfvNNy7inl/FfgdcEcavwEYnYavBL6Uhs8ErkzDo4Hr0/DQtO83BQanz6RHFfOdDHw+DW8C9OrM+xgYADwFbJ7bv5/tTPsZ+AiwLzA3F6vYPgUeTPMqLXtUlXI+AuiZhi/N5dzqvqOd75C2Pp9K55zig8hOJFoMbNcZ9nNV/ng72wv4EHB3bvx84Px655XL53bgcGAe0D/F+gPz0vBVwEm5+eel6ScBV+Xi68xX4RwHAvcCHwfuSL98L+T+ENfs4/RL/qE03DPNp5b7PT9fFfJ9H9kXslrEO/M+HgA8k/7oe6b9fGRn28/ATqz7hVyRfZqmPZmLrzNfJXNuMe144Jo03Oq+o43vkPb+DqqRM3ATsDewiLVFpK77ubs0Z5X+OEuaU6zuUhPEPsADQL+IWJomPQf0S8Nt5V/L9/UT4FzgnTS+LfBSRKxuZdtr8krTX07z1zLfwcDzwH8pa4L7laQt6cT7OCKWAD8CngaWku23WXTu/QyV26cD0nDLeLV9juy/cTrIrbV4e38HFSVpFLAkIh5pMamu+7m7FJFOSdJWwM3AORGxKj8tsn8ROsX515KOBpZHxKx657IeepI1B1wREfsAr5I1tazRmfYxQOpLGEVWALcHtgRG1DWp9dTZ9mlHJF0IrAauqXcu7ZG0BXAB8K1659JSdykine72KJI2Jisg10TELSm8TFL/NL0/sDzF28q/Vu/rIOBYSYuA68iatH4K9JJUumA1v+01eaXp7wNW1DBfyP67ao6IB9L4TWRFpbPuY4BPAE9FxPMR8RZwC9m+78z7GSq3T5ek4ZbxqpD0WeBo4ORU/Oggt9biK2j786mkXcj+uXgk/R0OBB6S9IECOVd2P1eqnbQzv8j+K12YPoRSp9gedcxHwBTgJy3iP2TdDsofpOFPsm7H2YMp3oes3b93ej0F9Kly7oeytmP9RtbtUDwzDZ/Fuh2+N6ThPVi303Ih1e1Yvx/YLQ2PT/u30+5j4ADgMWCLlMdk4MudbT/z7j6Riu1T3t3hO7JKOY8ge2RE3xbztbrvaOc7pK3Pp9I5t5i2iLV9InXdz1X54+2ML7IzGP5OdobFhXXO5WCyQ/45wOz0GknWvnovMB+4J/eBi+zhWwuAR4GG3Lo+BzSl12k1yP1Q1haRndMvY1P6Q9o0xTdL401p+s655S9M72MeFTjzpoNchwGNaT/flv6QOvU+Br4NPEn2uILfpC+zTrOfgWvJ+mveIjvaO72S+xRoSO99AfAzWpwYUcGcm8j6C0p/f1d2tO9o4zukrc+n0jm3mL6ItUWkrvvZtz0xM7PCukufiJmZVYGLiJmZFeYiYmZmhbmImJlZYS4iZmZWmIuIdXuS/lHl9Z+Trjje4O2lO/feI2m2pBNbTLta0lNp2kOSPrShuZp1xEXErPrOIbuAsBL2AYiIYRFxfSvTvxERw8gu+ruqwPormat1Ay4iZq2QtIukuyTNknS/pN1T/Or0/IX/kbRQ0qdTfCNJv0jPqJguaZqkT0v6d7L7YN0n6b7c+i+W9IikGZL6tbL9PpJuS8+HmCFpL0nvB4fcJ9QAAAJCSURBVH4L7JeONnZp5y38FfhgWtdXlT2fZK6kc1JsS0l/SDnMlXRiW7matcdFxKx1E4EvR8Rw4OvAL3LT+pPddeBo4JIU+xTZbSqGAqeQ3RKciJgAPAt8LCI+lubdEpgREXuTfdmf0cr2vw08HBF7kd14b0pELAc+D9yfjkQWtJP/McCjkoYDp5HdUuVA4AxJ+5Dd9uPZiNg7IvYE7mojV7N29ex4FrPuJd1d+cPAjbkHvm2am+W2iHgHeDx3FHEwcGOKP9fBf/Jvkj0rBLJbvR/eyjwHA/8GEBF/krStpG3KSP+Hkr5Jdhv804HDgFsj4tX03m4BDgHuAn4s6VKy29jcX8a6zd7FRcTs3TYie0bEsDamv5EbLvJY0bdi7f2G3qayf4ffiIibSiOSDmttpoj4e3qM6kjge5LujYjvVDAP6ybcnGXWQmTPdnlK0v+BNc+w3ruDxf4b+LfUN9KP7EaVJa+QPQZ5fdwPnJy2fyjwQrR45sx6rOc4SVukh3IdD9wvaXvgtYj4LdldeEvP5S6Sq3VjPhIxgy0k5Z/0dhnZF/gVqWloY7LnqLR8olzezWRNR4+T3R32IbInDULWv3KXpGfXo69hPDBJ0hzgNWBMmcutIyIeknQ12V1mAX4VEQ9LOpKs6esdsjvFfmkDcrVuzHfxNasQSVtFxD8kbUv2pX1QRDxX77zMqslHImaVc4ekXmQPLfquC4h1Bz4SMTOzwtyxbmZmhbmImJlZYS4iZmZWmIuImZkV5iJiZmaF/X9wDqjyo6pGQgAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "U7iOsz5gyu3X"
      },
      "source": [
        "### Save Data\n",
        "\n",
        "Save the dataset in a format that can be passed to the training script\n",
        "\n",
        "[Huggingface Docs](https://huggingface.co/docs/datasets/loading_datasets.html)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ezGBb1wIyuV7",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "12de30a5-045e-4ad6-daa3-a1cf0bc31a1e"
      },
      "source": [
        "# Shuffle\n",
        "shuffled = df.sample(n=len(df), random_state=25)\n",
        "print(shuffled)\n",
        "\n",
        "# Divide the post data in a 9:10 training:eval ratio and write to files\n",
        "if os.path.exists(f\"{output_path}/aarav2-dev-training.txt\"):\n",
        "  os.remove(f\"{output_path}/aarav2-dev-training.txt\")\n",
        "if os.path.exists(f\"{output_path}/aarav2-dev-eval.txt\"):\n",
        "  os.remove(f\"{output_path}/aarav2-dev-eval.txt\")\n",
        "\n",
        "dataset = open(f\"{output_path}/aarav2-dev-training.txt\", 'w', encoding=\"utf8\")\n",
        "eval = open(f\"{output_path}/aarav2-dev-eval.txt\", 'w', encoding=\"utf8\")\n",
        "\n",
        "point = round((9*len(df))/10)\n",
        "\n",
        "for index, row in shuffled.iterrows():\n",
        "  if index < point:\n",
        "    dataset.write(row['selftext'] + '\\n')\n",
        "  else:\n",
        "    eval.write(row['selftext'] + '\\n')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "                                                   title              author                                           selftext                                                url       date  textlength\n",
            "12971  Know about right to property and 78th ammendme...  Sunset-in-the-east  All rights are not fundamental in nature like ...  https://www.reddit.com/r/IndiaSpeaks/comments/... 2018-07-14        1073\n",
            "7293              A thought provoking for the privileged           chirayu89  The Indian Express: Listen, Mister Muslim. You...  https://www.reddit.com/r/IndiaSpeaks/comments/... 2019-11-14         261\n",
            "1910             Do Indians join the army for the money?             AdiCrio  I recently got into an argument with an americ...  https://www.reddit.com/r/IndiaSpeaks/comments/... 2020-08-19         193\n",
            "2495                      Solution for COVID-19 Pandemic         Anantaniium  SUGGESTED SOLUTION : As we know that the world...  https://www.reddit.com/r/IndiaSpeaks/comments/... 2020-12-07        4217\n",
            "4107   Understanding Islam through the hadis: Ram Swarup             iisdumb  Thanks to the internet, this once banned book,...  https://www.reddit.com/r/IndiaSpeaks/comments/... 2020-04-27         237\n",
            "...                                                  ...                 ...                                                ...                                                ...        ...         ...\n",
            "15677  Hi How are you guys and gals, if any ;) This i...    SuperAnupamSinha  Edit: Thank you friends It was really good tal...  https://www.reddit.com/r/IndiaSpeaks/comments/... 2016-05-22         215\n",
            "10383    Extremism , Fundamentalism , Doctrine And Islam        Modern_Asura  Some stupid things I noted about Islam . First...  https://www.reddit.com/r/IndiaSpeaks/comments/... 2019-04-22        1527\n",
            "6618   Mods, can we keep the CAB-related sticky posts...               waeva  The current 2 stickies are just generic and no...  https://www.reddit.com/r/IndiaSpeaks/comments/... 2019-12-18         101\n",
            "8510   The holy festival of love between brother and ...      igiveshittoppl  I'm the most broken person in the world now. I...  https://www.reddit.com/r/IndiaSpeaks/comments/... 2019-08-18        1340\n",
            "13444  When anyone says 'minority', why people consid...             DJWaled  Muslim population in India is more than 10% to...  https://www.reddit.com/r/IndiaSpeaks/comments/... 2018-04-29        1024\n",
            "\n",
            "[15707 rows x 6 columns]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ocu1qOd4K-j6"
      },
      "source": [
        " ## 2. Train Model\n",
        " \n",
        " Connect to a runtime with GPU acceleration and run."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "OGrqFzbZNWwA",
        "outputId": "b0b9f662-181c-4ca2-c71b-e85ee2231861"
      },
      "source": [
        "# Start training. should take about 30:00\n",
        "!python run_clm.py --output_dir=drive/MyDrive/aarav2-dev --model_type=gpt2 --model_name_or_path=gpt2 --do_train --train_file=drive/MyDrive/aarav2-dev-training.txt --do_eval --validation_file=drive/MyDrive/aarav2-dev-eval.txt --per_device_train_batch_size=2 --per_device_eval_batch_size=2 --num_train_epochs=1"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "2021-04-10 07:07:07.252708: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0\n",
            "04/10/2021 07:07:09 - WARNING - __main__ -   Process rank: -1, device: cuda:0, n_gpu: 1distributed training: False, 16-bits training: False\n",
            "04/10/2021 07:07:09 - INFO - __main__ -   Training/evaluation parameters TrainingArguments(output_dir=drive/MyDrive/aarav3, overwrite_output_dir=False, do_train=True, do_eval=True, do_predict=False, evaluation_strategy=IntervalStrategy.NO, prediction_loss_only=False, per_device_train_batch_size=2, per_device_eval_batch_size=2, gradient_accumulation_steps=1, eval_accumulation_steps=None, learning_rate=5e-05, weight_decay=0.0, adam_beta1=0.9, adam_beta2=0.999, adam_epsilon=1e-08, max_grad_norm=1.0, num_train_epochs=1.0, max_steps=-1, lr_scheduler_type=SchedulerType.LINEAR, warmup_ratio=0.0, warmup_steps=0, logging_dir=runs/Apr10_07-07-09_a01d947bab4b, logging_strategy=IntervalStrategy.STEPS, logging_first_step=False, logging_steps=500, save_strategy=IntervalStrategy.STEPS, save_steps=500, save_total_limit=None, no_cuda=False, seed=42, fp16=False, fp16_opt_level=O1, fp16_backend=auto, fp16_full_eval=False, local_rank=-1, tpu_num_cores=None, tpu_metrics_debug=False, debug=False, dataloader_drop_last=False, eval_steps=500, dataloader_num_workers=0, past_index=-1, run_name=drive/MyDrive/aarav3, disable_tqdm=False, remove_unused_columns=True, label_names=None, load_best_model_at_end=False, metric_for_best_model=None, greater_is_better=None, ignore_data_skip=False, sharded_ddp=[], deepspeed=None, label_smoothing_factor=0.0, adafactor=False, group_by_length=False, length_column_name=length, report_to=['tensorboard'], ddp_find_unused_parameters=None, dataloader_pin_memory=True, skip_memory_metrics=False, _n_gpu=1, mp_parameters=)\n",
            "04/10/2021 07:07:10 - WARNING - datasets.builder -   Using custom data configuration default-3cbe7bed0bf2b14f\n",
            "Downloading and preparing dataset text/default (download: Unknown size, generated: Unknown size, post-processed: Unknown size, total: Unknown size) to /root/.cache/huggingface/datasets/text/default-3cbe7bed0bf2b14f/0.0.0/e16f44aa1b321ece1f87b07977cc5d70be93d69b20486d6dacd62e12cf25c9a5...\n",
            "Dataset text downloaded and prepared to /root/.cache/huggingface/datasets/text/default-3cbe7bed0bf2b14f/0.0.0/e16f44aa1b321ece1f87b07977cc5d70be93d69b20486d6dacd62e12cf25c9a5. Subsequent calls will reuse this data.\n",
            "[INFO|file_utils.py:1394] 2021-04-10 07:07:11,559 >> https://huggingface.co/gpt2/resolve/main/config.json not found in cache or force_download set to True, downloading to /root/.cache/huggingface/transformers/tmp0t1r1syf\n",
            "Downloading: 100% 665/665 [00:00<00:00, 402kB/s]\n",
            "[INFO|file_utils.py:1398] 2021-04-10 07:07:12,112 >> storing https://huggingface.co/gpt2/resolve/main/config.json in cache at /root/.cache/huggingface/transformers/fc674cd6907b4c9e933cb42d67662436b89fa9540a1f40d7c919d0109289ad01.7d2e0efa5ca20cef4fb199382111e9d3ad96fd77b849e1d4bed13a66e1336f51\n",
            "[INFO|file_utils.py:1401] 2021-04-10 07:07:12,112 >> creating metadata file for /root/.cache/huggingface/transformers/fc674cd6907b4c9e933cb42d67662436b89fa9540a1f40d7c919d0109289ad01.7d2e0efa5ca20cef4fb199382111e9d3ad96fd77b849e1d4bed13a66e1336f51\n",
            "[INFO|configuration_utils.py:490] 2021-04-10 07:07:12,113 >> loading configuration file https://huggingface.co/gpt2/resolve/main/config.json from cache at /root/.cache/huggingface/transformers/fc674cd6907b4c9e933cb42d67662436b89fa9540a1f40d7c919d0109289ad01.7d2e0efa5ca20cef4fb199382111e9d3ad96fd77b849e1d4bed13a66e1336f51\n",
            "[INFO|configuration_utils.py:526] 2021-04-10 07:07:12,114 >> Model config GPT2Config {\n",
            "  \"activation_function\": \"gelu_new\",\n",
            "  \"architectures\": [\n",
            "    \"GPT2LMHeadModel\"\n",
            "  ],\n",
            "  \"attn_pdrop\": 0.1,\n",
            "  \"bos_token_id\": 50256,\n",
            "  \"embd_pdrop\": 0.1,\n",
            "  \"eos_token_id\": 50256,\n",
            "  \"gradient_checkpointing\": false,\n",
            "  \"initializer_range\": 0.02,\n",
            "  \"layer_norm_epsilon\": 1e-05,\n",
            "  \"model_type\": \"gpt2\",\n",
            "  \"n_ctx\": 1024,\n",
            "  \"n_embd\": 768,\n",
            "  \"n_head\": 12,\n",
            "  \"n_inner\": null,\n",
            "  \"n_layer\": 12,\n",
            "  \"n_positions\": 1024,\n",
            "  \"resid_pdrop\": 0.1,\n",
            "  \"summary_activation\": null,\n",
            "  \"summary_first_dropout\": 0.1,\n",
            "  \"summary_proj_to_labels\": true,\n",
            "  \"summary_type\": \"cls_index\",\n",
            "  \"summary_use_proj\": true,\n",
            "  \"task_specific_params\": {\n",
            "    \"text-generation\": {\n",
            "      \"do_sample\": true,\n",
            "      \"max_length\": 50\n",
            "    }\n",
            "  },\n",
            "  \"transformers_version\": \"4.6.0.dev0\",\n",
            "  \"use_cache\": true,\n",
            "  \"vocab_size\": 50257\n",
            "}\n",
            "\n",
            "[INFO|configuration_utils.py:490] 2021-04-10 07:07:12,682 >> loading configuration file https://huggingface.co/gpt2/resolve/main/config.json from cache at /root/.cache/huggingface/transformers/fc674cd6907b4c9e933cb42d67662436b89fa9540a1f40d7c919d0109289ad01.7d2e0efa5ca20cef4fb199382111e9d3ad96fd77b849e1d4bed13a66e1336f51\n",
            "[INFO|configuration_utils.py:526] 2021-04-10 07:07:12,683 >> Model config GPT2Config {\n",
            "  \"activation_function\": \"gelu_new\",\n",
            "  \"architectures\": [\n",
            "    \"GPT2LMHeadModel\"\n",
            "  ],\n",
            "  \"attn_pdrop\": 0.1,\n",
            "  \"bos_token_id\": 50256,\n",
            "  \"embd_pdrop\": 0.1,\n",
            "  \"eos_token_id\": 50256,\n",
            "  \"gradient_checkpointing\": false,\n",
            "  \"initializer_range\": 0.02,\n",
            "  \"layer_norm_epsilon\": 1e-05,\n",
            "  \"model_type\": \"gpt2\",\n",
            "  \"n_ctx\": 1024,\n",
            "  \"n_embd\": 768,\n",
            "  \"n_head\": 12,\n",
            "  \"n_inner\": null,\n",
            "  \"n_layer\": 12,\n",
            "  \"n_positions\": 1024,\n",
            "  \"resid_pdrop\": 0.1,\n",
            "  \"summary_activation\": null,\n",
            "  \"summary_first_dropout\": 0.1,\n",
            "  \"summary_proj_to_labels\": true,\n",
            "  \"summary_type\": \"cls_index\",\n",
            "  \"summary_use_proj\": true,\n",
            "  \"task_specific_params\": {\n",
            "    \"text-generation\": {\n",
            "      \"do_sample\": true,\n",
            "      \"max_length\": 50\n",
            "    }\n",
            "  },\n",
            "  \"transformers_version\": \"4.6.0.dev0\",\n",
            "  \"use_cache\": true,\n",
            "  \"vocab_size\": 50257\n",
            "}\n",
            "\n",
            "[INFO|file_utils.py:1394] 2021-04-10 07:07:13,250 >> https://huggingface.co/gpt2/resolve/main/vocab.json not found in cache or force_download set to True, downloading to /root/.cache/huggingface/transformers/tmpb3sigah2\n",
            "Downloading: 100% 1.04M/1.04M [00:01<00:00, 956kB/s]\n",
            "[INFO|file_utils.py:1398] 2021-04-10 07:07:14,907 >> storing https://huggingface.co/gpt2/resolve/main/vocab.json in cache at /root/.cache/huggingface/transformers/684fe667923972fb57f6b4dcb61a3c92763ad89882f3da5da9866baf14f2d60f.c7ed1f96aac49e745788faa77ba0a26a392643a50bb388b9c04ff469e555241f\n",
            "[INFO|file_utils.py:1401] 2021-04-10 07:07:14,907 >> creating metadata file for /root/.cache/huggingface/transformers/684fe667923972fb57f6b4dcb61a3c92763ad89882f3da5da9866baf14f2d60f.c7ed1f96aac49e745788faa77ba0a26a392643a50bb388b9c04ff469e555241f\n",
            "[INFO|file_utils.py:1394] 2021-04-10 07:07:15,461 >> https://huggingface.co/gpt2/resolve/main/merges.txt not found in cache or force_download set to True, downloading to /root/.cache/huggingface/transformers/tmpt0x1as_m\n",
            "Downloading: 100% 456k/456k [00:00<00:00, 515kB/s]\n",
            "[INFO|file_utils.py:1398] 2021-04-10 07:07:16,901 >> storing https://huggingface.co/gpt2/resolve/main/merges.txt in cache at /root/.cache/huggingface/transformers/c0c761a63004025aeadd530c4c27b860ec4ecbe8a00531233de21d865a402598.5d12962c5ee615a4c803841266e9c3be9a691a924f72d395d3a6c6c81157788b\n",
            "[INFO|file_utils.py:1401] 2021-04-10 07:07:16,901 >> creating metadata file for /root/.cache/huggingface/transformers/c0c761a63004025aeadd530c4c27b860ec4ecbe8a00531233de21d865a402598.5d12962c5ee615a4c803841266e9c3be9a691a924f72d395d3a6c6c81157788b\n",
            "[INFO|file_utils.py:1394] 2021-04-10 07:07:17,743 >> https://huggingface.co/gpt2/resolve/main/tokenizer.json not found in cache or force_download set to True, downloading to /root/.cache/huggingface/transformers/tmplqq1udiw\n",
            "Downloading: 100% 1.36M/1.36M [00:01<00:00, 1.24MB/s]\n",
            "[INFO|file_utils.py:1398] 2021-04-10 07:07:19,411 >> storing https://huggingface.co/gpt2/resolve/main/tokenizer.json in cache at /root/.cache/huggingface/transformers/16a2f78023c8dc511294f0c97b5e10fde3ef9889ad6d11ffaa2a00714e73926e.cf2d0ecb83b6df91b3dbb53f1d1e4c311578bfd3aa0e04934215a49bf9898df0\n",
            "[INFO|file_utils.py:1401] 2021-04-10 07:07:19,411 >> creating metadata file for /root/.cache/huggingface/transformers/16a2f78023c8dc511294f0c97b5e10fde3ef9889ad6d11ffaa2a00714e73926e.cf2d0ecb83b6df91b3dbb53f1d1e4c311578bfd3aa0e04934215a49bf9898df0\n",
            "[INFO|tokenization_utils_base.py:1713] 2021-04-10 07:07:21,062 >> loading file https://huggingface.co/gpt2/resolve/main/vocab.json from cache at /root/.cache/huggingface/transformers/684fe667923972fb57f6b4dcb61a3c92763ad89882f3da5da9866baf14f2d60f.c7ed1f96aac49e745788faa77ba0a26a392643a50bb388b9c04ff469e555241f\n",
            "[INFO|tokenization_utils_base.py:1713] 2021-04-10 07:07:21,063 >> loading file https://huggingface.co/gpt2/resolve/main/merges.txt from cache at /root/.cache/huggingface/transformers/c0c761a63004025aeadd530c4c27b860ec4ecbe8a00531233de21d865a402598.5d12962c5ee615a4c803841266e9c3be9a691a924f72d395d3a6c6c81157788b\n",
            "[INFO|tokenization_utils_base.py:1713] 2021-04-10 07:07:21,063 >> loading file https://huggingface.co/gpt2/resolve/main/tokenizer.json from cache at /root/.cache/huggingface/transformers/16a2f78023c8dc511294f0c97b5e10fde3ef9889ad6d11ffaa2a00714e73926e.cf2d0ecb83b6df91b3dbb53f1d1e4c311578bfd3aa0e04934215a49bf9898df0\n",
            "[INFO|tokenization_utils_base.py:1713] 2021-04-10 07:07:21,063 >> loading file https://huggingface.co/gpt2/resolve/main/added_tokens.json from cache at None\n",
            "[INFO|tokenization_utils_base.py:1713] 2021-04-10 07:07:21,063 >> loading file https://huggingface.co/gpt2/resolve/main/special_tokens_map.json from cache at None\n",
            "[INFO|tokenization_utils_base.py:1713] 2021-04-10 07:07:21,063 >> loading file https://huggingface.co/gpt2/resolve/main/tokenizer_config.json from cache at None\n",
            "[INFO|file_utils.py:1394] 2021-04-10 07:07:21,695 >> https://huggingface.co/gpt2/resolve/main/pytorch_model.bin not found in cache or force_download set to True, downloading to /root/.cache/huggingface/transformers/tmpydxnp4ie\n",
            "Downloading: 100% 548M/548M [00:13<00:00, 40.9MB/s]\n",
            "[INFO|file_utils.py:1398] 2021-04-10 07:07:35,132 >> storing https://huggingface.co/gpt2/resolve/main/pytorch_model.bin in cache at /root/.cache/huggingface/transformers/752929ace039baa8ef70fe21cdf9ab9445773d20e733cf693d667982e210837e.323c769945a351daa25546176f8208b3004b6f563438a7603e7932bae9025925\n",
            "[INFO|file_utils.py:1401] 2021-04-10 07:07:35,132 >> creating metadata file for /root/.cache/huggingface/transformers/752929ace039baa8ef70fe21cdf9ab9445773d20e733cf693d667982e210837e.323c769945a351daa25546176f8208b3004b6f563438a7603e7932bae9025925\n",
            "[INFO|modeling_utils.py:1069] 2021-04-10 07:07:35,132 >> loading weights file https://huggingface.co/gpt2/resolve/main/pytorch_model.bin from cache at /root/.cache/huggingface/transformers/752929ace039baa8ef70fe21cdf9ab9445773d20e733cf693d667982e210837e.323c769945a351daa25546176f8208b3004b6f563438a7603e7932bae9025925\n",
            "[INFO|modeling_utils.py:1198] 2021-04-10 07:07:40,862 >> All model checkpoint weights were used when initializing GPT2LMHeadModel.\n",
            "\n",
            "[INFO|modeling_utils.py:1207] 2021-04-10 07:07:40,862 >> All the weights of GPT2LMHeadModel were initialized from the model checkpoint at gpt2.\n",
            "If your task is similar to the task the model of the checkpoint was trained on, you can already use GPT2LMHeadModel for predictions without further training.\n",
            "  0% 0/15 [00:00<?, ?ba/s][WARNING|tokenization_utils_base.py:3143] 2021-04-10 07:07:41,285 >> Token indices sequence length is longer than the specified maximum sequence length for this model (1031 > 1024). Running this sequence through the model will result in indexing errors\n",
            "[WARNING|run_clm.py:329] 2021-04-10 07:07:41,286 >> ^^^^^^^^^^^^^^^^ Please ignore the warning above - this long input will be chunked into smaller bits before being passed to the model.\n",
            "100% 15/15 [00:05<00:00,  2.79ba/s]\n",
            "100% 2/2 [00:00<00:00,  4.67ba/s]\n",
            "100% 15/15 [00:15<00:00,  1.01s/ba]\n",
            "100% 2/2 [00:01<00:00,  1.95ba/s]\n",
            "[INFO|trainer.py:1011] 2021-04-10 07:08:12,153 >> ***** Running training *****\n",
            "[INFO|trainer.py:1012] 2021-04-10 07:08:12,153 >>   Num examples = 2224\n",
            "[INFO|trainer.py:1013] 2021-04-10 07:08:12,153 >>   Num Epochs = 1\n",
            "[INFO|trainer.py:1014] 2021-04-10 07:08:12,153 >>   Instantaneous batch size per device = 2\n",
            "[INFO|trainer.py:1015] 2021-04-10 07:08:12,153 >>   Total train batch size (w. parallel, distributed & accumulation) = 2\n",
            "[INFO|trainer.py:1016] 2021-04-10 07:08:12,153 >>   Gradient Accumulation steps = 1\n",
            "[INFO|trainer.py:1017] 2021-04-10 07:08:12,153 >>   Total optimization steps = 1112\n",
            "{'loss': 3.9298, 'learning_rate': 2.7517985611510793e-05, 'epoch': 0.45}\n",
            " 45% 500/1112 [12:12<14:57,  1.47s/it][INFO|trainer.py:1667] 2021-04-10 07:20:24,439 >> Saving model checkpoint to drive/MyDrive/aarav3/checkpoint-500\n",
            "[INFO|configuration_utils.py:329] 2021-04-10 07:20:24,447 >> Configuration saved in drive/MyDrive/aarav3/checkpoint-500/config.json\n",
            "[INFO|modeling_utils.py:848] 2021-04-10 07:20:27,027 >> Model weights saved in drive/MyDrive/aarav3/checkpoint-500/pytorch_model.bin\n",
            "[INFO|tokenization_utils_base.py:1907] 2021-04-10 07:20:27,035 >> tokenizer config file saved in drive/MyDrive/aarav3/checkpoint-500/tokenizer_config.json\n",
            "[INFO|tokenization_utils_base.py:1913] 2021-04-10 07:20:27,040 >> Special tokens file saved in drive/MyDrive/aarav3/checkpoint-500/special_tokens_map.json\n",
            "{'loss': 3.8637, 'learning_rate': 5.035971223021583e-06, 'epoch': 0.9}\n",
            " 90% 1000/1112 [24:31<02:43,  1.46s/it][INFO|trainer.py:1667] 2021-04-10 07:32:43,976 >> Saving model checkpoint to drive/MyDrive/aarav3/checkpoint-1000\n",
            "[INFO|configuration_utils.py:329] 2021-04-10 07:32:43,985 >> Configuration saved in drive/MyDrive/aarav3/checkpoint-1000/config.json\n",
            "[INFO|modeling_utils.py:848] 2021-04-10 07:32:46,587 >> Model weights saved in drive/MyDrive/aarav3/checkpoint-1000/pytorch_model.bin\n",
            "[INFO|tokenization_utils_base.py:1907] 2021-04-10 07:32:46,593 >> tokenizer config file saved in drive/MyDrive/aarav3/checkpoint-1000/tokenizer_config.json\n",
            "[INFO|tokenization_utils_base.py:1913] 2021-04-10 07:32:46,599 >> Special tokens file saved in drive/MyDrive/aarav3/checkpoint-1000/special_tokens_map.json\n",
            "100% 1112/1112 [27:25<00:00,  1.47s/it][INFO|trainer.py:1198] 2021-04-10 07:35:37,206 >> \n",
            "\n",
            "Training completed. Do not forget to share your model on huggingface.co/models =)\n",
            "\n",
            "\n",
            "{'train_runtime': 1645.0526, 'train_samples_per_second': 0.676, 'epoch': 1.0}\n",
            "100% 1112/1112 [27:25<00:00,  1.48s/it]\n",
            "[INFO|trainer.py:1667] 2021-04-10 07:35:37,987 >> Saving model checkpoint to drive/MyDrive/aarav3\n",
            "[INFO|configuration_utils.py:329] 2021-04-10 07:35:37,993 >> Configuration saved in drive/MyDrive/aarav3/config.json\n",
            "[INFO|modeling_utils.py:848] 2021-04-10 07:35:40,898 >> Model weights saved in drive/MyDrive/aarav3/pytorch_model.bin\n",
            "[INFO|tokenization_utils_base.py:1907] 2021-04-10 07:35:41,421 >> tokenizer config file saved in drive/MyDrive/aarav3/tokenizer_config.json\n",
            "[INFO|tokenization_utils_base.py:1913] 2021-04-10 07:35:41,431 >> Special tokens file saved in drive/MyDrive/aarav3/special_tokens_map.json\n",
            "[INFO|trainer_pt_utils.py:722] 2021-04-10 07:35:41,539 >> ***** train metrics *****\n",
            "[INFO|trainer_pt_utils.py:727] 2021-04-10 07:35:41,540 >>   epoch                      =        1.0\n",
            "[INFO|trainer_pt_utils.py:727] 2021-04-10 07:35:41,540 >>   init_mem_cpu_alloc_delta   =      873MB\n",
            "[INFO|trainer_pt_utils.py:727] 2021-04-10 07:35:41,540 >>   init_mem_cpu_peaked_delta  =        0MB\n",
            "[INFO|trainer_pt_utils.py:727] 2021-04-10 07:35:41,540 >>   init_mem_gpu_alloc_delta   =      487MB\n",
            "[INFO|trainer_pt_utils.py:727] 2021-04-10 07:35:41,540 >>   init_mem_gpu_peaked_delta  =        0MB\n",
            "[INFO|trainer_pt_utils.py:727] 2021-04-10 07:35:41,540 >>   train_mem_cpu_alloc_delta  =     -142MB\n",
            "[INFO|trainer_pt_utils.py:727] 2021-04-10 07:35:41,540 >>   train_mem_cpu_peaked_delta =      183MB\n",
            "[INFO|trainer_pt_utils.py:727] 2021-04-10 07:35:41,540 >>   train_mem_gpu_alloc_delta  =     1431MB\n",
            "[INFO|trainer_pt_utils.py:727] 2021-04-10 07:35:41,540 >>   train_mem_gpu_peaked_delta =     7403MB\n",
            "[INFO|trainer_pt_utils.py:727] 2021-04-10 07:35:41,540 >>   train_runtime              = 0:27:25.05\n",
            "[INFO|trainer_pt_utils.py:727] 2021-04-10 07:35:41,540 >>   train_samples              =       2224\n",
            "[INFO|trainer_pt_utils.py:727] 2021-04-10 07:35:41,541 >>   train_samples_per_second   =      0.676\n",
            "04/10/2021 07:35:41 - INFO - __main__ -   *** Evaluate ***\n",
            "[INFO|trainer.py:1894] 2021-04-10 07:35:41,667 >> ***** Running Evaluation *****\n",
            "[INFO|trainer.py:1895] 2021-04-10 07:35:41,667 >>   Num examples = 169\n",
            "[INFO|trainer.py:1896] 2021-04-10 07:35:41,667 >>   Batch size = 2\n",
            "100% 85/85 [00:39<00:00,  2.15it/s]\n",
            "[INFO|trainer_pt_utils.py:722] 2021-04-10 07:36:21,456 >> ***** eval metrics *****\n",
            "[INFO|trainer_pt_utils.py:727] 2021-04-10 07:36:21,456 >>   epoch                     =        1.0\n",
            "[INFO|trainer_pt_utils.py:727] 2021-04-10 07:36:21,456 >>   eval_loss                 =     3.8355\n",
            "[INFO|trainer_pt_utils.py:727] 2021-04-10 07:36:21,456 >>   eval_mem_cpu_alloc_delta  =        2MB\n",
            "[INFO|trainer_pt_utils.py:727] 2021-04-10 07:36:21,457 >>   eval_mem_cpu_peaked_delta =        0MB\n",
            "[INFO|trainer_pt_utils.py:727] 2021-04-10 07:36:21,457 >>   eval_mem_gpu_alloc_delta  =        0MB\n",
            "[INFO|trainer_pt_utils.py:727] 2021-04-10 07:36:21,457 >>   eval_mem_gpu_peaked_delta =     1399MB\n",
            "[INFO|trainer_pt_utils.py:727] 2021-04-10 07:36:21,457 >>   eval_runtime              = 0:00:39.57\n",
            "[INFO|trainer_pt_utils.py:727] 2021-04-10 07:36:21,457 >>   eval_samples              =        169\n",
            "[INFO|trainer_pt_utils.py:727] 2021-04-10 07:36:21,457 >>   eval_samples_per_second   =      4.271\n",
            "[INFO|trainer_pt_utils.py:727] 2021-04-10 07:36:21,457 >>   perplexity                =    46.3161\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lrgEM1clwpk2"
      },
      "source": [
        "## 3. Generate Text\n",
        "\n",
        "Fetch our newly trained model and generate text with it.\n",
        "\n",
        "[Huggingface Tutorial](https://huggingface.co/blog/how-to-generate)\n",
        "\n",
        "[Colab Reference](https://colab.research.google.com/gist/GeorgeDittmar/5c57a35332b2b5818e51618af7953351/lm-huggingface-finetune-gpt-2.ipynb)\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zwvDVoMqOP9v",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "fad97cb6-ad93-4c23-d785-96b1884a7a4f"
      },
      "source": [
        "from transformers import TFGPT2LMHeadModel\n",
        "from transformers import GPT2Tokenizer\n",
        "\n",
        "# Import the trained model\n",
        "model = TFGPT2LMHeadModel.from_pretrained(\"drive/MyDrive/aarav2-dev\", from_pt=True)\n",
        "tokenizer = GPT2Tokenizer.from_pretrained(\"gpt2\")\n",
        "\n",
        "# Generate new sequences from the trained model\n",
        "generated_text_samples = model.generate(\n",
        "    None, \n",
        "    max_length=300,  \n",
        "    num_return_sequences=30,\n",
        "    repetition_penalty=1.0,\n",
        "    top_p=0.9,\n",
        "    temperature=1.0,\n",
        "    do_sample=True,\n",
        "    top_k=50,\n",
        ")\n",
        "\n",
        "# Decode and display model output\n",
        "for i, beam in enumerate(generated_text_samples):\n",
        "  print(\"{}: {}\".format(i,tokenizer.decode(beam, skip_special_tokens=True)))\n",
        "  print()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Some weights of the PyTorch model were not used when initializing the TF 2.0 model TFGPT2LMHeadModel: ['transformer.h.4.attn.masked_bias', 'transformer.h.3.attn.masked_bias', 'transformer.h.8.attn.masked_bias', 'transformer.h.9.attn.masked_bias', 'transformer.h.1.attn.masked_bias', 'transformer.h.5.attn.masked_bias', 'transformer.h.0.attn.masked_bias', 'transformer.h.2.attn.masked_bias', 'transformer.h.7.attn.masked_bias', 'transformer.h.10.attn.masked_bias', 'lm_head.weight', 'transformer.h.6.attn.masked_bias', 'transformer.h.11.attn.masked_bias']\n",
            "- This IS expected if you are initializing TFGPT2LMHeadModel from a PyTorch model trained on another task or with another architecture (e.g. initializing a TFBertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing TFGPT2LMHeadModel from a PyTorch model that you expect to be exactly identical (e.g. initializing a TFBertForSequenceClassification model from a BertForSequenceClassification model).\n",
            "All the weights of TFGPT2LMHeadModel were initialized from the PyTorch model.\n",
            "If your task is similar to the task the model of the checkpoint was trained on, you can already use TFGPT2LMHeadModel for predictions without further training.\n",
            "Setting `pad_token_id` to 50256 (first `eos_token_id`) to generate sequence\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "0: This episode is sponsored by The Booking Podcast - The Booking Podcast is sponsored by Amazon.com and iTunes and Stitcher. You can buy it here. https://thebookingpodcast.com/ podcast/ booking-podcast/ It is sponsored by Amazon Gift Cards. If you buy any gift card, you can send money to Amazon via the card reader. The book is sponsored by Amazon. The book can be downloaded from Amazon and you can get it here: https://bookingpodcast.com/amazon/ book-book/ How does Booking podcasts differ from other podcast? &x200B; Please note: If you are new to podcasting or reading about it, please read the below links. &x200B;  &x200B;                                                                                                                                               \n",
            "\n",
            "1: \"We can all appreciate that we are better than this. We are not the only ones who suffer from this.\" - Dr. Tarek Mehbooba, an Indian doctor who was kidnapped from his office on 10 August, 2015.   \"So I am a huge fan of Rajesh's stories. But I feel that there is a clear pattern and logic behind these stories, and I feel that some things I read today are just not right for a British reader. A story about a doctor who escaped from his hospital in Rajasthan in 2015 to join a group called the Indian Defence Force, and was later killed. Is that okay? Is that okay to you?\"   \"He is such a genius that, I don't want him to leave our country. But if we go back to him, we will know what his intentions are. I want him to die, but I cannot let him die. I want the British to respect him and to take his life. It's like a game. I cannot accept it. We are all so different. And it will make all of us look alike. And I want the British to look at the atrocities in Rajasthan and look at the way we have been treated and the ways our people have been treated in the last 30 years.\"         \"There is no such thing as democracy, and if we try to, we will be in grave danger.\" - Sir Peter Cushing,\n",
            "\n",
            "2: , and a lot more. https://twitter.com/i/web/status/1082989148859058800?s=22A-12A2F0B20B9630I'm reading a lot of different books on Hinduism. I have a small library to store them. And it is full of books about how to worship Hindu gods, and Hindu traditions, and Hindu religion. So here's one of my favourite books on Hinduism. This is a book about the way to be a great Hindu. There are literally two or three versions of the same story. One one is about the Dharma, and one is about the Dharma and all the traditions, and Dharma and all that. You can read the book from the first place or from the second. The third book is a bit much more detailed and is about the Dharma, and how to be a great Hindu. It has a lot of Hindu customs, and a lot of Hindu religion. My point is, this book is very specific to a certain religious group and they feel that they should do something differently from the others. This is a bit like saying that we need to have different Hindu rites than our neighbors. And that it must be done. I'm thinking about a lot of religions in India, and I think they are well rounded and have a lot of rituals, and traditions. This is something I haven't read much. I'm looking for more information about Hinduism from the different\n",
            "\n",
            "3: Satellite photos showing the two spacecraft, which will be used to carry nuclear weapons into space. (AP) A satellite photos showing the two spacecraft, which will be used to carry nuclear weapons into space. (AP)\n",
            "Hello all! Thank you for your patience in helping us make a little bit of something useful out of this, but it is all thanks to your participation. &x200B; I'm trying to create a wiki, which I will follow along with this thread, but can't find the right location to start. Please tell me where. I want the Wiki to be in English, but not German. Or I just don't have the space. I just want to be a little more professional. &x200B; Thanks!   - Hello everyone, This is the first article in our series on India. Here are a couple of pointers that I'm taking from the previous posts:  -  -  -             &x200B;  The most important points that I want to highlight is the following.  -  -    -                                  &x200B;  If you think India has an \"A\" rating system, I want to stress that this isn't to say that India hasn't been successful. \n",
            "\n",
            "4: \"What does the future hold for us?\" he asked. She nodded. \"They're going to get more money, they'll have more houses, etc... But what about us? They're going to be able to buy anything else they want, but I don't think any company has the money to pay the rent.\" She gave a bitter laugh. \"The only one that has the money is my sister's dad and my mum.\" He said this with a deep breath. \"I don't know why they're so stupid but I don't think it would make any difference. I think they're going to be the worst company in the world. I'll have to think hard about whether to take a job with another company, buy a house with a lot of money, or if I should give up my job and go home to my friends.\" I looked at her gravelly. \"I know I can't do that but maybe I'll take a job with another company that offers free, cheap accommodation for people of low income. Is this possible? If not, are they going to accept me?\" \"We'll work with you guys. We have a very friendly and polite environment. Just don't take a risk. You'll have to take responsibility for yourself. Our country, our society is very good if someone wants to do that. And I want you to do the same. There's nothing you can do to change the outcome. We're just not gonna give up on our\n",
            "\n",
            "5: There is a strong correlation between income and education in the UK and it is very important to understand the relationship. According to the US Census Bureau, the average American family is earning $1,100. This is the average salary at a given point in the US. It is clear from the table that the average American family has less than one-third of the resources in the US compared to the US average of 10-15%. What are the implications of this? The richest 1% of Americans do not even have to give up on life after college. If they don't have a job then they can have one and enjoy the company while the poor have nowhere else to go. But it is not a simple matter and it makes them more miserable at the end of their college life. There is no point in having a job after you leave school. The only option left to be had is to go on a job. The bottom 80% are already well and rich. If they don't have the income, they either move out of town or get an apartment. It seems that the rich go to work. We can have a better life without the struggle of the poor. There is a huge gap between rich and poor and one can still have a better life, but the bottom 80% will always have a job. Why is this? Because the middle class is getting poorer and the middle class are getting better. I would like to discuss this in more detail and give you some perspective on the\n",
            "\n",
            "6: Hola! I know there are many theories, but I am asking here that it be said: If you want to be part of the game, do you have any other options? Also, it is a lot easier to make a new map in an instant, than it is to make a new player. That is the goal, and so much for this thread. But the main goal is to understand the game, not just how to create it.So, i am at the same time playing a game called \"Wormhole\" (in english, as it is sometimes called) and i am having a very bad night. Can anyone give any advice or advice regarding the issue? The one thing that i can recommend is that please use proper english. Also i am not sure if this is really a good time to give advice on this topic. I know a few people have taken some steps towards this, but i am not sure if it will work for everyone. The first thing i want to ask is your thoughts on it. Thank you so much. -SashaM - Thank you for taking the time to read the entire post.Hello! As of now, the thread is open to all users, please note that some of you who have been posting as part of the thread will have to join the sub for a while. There are some users who may want to join but don't want to join. Those who don't have a valid reason are welcome but they can still\n",
            "\n",
            "7: . The other man's name was a pseudonym and he was the guy who called his girlfriend to check the security deposit for him and then called her again. His girlfriend kept calling her. She started threatening him. Then he told her to leave the house as soon as he arrived home. He asked her again. And the guy was still threatening her. He tried to beat her but she kept getting angry and threatening him. The guy got angry again and again when he came back to the house. He came back to the house with a knife to her throat and she fell down unconscious. She was bleeding profusely. The guy asked her to come in the house and the girl said no and told him to go inside the house and then go outside. He told her to bring her in, she came in the house and called me and called me. The guy called my number too. They went inside the house, told me to bring her in. The woman and my brother came in after a few minutes and he came inside the house. He had her on his stomach and she was bleeding profusely. The guy told me to bring her in but she refused to comply. He tried to beat her but she kept getting angry and threatening him. The guy got angry again and again. She tried to fight but the guy got angry again and again when he came back and said it was his fault. He said it was because his girlfriend was an Indian woman, he had been raped and\n",
            "\n",
            "8: This is why the BJP should do something about it. It is about the future and not politics. This is India's future and the future will never be decided by its leaders and what they are doing. https://youtu.be/zLH8m0Vg3wQ https://en.wikipedia.org/wiki/Punjab_government_and_politics-2017: Punjab has a long history of communalism and is often seen as an integral part of the Punjab's identity. The country became a center of Hindu nationalism, though at the time it was not entirely free of anti-Gandhi sentiment and hatred of Indians. However, Punjab has experienced a similar pattern of nationalism that has seen the formation of Sikhism, and the subsequent revival of the Sikhs. What is the significance of this? https://www.hindutva.gov.in/sites/default/files/publications/sikhs-in-sikh-worship-of-pradesh-in-sikh-history.pdfI hope this is helpful to you guys. It seems that there are many anti-secular, anti-Islam and anti-India websites popping up every now and then that are attacking Islam, the Quran and the Sunnah, and are not only attacking Islam but attacking Christianity, Judaism and Christianity. We also have anti-Pakistan, Anti-Indology, anti-Indo-Pakistani and anti-Bollywood.\n",
            "\n",
            "9: You should have a backup account, but this isn't the case. Your phone may be disconnected, but you can recover it by going to Settings > General > Backup Now. If that's your problem, you can go to Settings > General > Backup > Backup Now > and search for the backup password for you. Please use the \"Unavailable\" link and follow the directions. You can also use the \"Unavailable\" link to restore your phone. The \"Unavailable\" link will allow you to search for a backup account without having to search again in the \"Unavailable\" tab, but after that you'll have a lot of trouble finding a backup account or recovery log. If you're looking for a backup account, you can go to Settings > General > Backup > Backup Now > and search for the backup password for your phone, and if it's still unavailable, you can go to Settings > General > Backup Now > and search for the backup password for you. The \"Unavailable\" link will allow you to search for a backup account without having to search again in the \"Unavailable\" tab, but after that you'll have a lot of trouble finding a backup account or recovery log. I don't know how to download the app. How do I use it? Will it help my friend with recovery? Also, will it hurt me if he doesn't find my phone?I am just trying to see if you guys will take any interest in the project. I've\n",
            "\n",
            "10: You know that we have all been waiting for this to happen to happen. But even today, we still have the fear that there might be some other country that is not in the same place that we are. India has been experiencing an alarming rise in illegal immigration and some of the people are fleeing India because of persecution and are afraid for their safety. I guess I'm just as surprised that our own PM is still talking like this, and it looks like they are not ready to move on the country's borders, and even after the recent ban on Muslim refugees and all of the rest of them are coming back and are living in the country illegally, that's all we've got. But India will get used to seeing a lot of this, and it seems like the fear over the coming years will get more pronounced. If the numbers don't change, the country may be looking at a major shift that it is not quite ready for. I am not sure what the government's plans are right now, but for now, there seems to be a lot of apprehension that maybe the country is in a bit of a crisis, and we might be heading for a situation where they can move on without us doing anything. So don't hold your breath, India is just getting started again.I am getting depressed lately. I am really tired. The time is definitely coming. I have a strong feeling that I have to do something. Is this not a good time to be depressed? The only time\n",
            "\n",
            "11: Rakshasa Samudra was a young girl who travelled to a local temple in Hyderabad to visit a monk who was visiting his friend. It was a dream and she was the first to join the temple. Her visit to the monk was arranged on condition that the monk gave her a promise that if she does not marry him within 2 years, he would give her a temple as well. Her marriage was annulled on condition that she get married within 15 years. In this situation she did not take any action that she would have taken if she had been married for over 15 years. But the monk had asked her to make a vow and to follow that practice and for this she did. When she was 17, she was accepted and went to his house to marry him. There, she told him that her marriage should not be annulled. She also mentioned that in order to not marry someone who is in love with her, she would have to marry the man who was in love with her. The monk also said that he could have given her the temple and even offered her the promise that if she did not marry him within 3 years, she would get a temple and would have to follow the practice of her father's temple. All in all, she stayed in a very beautiful hotel with her mother and her sister and did not lose any weight. At this time, Ram Mandir and his team had taken the oath to perform his duties. As if to show the\n",
            "\n",
            "12: 2 days ago. So, this is the first post. Today i wanted to talk about two things: 1. How a woman can gain social capital. 2. What role the economy plays in this ecosystem. https://t.co/eJ0QHXv2VUi 2nd, How can a society get stronger? Is it because of education? I will go into more detail about this later.https://twitter.com/R.K. Tharoor/status/111858259865473964? It's like there are people who think that Indian men and women are a threat to society. They are not. India's women and minorities are not threatening. They're just getting raped by the rich, being raped by their husbands and their wives. And it doesn't matter if it's Hindu or Muslim women. They're all being murdered for their own selfish needs, but in the absence of any threat to their lives, no one is even trying to fight them. But the sad thing is that while Indian men and women are in the world hating each other, women and minorities, not Indians are either or both. &x200B; So, I wanted to know what is the point of being in India as a man and as a man. Any other options that you could offer women and minorities?I have seen that every year in this country it is people who are the victims of extreme injustice in their lives that people are afraid\n",
            "\n",
            "13: Pakistan:   In 2009, it was the first country to have been awarded a Nobel Prize in Economics. And the Nobel Prizes were awarded in 2009 and 2010. But by late 2014, the Nobel Prizes had come to be known as the \"Noble\" Prize, which is awarded by a nation to a Nobel laureate to commemorate an individual's achievement.   India's Nobel Prize-Winning President Ashoka Chavan.   The award will be announced tomorrow and will be announced once the Supreme Court has been confirmed.   On February 3, 2015, Chavan will be giving a special address at Harvard University in Cambridge, Massachusetts.   The Nobel Prize will be given to a student in the U.S. during his two-day trip to India.   A statement by the Nobel Prizes' Executive Director Karan Johar on Monday said, \"It is the duty of our Nobel Laureates to bring justice to the poor victims of climate change and the world's most serious challenges and make the world a more prosperous place. To do that, they have given an official Nobel Prize speech to a group of Nobel Laureates, whose distinguished contributions and their long lasting commitment to the cause will be a significant achievement for the Nobel Prize.    \"It is a tremendous honour to award the Nobel Prize to one of our greatest and most celebrated figures, the country's most influential Nobel Laureates.\"   It was widely reported that the Nobel\n",
            "\n",
            "14: Maharashtra was supposed to have been the first state to adopt a \"no-harbour\" rule, but this is actually quite the opposite. Maharashtra has a much higher population of women than the national average of over 100 million. In order to maintain its fertility rate, the city has to also take in a lot of refugees, which can only increase fertility rates. If that were to happen, the Maharashtra government would be expected to impose a mandatory-burden on women. In the state of Maharashtra, if a Hindu woman is pregnant, they can expect to be forced to provide birth control pills to her. But women are allowed to refuse it. This is something which is not done by any government whatsoever, and the fact that women are getting this permission by the state government is not being acknowledged as fact. There's even a rape bill in the state of Maharashtra. The government is even asking for \"regulations to make sterilisation a mandatory requirement\" that it can't be done. I really hope Maharashtra does not get the \"wrong\" policy. There are other areas in the country where this isn't happening, but if you look at the fact that Maharashtra is so far from adopting this policy, there is absolutely no way the state could have done a better job of protecting women. There is no reason for the state government to allow these women to have the surgery to avoid the risk of their being raped by a religious person, for example. Any such laws or bans, which are\n",
            "\n",
            "15: .  -      \\-     &x200B;  \\-       &x200B;  \\-         &x200B;  \\-         &x200B;                &x200B;                  &x200B;                          &x200B;                                                                                                                                                                  \n",
            "\n",
            "16: : This is a list of the topics that were discussed at our discussion. &x200B; We will continue our discussion of these topics soon.                                                                                                                                                                                                                                                                             \n",
            "\n",
            "17: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -\n",
            "\n",
            "18: \"He says this is not the first time he's threatened to kill the innocent, he says this is a serious act of genocide,\" he said. \"There is no doubt that this is wrong and shameful. But I think that we should do everything we can to put an end to it.\"Earlier this year the Islamic State of Iraq and Syria (ISIS) was caught on video saying \"we'll be avenging the sins of the people of Syria\". This was a common refrain in Western media. But this time around the video was not shared widely. There are two factors which help explain this. 1. The first factor is that the terrorists are from the same network. The second factor is that ISIS has been known to carry out terror attacks in India on a daily basis for centuries. 2. The terrorists who made this video have been associated with the same network. I think there is no doubt that this is a serious act of genocide. I don't want to get into this in an abstract. The terrorists are being treated like terrorists in India. The problem is, it is not going to get any easier for the terrorists to use these attacks on the enemy and spread their message to the people of India. They are being treated like martyrs in India. It is happening in all the cities and towns of India. It is happening in every state in India. In many cities. In many parts of India. In all Indian villages. This is happening all over India. I think that\n",
            "\n",
            "19: We all know that we were asked to have some type of discussion at least once a day. But then again it wasn't exactly such a discussion. The only time when you can agree with someone is when they're saying something. It's a great way for everyone to ask other friends and acquaintances about your past or your life. So, this is something I decided to share with you guys. In my first week at work, I'd noticed some strange things in my boss' life. Some things that seemed to be strange to me. In the beginning, he wasn't interested in talking to me or even trying to understand me. When we talked to him (he was usually so very quiet), he seemed to have no idea what I was doing. And then came the strange thing that I couldn't understand or even really understand what was going on in my life. So, he decided to tell me something that I should understand. And he did it to me. But, in spite of that, I found that his lack of respect for me seemed to be a sign of deep psychological trauma. So, I started thinking about it in a deep, dark, dark way and what I should do about it. After discussing this, I eventually realized that he had no idea of how to talk to me or at the very least, not as I thought about it. He seemed to be doing the same thing over and over again. It's a shame that he didn't tell me what to\n",
            "\n",
            "20: A Pakistani woman named Nala Khan and her mother in a Mumbai mosque were arrested late on Saturday night. An FIR has been filed against them for inciting violence in a mosque that they were visiting on Sunday morning. It is the first time that such incidents have been reported in Mumbai, according to the Express Tribune, which reported the arrest early on Saturday morning. They were protesting against the construction of a new mosque in the city. The case is being prosecuted by police as a hate crime. It has also been accused of inciting violence against Hindus. The police had earlier arrested a man on suspicion of being a terrorist and was searching for the alleged culprits, according to a source close to the police. The source has said that the alleged culprits were not suspected of being related to the police, but had just left the mosque after a protest. While many Muslims in India consider it a religious test, the fact that the suspect has been released after being arrested shows that even this man is not suspected of being a terrorist, and they are also saying that he was arrested on Saturday night. It is a move to highlight the plight of those affected by the communal tensions. Police sources said that the police initially raided the mosque, but they did not take the suspect, nor did they take any action. The source said that the suspects were later released in custody, but that the accused, named as Nawazuddin Khan, had remained in custody till the end of Saturday night. Nawazuddin Khan was also\n",
            "\n",
            "21: A lot of people seem to think that it's a waste of time for a politician to show a certain face and do something without actually performing a proper role. And then there is the case of the 'bollywood superstar' and other celebrities who have a genuine, heartfelt and heartfelt, genuine and heartfelt and heartfelt desire for your image to stand out and shine. So why do people keep saying \"it's a waste of time for a politician to show a certain face and do something without actually performing a proper role\"? I understand that there are some people who are quite happy with who they are, but there are many who are not, and the people who still want to have your image as a symbol of their values are not worth the time or energy to spend. People will say \"it's a waste of time to show a certain face and perform a proper role\", and people will be happy to know they have a genuine and sincere desire to make you feel special, but it's not worth the time or energy to spend on a political campaign. But why should we let someone in whose entire life is devoted to promoting what we want and to do well? Should it not be a waste of time to entertain and entertain a candidate who is not even famous and who is not doing a good job of educating the public about his own side of things? Or is it a waste of time to entertain and entertain a celebrity who is being praised, applauded, admired and praised because he or she is a\n",
            "\n",
            "22: I understand that some people are very upset. I understand how there is a lot of tension and that we shouldn't have any more clashes or even protests when it comes to government activities. But to take it to mean there are no good things happening because people can get pissed off and then suddenly they're doing things like protesting against government actions, banning books, boycotting shops and other businesses and also making sure that their business activities are being held on the side of the government. And there have been a lot of incidents that happened here in India over the past few years (I remember the whole fiasco) and yet people still don't seem to understand. I know I was talking to a lot of people, but I don't understand. So I'm going to try and clarify this as best I can. I want to stress that I am not saying it is bad, and there are good things happening, but I am just saying that many, many people still need to understand this. Also, I want to make it clear that I am not suggesting any political parties be given an exemption for any such event or movement. But this will be more of an introspective exploration of the situation and what we can do to prevent this. There are many issues we need to get worked out and dealt with and there are plenty of things you can do to try and prevent such events. So if you have been participating in some kind of social experiment in which you create a social network, for example, social\n",
            "\n",
            "23: Haryana's ruling Congress party said Sunday that it has no intention of taking up a new term if BJP gains enough seats in the state's two assembly elections. The party has already been criticised by the Congress leadership over its plan to raise its own party candidate in the state. Congress had earlier claimed that it was trying to create a 'new generation' after the party was founded. The party, however, dismissed the claim as baseless and said that its members are being held responsible for taking up a new term, despite the fact that there are around 40 MLAs now, the party is already in power. Opposition parties, however, have claimed that the BJP would be forced to change its plan for a new term and that a new term would have to be created for any party that would try to form a majority government. Congress is still pursuing a fresh term as Congress is still leading a wave of protests in the state. A Congress spokesperson said the party will wait for the results of both the two elections before taking up a new term, and added that no new term would be needed. AAP leader M Jayalalithaa had said earlier that he and his party had been in talks with opposition parties and that there was no intention to change the plan. \"This plan is not a compromise, it is a serious plan and we are very close to finding out where we stand on this. We are not looking for a compromise. This is only one of several steps. The other is to\n",
            "\n",
            "24: \"It seems like you have been seeing it over and over again and every time, it's shown on your screen.\" \"We have never been this bad before. Now it feels like you have become this bad again.\" \"We are getting so much criticism for having these kind of standards, and being this bad in every aspect of our lives. It really is frustrating to watch your pathetic little face turn into a reality show. We are trying to take the positive side of it.\" \"The show has no good content. We are so overconfident about our progress that we have decided to focus our efforts on this, to help our viewers in finding some genuine hope. In the end, this show is just a continuation of how our characters were treated and I believe we have been able to bring about a change in our culture. The show was originally created by the Chinese Academy of Sciences. It was developed in India by Professor Yuryan Kumar of the Department of Physics, Kavli University in Bangalore. The series has been shown to over 300,000 people worldwide. You can view the series on YouTube here. And if you'd like to discuss your experiences with the show please ask!  This is the first of many articles in this series on Chinese intellectual history. So, I'm not going to try to make any specific claims about them, I'm just sharing this post so that other people will know about them.Hello Everyone, A few days ago, we posted a series about\n",
            "\n",
            "25: The BJP in the past is the enemy of all India and hence they have not been the biggest party in the country. The BJP has been in power for about 1.8 crore years and its a long time coming but it has finally broken free of the old coalition government. For the first time in history, a BJP government, backed by the Supreme court, can pass laws that give a party more power to run its economy without hindering other parts of the country. Now, after a decade of BJP rule, it has no chance to break free from government. As the BJP has a lot of experience and political experience, the government has to have more of a good track record to be able to tackle such a challenge. The government must have a good track record, its a real issue and we are doing it now. How does the government deal with the problem? The first step is for the government to get involved with the opposition parties. The opposition parties are a big part of the party because they want control of the entire state and they are opposed to any change in direction the government would want to take in a long time. As the party was a party of Hindutva and also a party of anti-nationalism, this would give them more power and power. The other step was for the opposition parties to get involved with the political opposition. This is the second step, the first step to break free of the old coalition government. To do this, the BJP must have a\n",
            "\n",
            "26: I'm an actor, I'm also from Chennai. Recently I decided to create a Twitter account, so I've decided to make an account where anyone can post anything I want on twitter. The rules are simple: 1. Tweet any relevant political views or political statements with the hashtag. For example, Modi vs Modi, Hindutva vs Shiv Sena, anti Hindu rightists vs anti Leftists. The only requirement is that you tweet anything that you believe in or are critical of. 2. Not post anything that has been reported in India online. In other words, you'll be banned if you tweet something that is critical of the BJP government, the Congress party or any other party. If you are just asking people to tweet, it will be a great platform to discuss issues and to create opinions that are just as pertinent to your own personal lives and that you can share easily and without incident. 3. Don't post anything that is not in the trending category. Any other opinions or opinions that are not relevant to your own personal life will be removed. 4. Don't post any opinion that has been reported in India online. In fact, if you tweet or share it online, you can face some serious punishment. The only way you can avoid this is by reporting it to the media and then posting it again. 5. Comment on any opinions or criticism that you think the BJP government will benefit from. 6. Don't post anything that is not in the trending category. For\n",
            "\n",
            "27: Ugandan and Bangladeshi Muslims are in a much worse plight. They are facing discrimination and have a choice - either fight for their land or leave the country. The Indian side of the equation - Pakistan - is one of the few countries with the largest Muslim population and the only country in Africa (even though Muslims are not as strong as the Indians). The Bangladeshi side of the equation - Bangladesh - is one of the only countries with high rates of violent crime. Their country is far from being safe for any reason, yet many Hindus in India continue to remain in constant fear of losing their home. They feel threatened and unsafe in their home. The reason? The government, in the face of such an overwhelming demand of Muslims from Bangladesh, has refused to acknowledge or recognize these Muslims - even though Muslims are not the sole beneficiaries of the Indian and Bangladeshi policies. They also see their home as their ultimate fate and will be sacrificed to ensure it. As a result, the Muslim population in India and the rest of Africa have experienced a crisis with rising rates of violence. Bangladeshis are facing a crisis. They are facing discrimination and fear. Their country is far from being safe for any reason, yet many Hindus in India continue to remain in constant fear of losing their home. They feel threatened and unsafe in their home. This is the problem Muslims face in the world - the Hindu majority are the largest population of any non-Hindus, in the world. They are\n",
            "\n",
            "28: 2.2% of the population lives in a rural area, with the majority of them working for themselves. Even though rural areas of India may have higher levels of poverty than urban areas, it is very difficult to find jobs in those areas. This is because rural areas lack an ecosystem to support the poor, and the economy does not have an effective supply chain to keep the poor happy. People who live in rural areas are very susceptible to diseases, such as malaria, which can kill off their immunity. So, farmers need more resources to provide people with healthcare and food. Rural areas are the only places with a great network of roads, railway lines and hospitals that is open to the poor, and that is exactly where most of the population is today. We need to understand how the system works, and what is being done to make that happen. And we will find ways to help rural communities become more aware of the effects of climate change. Rural populations need more money and resources, like water, electricity and food. The only solution, and one that the government can agree on is to invest in infrastructure to cope with climate change. In terms of the infrastructure, some countries are also very good, while some are not. India is the world's largest producer of hydrocarbons. For example, India produces the most hydrocarbons of any industrialized country (according to WHO, China produces only 2%), but the nation's infrastructure is so poor that even the country's biggest industrial power plants like\n",
            "\n",
            "29: .jpg?crop=200,000,000&auto=webp&s=e0fe7a9ec0d98a8ab5a8c8b0d8a2fce5a9.jpg?width=1000&format=pjpg&auto=webp&s=cce0c0d98d98a8ab5a8b0d8a2fce5a9.jpg&s=a87f6cdf4f6ea8fe8fe8fe8fe8fe8fe8fe8fe8fe8fe8fe8fe8fe8fe8fe8fe8fe8fe8fe8fe8fe8fe8fe8fe8fe8fe8fe8fe8fe8fe8fe8fe8fe8fe8fe8fe8fe8fe8fe8fe8fe8fe8fe8fe8fe8fe8fe8fe8fe8fe8fe8fe8fe8fe8fe8fe8fe8fe8fe8fe8fe8fe8fe8fe8fe8fe8fe8fe8fe8fe8fe8fe8fe8fe8fe8fe8fe8fe8fe8fe8fe8fe8fe8fe8fe8fe8fe8fe8fe8fe8fe8fe8fe8fe8fe8fe\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    }
  ]
}