ETL Pipeline Preparation.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# ETL Pipeline Preparation\n",
        "Follow the instructions below to help you create your ETL pipeline.\n",
        "### 1. Import libraries and load datasets.\n",
        "- Import Python libraries\n",
        "- Load `messages.csv` into a dataframe and inspect the first few lines.\n",
        "- Load `categories.csv` into a dataframe and inspect the first few lines."
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "# import libraries\n",
        "import pandas as pd\n",
        "import numpy as np\n",
        "from sqlalchemy import create_engine"
      ],
      "outputs": [],
      "execution_count": 54,
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "# load messages dataset\n",
        "messages = pd.read_csv('data/disaster_messages.csv')\n",
        "messages.head()"
      ],
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 55,
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>id</th>\n",
              "      <th>message</th>\n",
              "      <th>original</th>\n",
              "      <th>genre</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>2</td>\n",
              "      <td>Weather update - a cold front from Cuba that c...</td>\n",
              "      <td>Un front froid se retrouve sur Cuba ce matin. ...</td>\n",
              "      <td>direct</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>7</td>\n",
              "      <td>Is the Hurricane over or is it not over</td>\n",
              "      <td>Cyclone nan fini osinon li pa fini</td>\n",
              "      <td>direct</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>8</td>\n",
              "      <td>Looking for someone but no name</td>\n",
              "      <td>Patnm, di Maryani relem pou li banm nouvel li ...</td>\n",
              "      <td>direct</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>9</td>\n",
              "      <td>UN reports Leogane 80-90 destroyed. Only Hospi...</td>\n",
              "      <td>UN reports Leogane 80-90 destroyed. Only Hospi...</td>\n",
              "      <td>direct</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>12</td>\n",
              "      <td>says: west side of Haiti, rest of the country ...</td>\n",
              "      <td>facade ouest d Haiti et le reste du pays aujou...</td>\n",
              "      <td>direct</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   id                                            message  \\\n",
              "0   2  Weather update - a cold front from Cuba that c...   \n",
              "1   7            Is the Hurricane over or is it not over   \n",
              "2   8                    Looking for someone but no name   \n",
              "3   9  UN reports Leogane 80-90 destroyed. Only Hospi...   \n",
              "4  12  says: west side of Haiti, rest of the country ...   \n",
              "\n",
              "                                            original   genre  \n",
              "0  Un front froid se retrouve sur Cuba ce matin. ...  direct  \n",
              "1                 Cyclone nan fini osinon li pa fini  direct  \n",
              "2  Patnm, di Maryani relem pou li banm nouvel li ...  direct  \n",
              "3  UN reports Leogane 80-90 destroyed. Only Hospi...  direct  \n",
              "4  facade ouest d Haiti et le reste du pays aujou...  direct  "
            ]
          },
          "metadata": {}
        }
      ],
      "execution_count": 55,
      "metadata": {
        "scrolled": true
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# load categories dataset\n",
        "categories = pd.read_csv('data/disaster_categories.csv')\n",
        "categories.head()"
      ],
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 56,
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>id</th>\n",
              "      <th>categories</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>2</td>\n",
              "      <td>related-1;request-0;offer-0;aid_related-0;medi...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>7</td>\n",
              "      <td>related-1;request-0;offer-0;aid_related-1;medi...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>8</td>\n",
              "      <td>related-1;request-0;offer-0;aid_related-0;medi...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>9</td>\n",
              "      <td>related-1;request-1;offer-0;aid_related-1;medi...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>12</td>\n",
              "      <td>related-1;request-0;offer-0;aid_related-0;medi...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   id                                         categories\n",
              "0   2  related-1;request-0;offer-0;aid_related-0;medi...\n",
              "1   7  related-1;request-0;offer-0;aid_related-1;medi...\n",
              "2   8  related-1;request-0;offer-0;aid_related-0;medi...\n",
              "3   9  related-1;request-1;offer-0;aid_related-1;medi...\n",
              "4  12  related-1;request-0;offer-0;aid_related-0;medi..."
            ]
          },
          "metadata": {}
        }
      ],
      "execution_count": 56,
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 2. Merge datasets.\n",
        "- Merge the messages and categories datasets using the common id\n",
        "- Assign this combined dataset to `df`, which will be cleaned in the following steps"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "# merge datasets\n",
        "df = messages.merge(categories, how='left', on=['id'])\n",
        "df.head()"
      ],
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 57,
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>id</th>\n",
              "      <th>message</th>\n",
              "      <th>original</th>\n",
              "      <th>genre</th>\n",
              "      <th>categories</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>2</td>\n",
              "      <td>Weather update - a cold front from Cuba that c...</td>\n",
              "      <td>Un front froid se retrouve sur Cuba ce matin. ...</td>\n",
              "      <td>direct</td>\n",
              "      <td>related-1;request-0;offer-0;aid_related-0;medi...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>7</td>\n",
              "      <td>Is the Hurricane over or is it not over</td>\n",
              "      <td>Cyclone nan fini osinon li pa fini</td>\n",
              "      <td>direct</td>\n",
              "      <td>related-1;request-0;offer-0;aid_related-1;medi...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>8</td>\n",
              "      <td>Looking for someone but no name</td>\n",
              "      <td>Patnm, di Maryani relem pou li banm nouvel li ...</td>\n",
              "      <td>direct</td>\n",
              "      <td>related-1;request-0;offer-0;aid_related-0;medi...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>9</td>\n",
              "      <td>UN reports Leogane 80-90 destroyed. Only Hospi...</td>\n",
              "      <td>UN reports Leogane 80-90 destroyed. Only Hospi...</td>\n",
              "      <td>direct</td>\n",
              "      <td>related-1;request-1;offer-0;aid_related-1;medi...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>12</td>\n",
              "      <td>says: west side of Haiti, rest of the country ...</td>\n",
              "      <td>facade ouest d Haiti et le reste du pays aujou...</td>\n",
              "      <td>direct</td>\n",
              "      <td>related-1;request-0;offer-0;aid_related-0;medi...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   id                                            message  \\\n",
              "0   2  Weather update - a cold front from Cuba that c...   \n",
              "1   7            Is the Hurricane over or is it not over   \n",
              "2   8                    Looking for someone but no name   \n",
              "3   9  UN reports Leogane 80-90 destroyed. Only Hospi...   \n",
              "4  12  says: west side of Haiti, rest of the country ...   \n",
              "\n",
              "                                            original   genre  \\\n",
              "0  Un front froid se retrouve sur Cuba ce matin. ...  direct   \n",
              "1                 Cyclone nan fini osinon li pa fini  direct   \n",
              "2  Patnm, di Maryani relem pou li banm nouvel li ...  direct   \n",
              "3  UN reports Leogane 80-90 destroyed. Only Hospi...  direct   \n",
              "4  facade ouest d Haiti et le reste du pays aujou...  direct   \n",
              "\n",
              "                                          categories  \n",
              "0  related-1;request-0;offer-0;aid_related-0;medi...  \n",
              "1  related-1;request-0;offer-0;aid_related-1;medi...  \n",
              "2  related-1;request-0;offer-0;aid_related-0;medi...  \n",
              "3  related-1;request-1;offer-0;aid_related-1;medi...  \n",
              "4  related-1;request-0;offer-0;aid_related-0;medi...  "
            ]
          },
          "metadata": {}
        }
      ],
      "execution_count": 57,
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 3. Split `categories` into separate category columns.\n",
        "- Split the values in the `categories` column on the `;` character so that each value becomes a separate column. You'll find [this method](https://pandas.pydata.org/pandas-docs/version/0.23/generated/pandas.Series.str.split.html) very helpful! Make sure to set `expand=True`.\n",
        "- Use the first row of categories dataframe to create column names for the categories data.\n",
        "- Rename columns of `categories` with new column names."
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "# create a dataframe of the 36 individual category columns\n",
        "categories = df.categories.str.split(';', expand=True)\n",
        "categories.head()"
      ],
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 58,
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>0</th>\n",
              "      <th>1</th>\n",
              "      <th>2</th>\n",
              "      <th>3</th>\n",
              "      <th>4</th>\n",
              "      <th>5</th>\n",
              "      <th>6</th>\n",
              "      <th>7</th>\n",
              "      <th>8</th>\n",
              "      <th>9</th>\n",
              "      <th>...</th>\n",
              "      <th>26</th>\n",
              "      <th>27</th>\n",
              "      <th>28</th>\n",
              "      <th>29</th>\n",
              "      <th>30</th>\n",
              "      <th>31</th>\n",
              "      <th>32</th>\n",
              "      <th>33</th>\n",
              "      <th>34</th>\n",
              "      <th>35</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>related-1</td>\n",
              "      <td>request-0</td>\n",
              "      <td>offer-0</td>\n",
              "      <td>aid_related-0</td>\n",
              "      <td>medical_help-0</td>\n",
              "      <td>medical_products-0</td>\n",
              "      <td>search_and_rescue-0</td>\n",
              "      <td>security-0</td>\n",
              "      <td>military-0</td>\n",
              "      <td>child_alone-0</td>\n",
              "      <td>...</td>\n",
              "      <td>aid_centers-0</td>\n",
              "      <td>other_infrastructure-0</td>\n",
              "      <td>weather_related-0</td>\n",
              "      <td>floods-0</td>\n",
              "      <td>storm-0</td>\n",
              "      <td>fire-0</td>\n",
              "      <td>earthquake-0</td>\n",
              "      <td>cold-0</td>\n",
              "      <td>other_weather-0</td>\n",
              "      <td>direct_report-0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>related-1</td>\n",
              "      <td>request-0</td>\n",
              "      <td>offer-0</td>\n",
              "      <td>aid_related-1</td>\n",
              "      <td>medical_help-0</td>\n",
              "      <td>medical_products-0</td>\n",
              "      <td>search_and_rescue-0</td>\n",
              "      <td>security-0</td>\n",
              "      <td>military-0</td>\n",
              "      <td>child_alone-0</td>\n",
              "      <td>...</td>\n",
              "      <td>aid_centers-0</td>\n",
              "      <td>other_infrastructure-0</td>\n",
              "      <td>weather_related-1</td>\n",
              "      <td>floods-0</td>\n",
              "      <td>storm-1</td>\n",
              "      <td>fire-0</td>\n",
              "      <td>earthquake-0</td>\n",
              "      <td>cold-0</td>\n",
              "      <td>other_weather-0</td>\n",
              "      <td>direct_report-0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>related-1</td>\n",
              "      <td>request-0</td>\n",
              "      <td>offer-0</td>\n",
              "      <td>aid_related-0</td>\n",
              "      <td>medical_help-0</td>\n",
              "      <td>medical_products-0</td>\n",
              "      <td>search_and_rescue-0</td>\n",
              "      <td>security-0</td>\n",
              "      <td>military-0</td>\n",
              "      <td>child_alone-0</td>\n",
              "      <td>...</td>\n",
              "      <td>aid_centers-0</td>\n",
              "      <td>other_infrastructure-0</td>\n",
              "      <td>weather_related-0</td>\n",
              "      <td>floods-0</td>\n",
              "      <td>storm-0</td>\n",
              "      <td>fire-0</td>\n",
              "      <td>earthquake-0</td>\n",
              "      <td>cold-0</td>\n",
              "      <td>other_weather-0</td>\n",
              "      <td>direct_report-0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>related-1</td>\n",
              "      <td>request-1</td>\n",
              "      <td>offer-0</td>\n",
              "      <td>aid_related-1</td>\n",
              "      <td>medical_help-0</td>\n",
              "      <td>medical_products-1</td>\n",
              "      <td>search_and_rescue-0</td>\n",
              "      <td>security-0</td>\n",
              "      <td>military-0</td>\n",
              "      <td>child_alone-0</td>\n",
              "      <td>...</td>\n",
              "      <td>aid_centers-0</td>\n",
              "      <td>other_infrastructure-0</td>\n",
              "      <td>weather_related-0</td>\n",
              "      <td>floods-0</td>\n",
              "      <td>storm-0</td>\n",
              "      <td>fire-0</td>\n",
              "      <td>earthquake-0</td>\n",
              "      <td>cold-0</td>\n",
              "      <td>other_weather-0</td>\n",
              "      <td>direct_report-0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>related-1</td>\n",
              "      <td>request-0</td>\n",
              "      <td>offer-0</td>\n",
              "      <td>aid_related-0</td>\n",
              "      <td>medical_help-0</td>\n",
              "      <td>medical_products-0</td>\n",
              "      <td>search_and_rescue-0</td>\n",
              "      <td>security-0</td>\n",
              "      <td>military-0</td>\n",
              "      <td>child_alone-0</td>\n",
              "      <td>...</td>\n",
              "      <td>aid_centers-0</td>\n",
              "      <td>other_infrastructure-0</td>\n",
              "      <td>weather_related-0</td>\n",
              "      <td>floods-0</td>\n",
              "      <td>storm-0</td>\n",
              "      <td>fire-0</td>\n",
              "      <td>earthquake-0</td>\n",
              "      <td>cold-0</td>\n",
              "      <td>other_weather-0</td>\n",
              "      <td>direct_report-0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>5 rows × 36 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "          0          1        2              3               4   \\\n",
              "0  related-1  request-0  offer-0  aid_related-0  medical_help-0   \n",
              "1  related-1  request-0  offer-0  aid_related-1  medical_help-0   \n",
              "2  related-1  request-0  offer-0  aid_related-0  medical_help-0   \n",
              "3  related-1  request-1  offer-0  aid_related-1  medical_help-0   \n",
              "4  related-1  request-0  offer-0  aid_related-0  medical_help-0   \n",
              "\n",
              "                   5                    6           7           8   \\\n",
              "0  medical_products-0  search_and_rescue-0  security-0  military-0   \n",
              "1  medical_products-0  search_and_rescue-0  security-0  military-0   \n",
              "2  medical_products-0  search_and_rescue-0  security-0  military-0   \n",
              "3  medical_products-1  search_and_rescue-0  security-0  military-0   \n",
              "4  medical_products-0  search_and_rescue-0  security-0  military-0   \n",
              "\n",
              "              9   ...             26                      27  \\\n",
              "0  child_alone-0  ...  aid_centers-0  other_infrastructure-0   \n",
              "1  child_alone-0  ...  aid_centers-0  other_infrastructure-0   \n",
              "2  child_alone-0  ...  aid_centers-0  other_infrastructure-0   \n",
              "3  child_alone-0  ...  aid_centers-0  other_infrastructure-0   \n",
              "4  child_alone-0  ...  aid_centers-0  other_infrastructure-0   \n",
              "\n",
              "                  28        29       30      31            32      33  \\\n",
              "0  weather_related-0  floods-0  storm-0  fire-0  earthquake-0  cold-0   \n",
              "1  weather_related-1  floods-0  storm-1  fire-0  earthquake-0  cold-0   \n",
              "2  weather_related-0  floods-0  storm-0  fire-0  earthquake-0  cold-0   \n",
              "3  weather_related-0  floods-0  storm-0  fire-0  earthquake-0  cold-0   \n",
              "4  weather_related-0  floods-0  storm-0  fire-0  earthquake-0  cold-0   \n",
              "\n",
              "                34               35  \n",
              "0  other_weather-0  direct_report-0  \n",
              "1  other_weather-0  direct_report-0  \n",
              "2  other_weather-0  direct_report-0  \n",
              "3  other_weather-0  direct_report-0  \n",
              "4  other_weather-0  direct_report-0  \n",
              "\n[5 rows x 36 columns]"
            ]
          },
          "metadata": {}
        }
      ],
      "execution_count": 58,
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "# select the first row of the categories dataframe\n",
        "row = categories.iloc[0]\n",
        "\n",
        "# use this row to extract a list of new column names for categories.\n",
        "# one way is to apply a lambda function that takes everything \n",
        "# up to the second to last character of each string with slicing\n",
        "category_colnames = list(row.transform(lambda x: x[:-2]))\n",
        "print(category_colnames)"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['related', 'request', 'offer', 'aid_related', 'medical_help', 'medical_products', 'search_and_rescue', 'security', 'military', 'child_alone', 'water', 'food', 'shelter', 'clothing', 'money', 'missing_people', 'refugees', 'death', 'other_aid', 'infrastructure_related', 'transport', 'buildings', 'electricity', 'tools', 'hospitals', 'shops', 'aid_centers', 'other_infrastructure', 'weather_related', 'floods', 'storm', 'fire', 'earthquake', 'cold', 'other_weather', 'direct_report']\n"
          ]
        }
      ],
      "execution_count": 59,
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "# rename the columns of `categories`\n",
        "categories.columns = category_colnames\n",
        "categories.head()"
      ],
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 60,
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>related</th>\n",
              "      <th>request</th>\n",
              "      <th>offer</th>\n",
              "      <th>aid_related</th>\n",
              "      <th>medical_help</th>\n",
              "      <th>medical_products</th>\n",
              "      <th>search_and_rescue</th>\n",
              "      <th>security</th>\n",
              "      <th>military</th>\n",
              "      <th>child_alone</th>\n",
              "      <th>...</th>\n",
              "      <th>aid_centers</th>\n",
              "      <th>other_infrastructure</th>\n",
              "      <th>weather_related</th>\n",
              "      <th>floods</th>\n",
              "      <th>storm</th>\n",
              "      <th>fire</th>\n",
              "      <th>earthquake</th>\n",
              "      <th>cold</th>\n",
              "      <th>other_weather</th>\n",
              "      <th>direct_report</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>related-1</td>\n",
              "      <td>request-0</td>\n",
              "      <td>offer-0</td>\n",
              "      <td>aid_related-0</td>\n",
              "      <td>medical_help-0</td>\n",
              "      <td>medical_products-0</td>\n",
              "      <td>search_and_rescue-0</td>\n",
              "      <td>security-0</td>\n",
              "      <td>military-0</td>\n",
              "      <td>child_alone-0</td>\n",
              "      <td>...</td>\n",
              "      <td>aid_centers-0</td>\n",
              "      <td>other_infrastructure-0</td>\n",
              "      <td>weather_related-0</td>\n",
              "      <td>floods-0</td>\n",
              "      <td>storm-0</td>\n",
              "      <td>fire-0</td>\n",
              "      <td>earthquake-0</td>\n",
              "      <td>cold-0</td>\n",
              "      <td>other_weather-0</td>\n",
              "      <td>direct_report-0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>related-1</td>\n",
              "      <td>request-0</td>\n",
              "      <td>offer-0</td>\n",
              "      <td>aid_related-1</td>\n",
              "      <td>medical_help-0</td>\n",
              "      <td>medical_products-0</td>\n",
              "      <td>search_and_rescue-0</td>\n",
              "      <td>security-0</td>\n",
              "      <td>military-0</td>\n",
              "      <td>child_alone-0</td>\n",
              "      <td>...</td>\n",
              "      <td>aid_centers-0</td>\n",
              "      <td>other_infrastructure-0</td>\n",
              "      <td>weather_related-1</td>\n",
              "      <td>floods-0</td>\n",
              "      <td>storm-1</td>\n",
              "      <td>fire-0</td>\n",
              "      <td>earthquake-0</td>\n",
              "      <td>cold-0</td>\n",
              "      <td>other_weather-0</td>\n",
              "      <td>direct_report-0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>related-1</td>\n",
              "      <td>request-0</td>\n",
              "      <td>offer-0</td>\n",
              "      <td>aid_related-0</td>\n",
              "      <td>medical_help-0</td>\n",
              "      <td>medical_products-0</td>\n",
              "      <td>search_and_rescue-0</td>\n",
              "      <td>security-0</td>\n",
              "      <td>military-0</td>\n",
              "      <td>child_alone-0</td>\n",
              "      <td>...</td>\n",
              "      <td>aid_centers-0</td>\n",
              "      <td>other_infrastructure-0</td>\n",
              "      <td>weather_related-0</td>\n",
              "      <td>floods-0</td>\n",
              "      <td>storm-0</td>\n",
              "      <td>fire-0</td>\n",
              "      <td>earthquake-0</td>\n",
              "      <td>cold-0</td>\n",
              "      <td>other_weather-0</td>\n",
              "      <td>direct_report-0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>related-1</td>\n",
              "      <td>request-1</td>\n",
              "      <td>offer-0</td>\n",
              "      <td>aid_related-1</td>\n",
              "      <td>medical_help-0</td>\n",
              "      <td>medical_products-1</td>\n",
              "      <td>search_and_rescue-0</td>\n",
              "      <td>security-0</td>\n",
              "      <td>military-0</td>\n",
              "      <td>child_alone-0</td>\n",
              "      <td>...</td>\n",
              "      <td>aid_centers-0</td>\n",
              "      <td>other_infrastructure-0</td>\n",
              "      <td>weather_related-0</td>\n",
              "      <td>floods-0</td>\n",
              "      <td>storm-0</td>\n",
              "      <td>fire-0</td>\n",
              "      <td>earthquake-0</td>\n",
              "      <td>cold-0</td>\n",
              "      <td>other_weather-0</td>\n",
              "      <td>direct_report-0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>related-1</td>\n",
              "      <td>request-0</td>\n",
              "      <td>offer-0</td>\n",
              "      <td>aid_related-0</td>\n",
              "      <td>medical_help-0</td>\n",
              "      <td>medical_products-0</td>\n",
              "      <td>search_and_rescue-0</td>\n",
              "      <td>security-0</td>\n",
              "      <td>military-0</td>\n",
              "      <td>child_alone-0</td>\n",
              "      <td>...</td>\n",
              "      <td>aid_centers-0</td>\n",
              "      <td>other_infrastructure-0</td>\n",
              "      <td>weather_related-0</td>\n",
              "      <td>floods-0</td>\n",
              "      <td>storm-0</td>\n",
              "      <td>fire-0</td>\n",
              "      <td>earthquake-0</td>\n",
              "      <td>cold-0</td>\n",
              "      <td>other_weather-0</td>\n",
              "      <td>direct_report-0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>5 rows × 36 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "     related    request    offer    aid_related    medical_help  \\\n",
              "0  related-1  request-0  offer-0  aid_related-0  medical_help-0   \n",
              "1  related-1  request-0  offer-0  aid_related-1  medical_help-0   \n",
              "2  related-1  request-0  offer-0  aid_related-0  medical_help-0   \n",
              "3  related-1  request-1  offer-0  aid_related-1  medical_help-0   \n",
              "4  related-1  request-0  offer-0  aid_related-0  medical_help-0   \n",
              "\n",
              "     medical_products    search_and_rescue    security    military  \\\n",
              "0  medical_products-0  search_and_rescue-0  security-0  military-0   \n",
              "1  medical_products-0  search_and_rescue-0  security-0  military-0   \n",
              "2  medical_products-0  search_and_rescue-0  security-0  military-0   \n",
              "3  medical_products-1  search_and_rescue-0  security-0  military-0   \n",
              "4  medical_products-0  search_and_rescue-0  security-0  military-0   \n",
              "\n",
              "     child_alone  ...    aid_centers    other_infrastructure  \\\n",
              "0  child_alone-0  ...  aid_centers-0  other_infrastructure-0   \n",
              "1  child_alone-0  ...  aid_centers-0  other_infrastructure-0   \n",
              "2  child_alone-0  ...  aid_centers-0  other_infrastructure-0   \n",
              "3  child_alone-0  ...  aid_centers-0  other_infrastructure-0   \n",
              "4  child_alone-0  ...  aid_centers-0  other_infrastructure-0   \n",
              "\n",
              "     weather_related    floods    storm    fire    earthquake    cold  \\\n",
              "0  weather_related-0  floods-0  storm-0  fire-0  earthquake-0  cold-0   \n",
              "1  weather_related-1  floods-0  storm-1  fire-0  earthquake-0  cold-0   \n",
              "2  weather_related-0  floods-0  storm-0  fire-0  earthquake-0  cold-0   \n",
              "3  weather_related-0  floods-0  storm-0  fire-0  earthquake-0  cold-0   \n",
              "4  weather_related-0  floods-0  storm-0  fire-0  earthquake-0  cold-0   \n",
              "\n",
              "     other_weather    direct_report  \n",
              "0  other_weather-0  direct_report-0  \n",
              "1  other_weather-0  direct_report-0  \n",
              "2  other_weather-0  direct_report-0  \n",
              "3  other_weather-0  direct_report-0  \n",
              "4  other_weather-0  direct_report-0  \n",
              "\n[5 rows x 36 columns]"
            ]
          },
          "metadata": {}
        }
      ],
      "execution_count": 60,
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 4. Convert category values to just numbers 0 or 1.\n",
        "- Iterate through the category columns in df to keep only the last character of each string (the 1 or 0). For example, `related-0` becomes `0`, `related-1` becomes `1`. Convert the string to a numeric value.\n",
        "- You can perform [normal string actions on Pandas Series](https://pandas.pydata.org/pandas-docs/stable/text.html#indexing-with-str), like indexing, by including `.str` after the Series. You may need to first convert the Series to be of type string, which you can do with `astype(str)`."
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "# converting category values to just numbers 0 or 1.\n",
        "\n",
        "for column in categories:\n",
        "    # set each value to be the last character of the string\n",
        "    categories[column] = categories[column].str[-1].astype(str)\n",
        "    \n",
        "    # convert column from string to numeric\n",
        "    categories[column] = categories[column].astype(int)\n",
        "categories.head()"
      ],
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 61,
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>related</th>\n",
              "      <th>request</th>\n",
              "      <th>offer</th>\n",
              "      <th>aid_related</th>\n",
              "      <th>medical_help</th>\n",
              "      <th>medical_products</th>\n",
              "      <th>search_and_rescue</th>\n",
              "      <th>security</th>\n",
              "      <th>military</th>\n",
              "      <th>child_alone</th>\n",
              "      <th>...</th>\n",
              "      <th>aid_centers</th>\n",
              "      <th>other_infrastructure</th>\n",
              "      <th>weather_related</th>\n",
              "      <th>floods</th>\n",
              "      <th>storm</th>\n",
              "      <th>fire</th>\n",
              "      <th>earthquake</th>\n",
              "      <th>cold</th>\n",
              "      <th>other_weather</th>\n",
              "      <th>direct_report</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>5 rows × 36 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "   related  request  offer  aid_related  medical_help  medical_products  \\\n",
              "0        1        0      0            0             0                 0   \n",
              "1        1        0      0            1             0                 0   \n",
              "2        1        0      0            0             0                 0   \n",
              "3        1        1      0            1             0                 1   \n",
              "4        1        0      0            0             0                 0   \n",
              "\n",
              "   search_and_rescue  security  military  child_alone  ...  aid_centers  \\\n",
              "0                  0         0         0            0  ...            0   \n",
              "1                  0         0         0            0  ...            0   \n",
              "2                  0         0         0            0  ...            0   \n",
              "3                  0         0         0            0  ...            0   \n",
              "4                  0         0         0            0  ...            0   \n",
              "\n",
              "   other_infrastructure  weather_related  floods  storm  fire  earthquake  \\\n",
              "0                     0                0       0      0     0           0   \n",
              "1                     0                1       0      1     0           0   \n",
              "2                     0                0       0      0     0           0   \n",
              "3                     0                0       0      0     0           0   \n",
              "4                     0                0       0      0     0           0   \n",
              "\n",
              "   cold  other_weather  direct_report  \n",
              "0     0              0              0  \n",
              "1     0              0              0  \n",
              "2     0              0              0  \n",
              "3     0              0              0  \n",
              "4     0              0              0  \n",
              "\n[5 rows x 36 columns]"
            ]
          },
          "metadata": {}
        }
      ],
      "execution_count": 61,
      "metadata": {
        "scrolled": true
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 5. Replace `categories` column in `df` with new category columns.\n",
        "- Drop the categories column from the df dataframe since it is no longer needed.\n",
        "- Concatenate df and categories data frames."
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "# drop the original categories column from `df`\n",
        "df = df.drop('categories', axis=1)\n",
        "df.head()"
      ],
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 62,
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>id</th>\n",
              "      <th>message</th>\n",
              "      <th>original</th>\n",
              "      <th>genre</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>2</td>\n",
              "      <td>Weather update - a cold front from Cuba that c...</td>\n",
              "      <td>Un front froid se retrouve sur Cuba ce matin. ...</td>\n",
              "      <td>direct</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>7</td>\n",
              "      <td>Is the Hurricane over or is it not over</td>\n",
              "      <td>Cyclone nan fini osinon li pa fini</td>\n",
              "      <td>direct</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>8</td>\n",
              "      <td>Looking for someone but no name</td>\n",
              "      <td>Patnm, di Maryani relem pou li banm nouvel li ...</td>\n",
              "      <td>direct</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>9</td>\n",
              "      <td>UN reports Leogane 80-90 destroyed. Only Hospi...</td>\n",
              "      <td>UN reports Leogane 80-90 destroyed. Only Hospi...</td>\n",
              "      <td>direct</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>12</td>\n",
              "      <td>says: west side of Haiti, rest of the country ...</td>\n",
              "      <td>facade ouest d Haiti et le reste du pays aujou...</td>\n",
              "      <td>direct</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   id                                            message  \\\n",
              "0   2  Weather update - a cold front from Cuba that c...   \n",
              "1   7            Is the Hurricane over or is it not over   \n",
              "2   8                    Looking for someone but no name   \n",
              "3   9  UN reports Leogane 80-90 destroyed. Only Hospi...   \n",
              "4  12  says: west side of Haiti, rest of the country ...   \n",
              "\n",
              "                                            original   genre  \n",
              "0  Un front froid se retrouve sur Cuba ce matin. ...  direct  \n",
              "1                 Cyclone nan fini osinon li pa fini  direct  \n",
              "2  Patnm, di Maryani relem pou li banm nouvel li ...  direct  \n",
              "3  UN reports Leogane 80-90 destroyed. Only Hospi...  direct  \n",
              "4  facade ouest d Haiti et le reste du pays aujou...  direct  "
            ]
          },
          "metadata": {}
        }
      ],
      "execution_count": 62,
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "# concatenate the original dataframe with the new `categories` dataframe\n",
        "df = pd.concat([df, categories], axis=1)\n",
        "df.head()"
      ],
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 63,
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>id</th>\n",
              "      <th>message</th>\n",
              "      <th>original</th>\n",
              "      <th>genre</th>\n",
              "      <th>related</th>\n",
              "      <th>request</th>\n",
              "      <th>offer</th>\n",
              "      <th>aid_related</th>\n",
              "      <th>medical_help</th>\n",
              "      <th>medical_products</th>\n",
              "      <th>...</th>\n",
              "      <th>aid_centers</th>\n",
              "      <th>other_infrastructure</th>\n",
              "      <th>weather_related</th>\n",
              "      <th>floods</th>\n",
              "      <th>storm</th>\n",
              "      <th>fire</th>\n",
              "      <th>earthquake</th>\n",
              "      <th>cold</th>\n",
              "      <th>other_weather</th>\n",
              "      <th>direct_report</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>2</td>\n",
              "      <td>Weather update - a cold front from Cuba that c...</td>\n",
              "      <td>Un front froid se retrouve sur Cuba ce matin. ...</td>\n",
              "      <td>direct</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>7</td>\n",
              "      <td>Is the Hurricane over or is it not over</td>\n",
              "      <td>Cyclone nan fini osinon li pa fini</td>\n",
              "      <td>direct</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>8</td>\n",
              "      <td>Looking for someone but no name</td>\n",
              "      <td>Patnm, di Maryani relem pou li banm nouvel li ...</td>\n",
              "      <td>direct</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>9</td>\n",
              "      <td>UN reports Leogane 80-90 destroyed. Only Hospi...</td>\n",
              "      <td>UN reports Leogane 80-90 destroyed. Only Hospi...</td>\n",
              "      <td>direct</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>12</td>\n",
              "      <td>says: west side of Haiti, rest of the country ...</td>\n",
              "      <td>facade ouest d Haiti et le reste du pays aujou...</td>\n",
              "      <td>direct</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>5 rows × 40 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "   id                                            message  \\\n",
              "0   2  Weather update - a cold front from Cuba that c...   \n",
              "1   7            Is the Hurricane over or is it not over   \n",
              "2   8                    Looking for someone but no name   \n",
              "3   9  UN reports Leogane 80-90 destroyed. Only Hospi...   \n",
              "4  12  says: west side of Haiti, rest of the country ...   \n",
              "\n",
              "                                            original   genre  related  \\\n",
              "0  Un front froid se retrouve sur Cuba ce matin. ...  direct        1   \n",
              "1                 Cyclone nan fini osinon li pa fini  direct        1   \n",
              "2  Patnm, di Maryani relem pou li banm nouvel li ...  direct        1   \n",
              "3  UN reports Leogane 80-90 destroyed. Only Hospi...  direct        1   \n",
              "4  facade ouest d Haiti et le reste du pays aujou...  direct        1   \n",
              "\n",
              "   request  offer  aid_related  medical_help  medical_products  ...  \\\n",
              "0        0      0            0             0                 0  ...   \n",
              "1        0      0            1             0                 0  ...   \n",
              "2        0      0            0             0                 0  ...   \n",
              "3        1      0            1             0                 1  ...   \n",
              "4        0      0            0             0                 0  ...   \n",
              "\n",
              "   aid_centers  other_infrastructure  weather_related  floods  storm  fire  \\\n",
              "0            0                     0                0       0      0     0   \n",
              "1            0                     0                1       0      1     0   \n",
              "2            0                     0                0       0      0     0   \n",
              "3            0                     0                0       0      0     0   \n",
              "4            0                     0                0       0      0     0   \n",
              "\n",
              "   earthquake  cold  other_weather  direct_report  \n",
              "0           0     0              0              0  \n",
              "1           0     0              0              0  \n",
              "2           0     0              0              0  \n",
              "3           0     0              0              0  \n",
              "4           0     0              0              0  \n",
              "\n[5 rows x 40 columns]"
            ]
          },
          "metadata": {}
        }
      ],
      "execution_count": 63,
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 6. Remove duplicates.\n",
        "- Check how many duplicates are in this dataset.\n",
        "- Drop the duplicates.\n",
        "- Confirm duplicates were removed."
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "# making a copy before cleaning the dataset\n",
        "df_clean = df.copy()"
      ],
      "outputs": [],
      "execution_count": 64,
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "# checking the number of columns and rows before cleaning\n",
        "df_clean.shape"
      ],
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 65,
          "data": {
            "text/plain": [
              "(26386, 40)"
            ]
          },
          "metadata": {}
        }
      ],
      "execution_count": 65,
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "# checking the number of duplicates\n",
        "df_clean.duplicated().sum()"
      ],
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 66,
          "data": {
            "text/plain": [
              "170"
            ]
          },
          "metadata": {}
        }
      ],
      "execution_count": 66,
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "# dropping duplicates\n",
        "df_clean = df.drop_duplicates()"
      ],
      "outputs": [],
      "execution_count": 67,
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "# checking number of duplicates\n",
        "print(df_clean.duplicated().any())\n",
        "print(df_clean.duplicated().sum())"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "False\n",
            "0\n"
          ]
        }
      ],
      "execution_count": 68,
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "# checking the number of columns and rows after dropping dublicates\n",
        "df_clean.shape"
      ],
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 69,
          "data": {
            "text/plain": [
              "(26216, 40)"
            ]
          },
          "metadata": {}
        }
      ],
      "execution_count": 69,
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "# making sure the data is clean and has only two values 0 and 1 for each category by \n",
        "#reporting the columns with other than 2 unique values\n",
        "col_values = [x for x in categories if len(np.unique(categories[x])) != 2]\n",
        "col_values"
      ],
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 70,
          "data": {
            "text/plain": [
              "['related', 'child_alone']"
            ]
          },
          "metadata": {}
        }
      ],
      "execution_count": 70,
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "# printing the number of messages of each unique for related and child_alone\n",
        "print(categories['related'].value_counts())\n",
        "print(categories['child_alone'].value_counts())"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "1    20042\n",
            "0     6140\n",
            "2      204\n",
            "Name: related, dtype: int64\n",
            "0    26386\n",
            "Name: child_alone, dtype: int64\n"
          ]
        }
      ],
      "execution_count": 71,
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "# dropping child_alone as it doesn't add any info\n",
        "df_clean = df_clean.drop('child_alone', axis=1)"
      ],
      "outputs": [],
      "execution_count": 72,
      "metadata": {
        "scrolled": true
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# priniting the messages that have a category related_2\n",
        "print(df_clean.message[df_clean.related == 2])"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "117      Dans la zone de Saint Etienne la route de Jacm...\n",
            "221      . .. i with limited means. Certain patients co...\n",
            "307      The internet caf Net@le that's by the Dal road...\n",
            "462      Bonsoir, on est a bon repos aprs la compagnie ...\n",
            "578      URGENT CRECHE ORPHANAGE KAY TOUT TIMOUN CROIX ...\n",
            "657      elle est vraiment malade et a besoin d'aide. u...\n",
            "889      no authority has passed by to see us. We don't...\n",
            "903      It's Over in Gressier. The population in the a...\n",
            "931      we sleep with the baby. Thanks in advance for ...\n",
            "937      I need help in Jrmie because I was in Port-au-...\n",
            "939      fsa pou mwen s v p map mouri mwen gen tout po ...\n",
            "1234     .. Gonaives, in a place called Canal Bois in F...\n",
            "1256     GEN YON TIBEBE KI MALAD NAN KOU PASKE BLOK TON...\n",
            "1317     don't understand the first part. .. understand...\n",
            "1409     Est-ce que ya monde qui aller U. s. a et qui o...\n",
            "1506     Gens ont information qui dit que si on moins c...\n",
            "1694         This is my address : Cersine 8 Prolong. .. . \n",
            "1788     I would like to have some information on the l...\n",
            "2354     No location : Ou se trouve l'ambassade du Snga...\n",
            "2479     Je veux savoir comment je peux obtenir de l'ai...\n",
            "2544         142, Ruelle Beaulieu, Mon Repos 44 Carrefour \n",
            "2560     . ..  Esdras. Address : Solidarit Village, Tou...\n",
            "3033     Route des dalles prolonges #277 after Morne Je...\n",
            "3138     we are in rue Dessalines Petit goaves and it n...\n",
            "3374     I am very happy to get the messages that tell ...\n",
            "3630                   mouin vle kite ayiti depi kek jou. \n",
            "4609     Mwen ta renmen jwenn yon djb. ske gen posiblit...\n",
            "4651     (Delmas 33 Charboniere infomatyon s'il vous pl...\n",
            "4746                  Carrefour Feuille rue Mgr Prol # 39 \n",
            "4805                 Ede m nan lort pou genyen yon bourss \n",
            "                               ...                        \n",
            "12392    SYED EJAZ BUKHARI MAOZA SHAH WALI TEHSIL ALI P...\n",
            "12396    TAUSEEF KHAN S/O AKBAR DIN MIANWALI TEH ESSA K...\n",
            "12397    TAUSEEF KHAN S/O AKBAR DIN MIANWALI TEH ESSA K...\n",
            "12398    TEH SUHBAT PUR VILL NASEER KHAN KHOSA MY PRESE...\n",
            "12407    UN aur UN K Nimaidon ko Apeel krt hain k? Hama...\n",
            "12416    I allhadino khaskheli UC chatto a dehe a shalu...\n",
            "12418    village koor mir mohmmad u c burira takuka kha...\n",
            "12425    gaav valay aahar aor aarthik samashyaa saamna ...\n",
            "12427      CONNFCTION     /         .     SOLANGI,     ...\n",
            "12479    @jujoffer: So Frankenstorm has been reduced to...\n",
            "12685    #SandyUntappd ‚àö‚â• Drinking an Oatmeal Stout...\n",
            "12758    atrickSF @antmarshall @kleankut1 @somarlavous ...\n",
            "13732    Les habitants, dont l'activité est essentielle...\n",
            "14938    Jusqu'à présent, les 133 femmes qui participen...\n",
            "15029    Despite rapid deliveries of tents and other re...\n",
            "15609    All available resources were put forward to de...\n",
            "15933    The six countries will also adopt a training c...\n",
            "15979    \"It was very windy and there were heavy thunde...\n",
            "16046    The 50 international hospital staff lived in a...\n",
            "16925    L'Armée suisse a cependant le savoir-faire et ...\n",
            "17528    PETR V. ILIICHEV (Russian Federation) said the...\n",
            "18381    * Par ailleurs, aucune victime française n'a é...\n",
            "18644    Refugees International therefore recommends that:\n",
            "19849    This combination of low agricultural productio...\n",
            "20192    crop, loss of hydro-electric generating capaci...\n",
            "20465         transferred to a sanitary landfill site by a\n",
            "20636    Families also have solar lamps which can be re...\n",
            "22481    Read the [full blog post](http://www.odi.org.u...\n",
            "23537    Actualmente e independientemente de la ayuda d...\n",
            "25385    Mali's former President Amadou Toumani Touré -...\n",
            "Name: message, Length: 188, dtype: object\n"
          ]
        }
      ],
      "execution_count": 73,
      "metadata": {
        "scrolled": true
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# dropping messages in related_2 as they appear in many different languages\n",
        "df_clean = df_clean[df_clean.related != 2]\n",
        "df_clean.sample(3)"
      ],
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 74,
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>id</th>\n",
              "      <th>message</th>\n",
              "      <th>original</th>\n",
              "      <th>genre</th>\n",
              "      <th>related</th>\n",
              "      <th>request</th>\n",
              "      <th>offer</th>\n",
              "      <th>aid_related</th>\n",
              "      <th>medical_help</th>\n",
              "      <th>medical_products</th>\n",
              "      <th>...</th>\n",
              "      <th>aid_centers</th>\n",
              "      <th>other_infrastructure</th>\n",
              "      <th>weather_related</th>\n",
              "      <th>floods</th>\n",
              "      <th>storm</th>\n",
              "      <th>fire</th>\n",
              "      <th>earthquake</th>\n",
              "      <th>cold</th>\n",
              "      <th>other_weather</th>\n",
              "      <th>direct_report</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>24288</th>\n",
              "      <td>27899</td>\n",
              "      <td>As news of the killing spread, people in Gilgi...</td>\n",
              "      <td>NaN</td>\n",
              "      <td>news</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>21972</th>\n",
              "      <td>25319</td>\n",
              "      <td>\"We have lived with burst sewers and heaps of ...</td>\n",
              "      <td>NaN</td>\n",
              "      <td>news</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9676</th>\n",
              "      <td>10825</td>\n",
              "      <td>NOTES: This kind of message is not important f...</td>\n",
              "      <td>Fatra nan cadet jeremie kamyon an pa janm vini...</td>\n",
              "      <td>direct</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>3 rows × 39 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "          id                                            message  \\\n",
              "24288  27899  As news of the killing spread, people in Gilgi...   \n",
              "21972  25319  \"We have lived with burst sewers and heaps of ...   \n",
              "9676   10825  NOTES: This kind of message is not important f...   \n",
              "\n",
              "                                                original   genre  related  \\\n",
              "24288                                                NaN    news        1   \n",
              "21972                                                NaN    news        1   \n",
              "9676   Fatra nan cadet jeremie kamyon an pa janm vini...  direct        0   \n",
              "\n",
              "       request  offer  aid_related  medical_help  medical_products  ...  \\\n",
              "24288        0      0            1             0                 0  ...   \n",
              "21972        0      0            0             0                 0  ...   \n",
              "9676         0      0            0             0                 0  ...   \n",
              "\n",
              "       aid_centers  other_infrastructure  weather_related  floods  storm  \\\n",
              "24288            0                     0                0       0      0   \n",
              "21972            0                     0                0       0      0   \n",
              "9676             0                     0                0       0      0   \n",
              "\n",
              "       fire  earthquake  cold  other_weather  direct_report  \n",
              "24288     0           0     0              0              0  \n",
              "21972     0           0     0              0              1  \n",
              "9676      0           0     0              0              0  \n",
              "\n[3 rows x 39 columns]"
            ]
          },
          "metadata": {}
        }
      ],
      "execution_count": 74,
      "metadata": {
        "scrolled": false
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# checking the new number of columns and rows\n",
        "df_clean.shape"
      ],
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 75,
          "data": {
            "text/plain": [
              "(26028, 39)"
            ]
          },
          "metadata": {}
        }
      ],
      "execution_count": 75,
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "# getting dummies for genre\n",
        "print(df_clean.genre.unique())\n",
        "genre_dummies = pd.DataFrame(pd.get_dummies(df_clean['genre']))\n",
        "genre_dummies.sample(5)"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['direct' 'social' 'news']\n"
          ]
        },
        {
          "output_type": "execute_result",
          "execution_count": 76,
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>direct</th>\n",
              "      <th>news</th>\n",
              "      <th>social</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>23835</th>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>17351</th>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1925</th>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>21676</th>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>16201</th>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "       direct  news  social\n",
              "23835       0     1       0\n",
              "17351       0     1       0\n",
              "1925        1     0       0\n",
              "21676       0     1       0\n",
              "16201       0     1       0"
            ]
          },
          "metadata": {}
        }
      ],
      "execution_count": 76,
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "# concatenate the dataframe with the new genre dataframe\n",
        "df_final = pd.concat([df_clean.drop('genre', axis=1), genre_dummies], axis=1)\n",
        "df_final = df_final.drop('social', axis=1)\n",
        "df_final.sample(3)"
      ],
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 77,
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>id</th>\n",
              "      <th>message</th>\n",
              "      <th>original</th>\n",
              "      <th>related</th>\n",
              "      <th>request</th>\n",
              "      <th>offer</th>\n",
              "      <th>aid_related</th>\n",
              "      <th>medical_help</th>\n",
              "      <th>medical_products</th>\n",
              "      <th>search_and_rescue</th>\n",
              "      <th>...</th>\n",
              "      <th>weather_related</th>\n",
              "      <th>floods</th>\n",
              "      <th>storm</th>\n",
              "      <th>fire</th>\n",
              "      <th>earthquake</th>\n",
              "      <th>cold</th>\n",
              "      <th>other_weather</th>\n",
              "      <th>direct_report</th>\n",
              "      <th>direct</th>\n",
              "      <th>news</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>15870</th>\n",
              "      <td>18576</td>\n",
              "      <td>As per IMD, snowfall upto 2m has occurred at m...</td>\n",
              "      <td>NaN</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>...</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>23199</th>\n",
              "      <td>26682</td>\n",
              "      <td>However, the impact of the floods on industry,...</td>\n",
              "      <td>NaN</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>...</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1325</th>\n",
              "      <td>1559</td>\n",
              "      <td>after how muche time does the earthquake stop?...</td>\n",
              "      <td>apre Kombyen tan tranbleman de ta a ap Fini?es...</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>...</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>3 rows × 40 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "          id                                            message  \\\n",
              "15870  18576  As per IMD, snowfall upto 2m has occurred at m...   \n",
              "23199  26682  However, the impact of the floods on industry,...   \n",
              "1325    1559  after how muche time does the earthquake stop?...   \n",
              "\n",
              "                                                original  related  request  \\\n",
              "15870                                                NaN        1        0   \n",
              "23199                                                NaN        1        0   \n",
              "1325   apre Kombyen tan tranbleman de ta a ap Fini?es...        1        0   \n",
              "\n",
              "       offer  aid_related  medical_help  medical_products  search_and_rescue  \\\n",
              "15870      0            0             0                 0                  0   \n",
              "23199      0            0             0                 0                  0   \n",
              "1325       0            0             0                 0                  0   \n",
              "\n",
              "       ...  weather_related  floods  storm  fire  earthquake  cold  \\\n",
              "15870  ...                1       0      0     0           0     1   \n",
              "23199  ...                1       1      0     0           0     0   \n",
              "1325   ...                1       0      0     0           1     0   \n",
              "\n",
              "       other_weather  direct_report  direct  news  \n",
              "15870              0              0       0     1  \n",
              "23199              0              0       0     1  \n",
              "1325               0              1       1     0  \n",
              "\n[3 rows x 40 columns]"
            ]
          },
          "metadata": {}
        }
      ],
      "execution_count": 77,
      "metadata": {
        "scrolled": true
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# checking the final number of columns and rows\n",
        "df_final.shape"
      ],
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 78,
          "data": {
            "text/plain": [
              "(26028, 40)"
            ]
          },
          "metadata": {}
        }
      ],
      "execution_count": 78,
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 7. Save the clean dataset into an sqlite database.\n",
        "You can do this with pandas [`to_sql` method](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.to_sql.html) combined with the SQLAlchemy library. Remember to import SQLAlchemy's `create_engine` in the first cell of this notebook to use it below."
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "engine = create_engine('sqlite:///DisasterResponse.db')\n",
        "df_final.to_sql('Messages', engine, index=False, if_exists='replace')"
      ],
      "outputs": [],
      "execution_count": 79,
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 8. Use this notebook to complete `etl_pipeline.py`\n",
        "Use the template file attached in the Resources folder to write a script that runs the steps above to create a database based on new datasets specified by the user. Alternatively, you can complete `etl_pipeline.py` in the classroom on the `Project Workspace IDE` coming later."
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [],
      "outputs": [],
      "execution_count": null,
      "metadata": {}
    }
  ],
  "metadata": {
    "kernelspec": {
      "name": "python3",
      "language": "python",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python",
      "version": "3.7.3",
      "mimetype": "text/x-python",
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "pygments_lexer": "ipython3",
      "nbconvert_exporter": "python",
      "file_extension": ".py"
    },
    "kernel_info": {
      "name": "python3"
    },
    "nteract": {
      "version": "0.12.3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 2
}