preprocessing/Part_1_Missing_Values.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/drshahizan/Python_Tutorial/blob/main/preprocessing/Part_1_Missing_Values.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Source**: [Data preprocessing with Python Pandas](https://towardsdatascience.com/data-preprocessing-with-python-pandas-part-1-missing-data-45e76b781993)"
      ],
      "metadata": {
        "id": "PmCn7qCjf3q_"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e9ddWFTieuC2"
      },
      "source": [
        "# Data Preprocessing using pandas\n",
        "This tutorial explains how to preprocess data using the pandas library. Preprocessing is the process of doing a pre-analysis of data, in order to transform them into a standard and normalized format.\n",
        "Preprocessing involves the following aspects:\n",
        "* missing values\n",
        "* data standardization\n",
        "* data normalization\n",
        "* data binning\n",
        "\n",
        "In this tutorial we will use the dataset related to Hepatitis, which can be downloaded from [this link](https://https://raw.githubusercontent.com/drshahizan/dataset/main/hepatitis.csv).\n",
        "\n",
        "In this tutorial we deal only with missing values.\n",
        "\n",
        "## Import data\n",
        "Firstly, import data using the pandas library and convert them into a dataframe. Through the `head(10)` method we print only the first 10 rows of the dataset"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "scrolled": true,
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 427
        },
        "id": "CuoE8y-keuC5",
        "outputId": "dd9a07a4-7a35-4c78-eb66-b8b2d1701621"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "   age     sex steroid  antivirals fatigue malaise anorexia liver_big  \\\n",
              "0   30    male   False       False   False   False    False     False   \n",
              "1   50  female   False       False    True   False    False     False   \n",
              "2   78  female    True       False    True   False    False      True   \n",
              "3   31  female     NaN        True   False   False    False      True   \n",
              "4   34  female    True       False   False   False    False      True   \n",
              "5   34  female    True       False   False   False    False      True   \n",
              "6   51  female   False       False    True   False     True      True   \n",
              "7   23  female    True       False   False   False    False      True   \n",
              "8   39  female    True       False    True   False    False      True   \n",
              "9   30  female    True       False   False   False    False      True   \n",
              "\n",
              "  liver_firm spleen_palpable spiders ascites varices  bilirubin  \\\n",
              "0      False           False   False   False   False        1.0   \n",
              "1      False           False   False   False   False        0.9   \n",
              "2      False           False   False   False   False        0.7   \n",
              "3      False           False   False   False   False        0.7   \n",
              "4      False           False   False   False   False        1.0   \n",
              "5      False           False   False   False   False        0.9   \n",
              "6      False            True    True   False   False        NaN   \n",
              "7      False           False   False   False   False        1.0   \n",
              "8       True           False   False   False   False        0.7   \n",
              "9      False           False   False   False   False        1.0   \n",
              "\n",
              "   alk_phosphate   sgot  albumin  protime  histology class  \n",
              "0           85.0   18.0      4.0      NaN      False  live  \n",
              "1          135.0   42.0      3.5      NaN      False  live  \n",
              "2           96.0   32.0      4.0      NaN      False  live  \n",
              "3           46.0   52.0      4.0     80.0      False  live  \n",
              "4            NaN  200.0      4.0      NaN      False  live  \n",
              "5           95.0   28.0      4.0     75.0      False  live  \n",
              "6            NaN    NaN      NaN      NaN      False   die  \n",
              "7            NaN    NaN      NaN      NaN      False  live  \n",
              "8            NaN   48.0      4.4      NaN      False  live  \n",
              "9            NaN  120.0      3.9      NaN      False  live  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-a1793d8b-cbc8-417a-aee5-72b0f7feb2f7\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>age</th>\n",
              "      <th>sex</th>\n",
              "      <th>steroid</th>\n",
              "      <th>antivirals</th>\n",
              "      <th>fatigue</th>\n",
              "      <th>malaise</th>\n",
              "      <th>anorexia</th>\n",
              "      <th>liver_big</th>\n",
              "      <th>liver_firm</th>\n",
              "      <th>spleen_palpable</th>\n",
              "      <th>spiders</th>\n",
              "      <th>ascites</th>\n",
              "      <th>varices</th>\n",
              "      <th>bilirubin</th>\n",
              "      <th>alk_phosphate</th>\n",
              "      <th>sgot</th>\n",
              "      <th>albumin</th>\n",
              "      <th>protime</th>\n",
              "      <th>histology</th>\n",
              "      <th>class</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>30</td>\n",
              "      <td>male</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>1.0</td>\n",
              "      <td>85.0</td>\n",
              "      <td>18.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>False</td>\n",
              "      <td>live</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>50</td>\n",
              "      <td>female</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>0.9</td>\n",
              "      <td>135.0</td>\n",
              "      <td>42.0</td>\n",
              "      <td>3.5</td>\n",
              "      <td>NaN</td>\n",
              "      <td>False</td>\n",
              "      <td>live</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>78</td>\n",
              "      <td>female</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>0.7</td>\n",
              "      <td>96.0</td>\n",
              "      <td>32.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>False</td>\n",
              "      <td>live</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>31</td>\n",
              "      <td>female</td>\n",
              "      <td>NaN</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>0.7</td>\n",
              "      <td>46.0</td>\n",
              "      <td>52.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>80.0</td>\n",
              "      <td>False</td>\n",
              "      <td>live</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>34</td>\n",
              "      <td>female</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>1.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>200.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>False</td>\n",
              "      <td>live</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>34</td>\n",
              "      <td>female</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>0.9</td>\n",
              "      <td>95.0</td>\n",
              "      <td>28.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>75.0</td>\n",
              "      <td>False</td>\n",
              "      <td>live</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>51</td>\n",
              "      <td>female</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>False</td>\n",
              "      <td>die</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>23</td>\n",
              "      <td>female</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>1.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>False</td>\n",
              "      <td>live</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>39</td>\n",
              "      <td>female</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>0.7</td>\n",
              "      <td>NaN</td>\n",
              "      <td>48.0</td>\n",
              "      <td>4.4</td>\n",
              "      <td>NaN</td>\n",
              "      <td>False</td>\n",
              "      <td>live</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>30</td>\n",
              "      <td>female</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>1.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>120.0</td>\n",
              "      <td>3.9</td>\n",
              "      <td>NaN</td>\n",
              "      <td>False</td>\n",
              "      <td>live</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-a1793d8b-cbc8-417a-aee5-72b0f7feb2f7')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-a1793d8b-cbc8-417a-aee5-72b0f7feb2f7 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-a1793d8b-cbc8-417a-aee5-72b0f7feb2f7');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 1
        }
      ],
      "source": [
        "import pandas as pd\n",
        "df = pd.read_csv('https://raw.githubusercontent.com/drshahizan/dataset/main/hepatitis.csv')\n",
        "df.head(10)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "z-NQC3UpeuC6"
      },
      "source": [
        "## Identify missing values\n",
        "We note that the dataset presents some problems. For example, the column email is not available for all the rows. In some cases it presents the `NaN` value, which means that the value is missing. \n",
        "\n",
        "In order to check whether our dataset contains missing values, we can use the function `isna()`, which returns if an cell of the dataset if `NaN` or not. Then we can count how many missing values there are for each column."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "tayZ8jC2euC6",
        "outputId": "7eee08b8-c2c6-41b9-c407-04449a22afd1"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "age                 0\n",
              "sex                 0\n",
              "steroid             1\n",
              "antivirals          0\n",
              "fatigue             1\n",
              "malaise             1\n",
              "anorexia            1\n",
              "liver_big          10\n",
              "liver_firm         11\n",
              "spleen_palpable     5\n",
              "spiders             5\n",
              "ascites             5\n",
              "varices             5\n",
              "bilirubin           6\n",
              "alk_phosphate      29\n",
              "sgot                4\n",
              "albumin            16\n",
              "protime            67\n",
              "histology           0\n",
              "class               0\n",
              "dtype: int64"
            ]
          },
          "metadata": {},
          "execution_count": 2
        }
      ],
      "source": [
        "df.isna().sum()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1IpfeymdeuC7"
      },
      "source": [
        "Now we can count the percentage of missing values for each column, simply by dividing the previous result by the length of the dataset (`len(df)`) and multiplying per 100."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "sEZXF1CxeuC7",
        "outputId": "7438b79b-eeec-4dd8-87b8-7c728cc02ffd"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "age                 0.000000\n",
              "sex                 0.000000\n",
              "steroid             0.645161\n",
              "antivirals          0.000000\n",
              "fatigue             0.645161\n",
              "malaise             0.645161\n",
              "anorexia            0.645161\n",
              "liver_big           6.451613\n",
              "liver_firm          7.096774\n",
              "spleen_palpable     3.225806\n",
              "spiders             3.225806\n",
              "ascites             3.225806\n",
              "varices             3.225806\n",
              "bilirubin           3.870968\n",
              "alk_phosphate      18.709677\n",
              "sgot                2.580645\n",
              "albumin            10.322581\n",
              "protime            43.225806\n",
              "histology           0.000000\n",
              "class               0.000000\n",
              "dtype: float64"
            ]
          },
          "metadata": {},
          "execution_count": 3
        }
      ],
      "source": [
        "df.isna().sum()/len(df)*100"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Be4sGwBteuC7"
      },
      "source": [
        "When dealing with missing values, different alternatives can be applied:\n",
        "* check the source, for example by contacting the data source to correct the missing values\n",
        "* drop missing values\n",
        "* replace the missing value with a value\n",
        "* leave the missing value as it is.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fqu1dIVAeuC7"
      },
      "source": [
        "## Drop missing values\n",
        "Dropping missing values can be one of the following alternatives:\n",
        "* remove rows having missing values\n",
        "* remove the whole column containing missing values\n",
        "We can use the `dropna()` by specifying the `axis` to be considered. If we set `axis = 0` we drop the entire row, if we set `axis = 1` we drop the whole column. If we apply the function `df.dropna(axis=0)` 80 rows of the dataset remain. If we apply the function `df.dropna(axis=1)`, only the columns age, sex, antivirals, histology and class remain. However, removed values are not applied to the original dataframe, but only to the result. We can use the argument `inplace=True` in order to store changes in the original dataframe `df` (`df.dropna(axis=1,inplace=True)`)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 424
        },
        "id": "0gh8PwUgeuC8",
        "outputId": "50a8c826-61f4-49cb-866c-e384616c9e36"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "     age     sex  antivirals  histology class\n",
              "0     30    male       False      False  live\n",
              "1     50  female       False      False  live\n",
              "2     78  female       False      False  live\n",
              "3     31  female        True      False  live\n",
              "4     34  female       False      False  live\n",
              "..   ...     ...         ...        ...   ...\n",
              "150   46  female       False       True   die\n",
              "151   44  female       False       True  live\n",
              "152   61  female       False       True  live\n",
              "153   53    male       False       True  live\n",
              "154   43  female       False       True   die\n",
              "\n",
              "[155 rows x 5 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-cbf42136-3842-470f-abc3-d51d960143b5\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>age</th>\n",
              "      <th>sex</th>\n",
              "      <th>antivirals</th>\n",
              "      <th>histology</th>\n",
              "      <th>class</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>30</td>\n",
              "      <td>male</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>live</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>50</td>\n",
              "      <td>female</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>live</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>78</td>\n",
              "      <td>female</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>live</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>31</td>\n",
              "      <td>female</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>live</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>34</td>\n",
              "      <td>female</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>live</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>150</th>\n",
              "      <td>46</td>\n",
              "      <td>female</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>die</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>151</th>\n",
              "      <td>44</td>\n",
              "      <td>female</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>live</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>152</th>\n",
              "      <td>61</td>\n",
              "      <td>female</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>live</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>153</th>\n",
              "      <td>53</td>\n",
              "      <td>male</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>live</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>154</th>\n",
              "      <td>43</td>\n",
              "      <td>female</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>die</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>155 rows × 5 columns</p>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-cbf42136-3842-470f-abc3-d51d960143b5')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-cbf42136-3842-470f-abc3-d51d960143b5 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-cbf42136-3842-470f-abc3-d51d960143b5');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 4
        }
      ],
      "source": [
        "df.dropna(axis=1)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fNeLmJOCeuC8"
      },
      "source": [
        "As an alternative, we can specify only the column on which the dropping operation must be applied. In the following example, only missing rows related to the column `liver_big` are considered. This can be achieved through the `subset` parameter, which permits to specify the subset of columns where to apply the dropping operation."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "id": "eI38BwB5euC8"
      },
      "outputs": [],
      "source": [
        "df.dropna(subset=['liver_big'],axis=0,inplace=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RjCMZzhFeuC9"
      },
      "source": [
        "Now we can check whether there are still missing values for the column `indirizzo`."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ayq3MplZeuC9"
      },
      "source": [
        "df.isna().sum()/len(df)*100"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Rp8BRtGEeuC9"
      },
      "source": [
        "Another alternative involves the dropping of columns where a certain percentage of not-null values is available. This can be achieved through the `thresh` parameter. In the following example we keep only columns where there are at least the 80% of not null values. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "id": "VOOjtF4eeuC9"
      },
      "outputs": [],
      "source": [
        "df.dropna(thresh=0.8*len(df),axis=1,inplace=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fw1QuF9VeuC9"
      },
      "source": [
        "## Replace missing values\n",
        "A good strategy when dealing with missing values involves their replacement with another value. Usually, the following strategies are adopted:\n",
        "* for numerical values replace the missing value with the average value of the column\n",
        "* for categorial values replace the missing value with the most frequent value of the column \n",
        "* use other functions\n",
        "\n",
        "In order to replace missing values, three functions can be used: `fillna()`, `replace()` and `interpolate()`.\n",
        "The `fillna()` function replaces all the NaN values with the value passed as argument.\n",
        "For example, for numerical values, all the NaN values in the numeric columns could be replaced with the average value. \n",
        "In order to list the type of a column, we can use the attribute `dtypes` as follows:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "JW3U61MDeuC9",
        "outputId": "c43c33f7-a7e9-4d3e-c874-dfe2e8f82e88"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "age                  int64\n",
              "sex                 object\n",
              "steroid             object\n",
              "antivirals            bool\n",
              "fatigue             object\n",
              "malaise             object\n",
              "anorexia            object\n",
              "liver_big           object\n",
              "liver_firm          object\n",
              "spleen_palpable     object\n",
              "spiders             object\n",
              "ascites             object\n",
              "varices             object\n",
              "bilirubin          float64\n",
              "alk_phosphate      float64\n",
              "sgot               float64\n",
              "albumin            float64\n",
              "histology             bool\n",
              "class               object\n",
              "dtype: object"
            ]
          },
          "metadata": {},
          "execution_count": 7
        }
      ],
      "source": [
        "df.dtypes"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "irmIJ0-peuC9"
      },
      "source": [
        "## Numeric columns\n",
        "Firstly, we select numeric columns."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "id": "JDG932hVeuC-"
      },
      "outputs": [],
      "source": [
        "import numpy as np\n",
        "numeric = df.select_dtypes(include=np.number)\n",
        "numeric_columns = numeric.columns"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3VdBwYfqeuC-"
      },
      "source": [
        "Then, we fill the NaN values of numeric columns with the average value, given by the `df.mean()` function."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "GG1fJ_XzeuC-",
        "outputId": "b11137a7-466a-43c1-cbaf-e8ae734a6396"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "<ipython-input-9-fdaac0744f5f>:1: FutureWarning: Dropping of nuisance columns in DataFrame reductions (with 'numeric_only=None') is deprecated; in a future version this will raise TypeError.  Select only valid columns before calling the reduction.\n",
            "  df[numeric_columns] = df[numeric_columns].fillna(df.mean())\n"
          ]
        }
      ],
      "source": [
        "df[numeric_columns] = df[numeric_columns].fillna(df.mean())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mgvkXcQteuC-"
      },
      "source": [
        "Now, we can check whether the NaN values in numeric columns have been removed."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "olMaxNGzeuC-",
        "outputId": "9b77dae8-9e05-496b-9fb3-bc3dd3e525b3"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "age                0.000000\n",
              "sex                0.000000\n",
              "steroid            0.689655\n",
              "antivirals         0.000000\n",
              "fatigue            0.000000\n",
              "malaise            0.000000\n",
              "anorexia           0.000000\n",
              "liver_big          0.000000\n",
              "liver_firm         0.689655\n",
              "spleen_palpable    0.689655\n",
              "spiders            0.689655\n",
              "ascites            0.689655\n",
              "varices            0.689655\n",
              "bilirubin          0.000000\n",
              "alk_phosphate      0.000000\n",
              "sgot               0.000000\n",
              "albumin            0.000000\n",
              "histology          0.000000\n",
              "class              0.000000\n",
              "dtype: float64"
            ]
          },
          "metadata": {},
          "execution_count": 10
        }
      ],
      "source": [
        "df.isna().sum()/len(df)*100"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SIhe0xy6euC-"
      },
      "source": [
        "## Categorial columns\n",
        "We note that in `dtypes` the categorial columns are described as objects. Thus we can select the `object` columns. We would like to consider only boolean columns. However the `object` type includes also the column `class`, which is a string. We select all the object columns, and then we remove from them the column `class`. Then we can convert the type of the result to `bool`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "S3H9MOiVeuC-",
        "outputId": "e1451e48-5119-46f7-b2c7-41afa3cd77f0"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "<ipython-input-11-f600210b030b>:1: DeprecationWarning: `np.object` is a deprecated alias for the builtin `object`. To silence this warning, use `object` by itself. Doing this will not modify any behavior and is safe. \n",
            "Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations\n",
            "  boolean_columns = df.select_dtypes(include=np.object).columns.tolist()\n"
          ]
        }
      ],
      "source": [
        "boolean_columns = df.select_dtypes(include=np.object).columns.tolist()\n",
        "boolean_columns.remove('class')\n",
        "df[boolean_columns] = df[boolean_columns].astype('bool')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UQ8UYK2meuC-"
      },
      "source": [
        "Now we can replace all the missing values for booleans with the most frequent value. We can use the `mode()` function to calculate the most frequent value. We use the `fillna()` function to replace missing values, but we could use also the `replace(old_value,new_value)` function."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 488
        },
        "id": "VzUS6tQHeuC-",
        "outputId": "6d4ee597-ce62-47fa-d1cb-2c7fb3b393e7"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "      sex  steroid  fatigue  malaise  anorexia  liver_big  liver_firm  \\\n",
              "0    True    False    False    False     False      False       False   \n",
              "1    True    False     True    False     False      False       False   \n",
              "2    True     True     True    False     False       True       False   \n",
              "3    True     True    False    False     False       True       False   \n",
              "4    True     True    False    False     False       True       False   \n",
              "..    ...      ...      ...      ...       ...        ...         ...   \n",
              "150  True     True     True     True      True       True       False   \n",
              "151  True     True     True    False     False       True        True   \n",
              "152  True    False     True     True     False      False        True   \n",
              "153  True    False     True    False     False       True       False   \n",
              "154  True     True     True    False     False       True       False   \n",
              "\n",
              "     spleen_palpable  spiders  ascites  varices  \n",
              "0              False    False    False    False  \n",
              "1              False    False    False    False  \n",
              "2              False    False    False    False  \n",
              "3              False    False    False    False  \n",
              "4              False    False    False    False  \n",
              "..               ...      ...      ...      ...  \n",
              "150            False     True     True     True  \n",
              "151            False    False    False    False  \n",
              "152            False     True    False    False  \n",
              "153             True     True    False     True  \n",
              "154             True     True     True    False  \n",
              "\n",
              "[145 rows x 11 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-49bf5f3f-ff7d-4c45-a18c-b0bd3469e6cf\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>sex</th>\n",
              "      <th>steroid</th>\n",
              "      <th>fatigue</th>\n",
              "      <th>malaise</th>\n",
              "      <th>anorexia</th>\n",
              "      <th>liver_big</th>\n",
              "      <th>liver_firm</th>\n",
              "      <th>spleen_palpable</th>\n",
              "      <th>spiders</th>\n",
              "      <th>ascites</th>\n",
              "      <th>varices</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>150</th>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>151</th>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>152</th>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>153</th>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>154</th>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>145 rows × 11 columns</p>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-49bf5f3f-ff7d-4c45-a18c-b0bd3469e6cf')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-49bf5f3f-ff7d-4c45-a18c-b0bd3469e6cf button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-49bf5f3f-ff7d-4c45-a18c-b0bd3469e6cf');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 12
        }
      ],
      "source": [
        "df[boolean_columns].fillna(df.mode())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "N-bNNUIheuC-"
      },
      "source": [
        "Now our dataset does not contain any missing value."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "gkSjl8c8euC-",
        "outputId": "1614b069-e604-47dc-f623-c611b4f45101"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "age                0.0\n",
              "sex                0.0\n",
              "steroid            0.0\n",
              "antivirals         0.0\n",
              "fatigue            0.0\n",
              "malaise            0.0\n",
              "anorexia           0.0\n",
              "liver_big          0.0\n",
              "liver_firm         0.0\n",
              "spleen_palpable    0.0\n",
              "spiders            0.0\n",
              "ascites            0.0\n",
              "varices            0.0\n",
              "bilirubin          0.0\n",
              "alk_phosphate      0.0\n",
              "sgot               0.0\n",
              "albumin            0.0\n",
              "histology          0.0\n",
              "class              0.0\n",
              "dtype: float64"
            ]
          },
          "metadata": {},
          "execution_count": 13
        }
      ],
      "source": [
        "df.isna().sum()/len(df)*100"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ylGqVPQheuC_"
      },
      "source": [
        "## Interpolation\n",
        "Another solution to replace missing values involves the usage of other functions, such as linear interpolation. In this case, for example, we could replace a missing value over a column, with the interpolation between the previous and the next ones. This can be achieved through the use of the `interpolate()` function.\n",
        "\n",
        "Since we have already managed all the missing values, we reload the dataset."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "WsDYu1RWeuC_",
        "outputId": "cdd6ec33-d8e3-4fc4-ff7c-ccba0008b727"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "age                 0.000000\n",
              "sex                 0.000000\n",
              "steroid             0.645161\n",
              "antivirals          0.000000\n",
              "fatigue             0.645161\n",
              "malaise             0.645161\n",
              "anorexia            0.645161\n",
              "liver_big           6.451613\n",
              "liver_firm          7.096774\n",
              "spleen_palpable     3.225806\n",
              "spiders             3.225806\n",
              "ascites             3.225806\n",
              "varices             3.225806\n",
              "bilirubin           3.870968\n",
              "alk_phosphate      18.709677\n",
              "sgot                2.580645\n",
              "albumin            10.322581\n",
              "protime            43.225806\n",
              "histology           0.000000\n",
              "class               0.000000\n",
              "dtype: float64"
            ]
          },
          "metadata": {},
          "execution_count": 14
        }
      ],
      "source": [
        "df = pd.read_csv('https://raw.githubusercontent.com/drshahizan/dataset/main/hepatitis.csv')\n",
        "df.isna().sum()/len(df)*100"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jIoQq-2EeuC_"
      },
      "source": [
        "We select only numeric columns."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {
        "id": "uS89zGrreuC_"
      },
      "outputs": [],
      "source": [
        "numeric = df.select_dtypes(include=np.number)\n",
        "numeric_columns = numeric.columns"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "_vNHI9yYeuC_",
        "outputId": "e95a584c-9360-486a-f4fd-39d08ac80c3f"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>age</th>\n",
              "      <th>sex</th>\n",
              "      <th>steroid</th>\n",
              "      <th>antivirals</th>\n",
              "      <th>fatigue</th>\n",
              "      <th>malaise</th>\n",
              "      <th>anorexia</th>\n",
              "      <th>liver_big</th>\n",
              "      <th>liver_firm</th>\n",
              "      <th>spleen_palpable</th>\n",
              "      <th>spiders</th>\n",
              "      <th>ascites</th>\n",
              "      <th>varices</th>\n",
              "      <th>bilirubin</th>\n",
              "      <th>alk_phosphate</th>\n",
              "      <th>sgot</th>\n",
              "      <th>albumin</th>\n",
              "      <th>protime</th>\n",
              "      <th>histology</th>\n",
              "      <th>class</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>30</td>\n",
              "      <td>male</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>1.0</td>\n",
              "      <td>85.0</td>\n",
              "      <td>18.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>False</td>\n",
              "      <td>live</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>50</td>\n",
              "      <td>female</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>0.9</td>\n",
              "      <td>135.0</td>\n",
              "      <td>42.0</td>\n",
              "      <td>3.5</td>\n",
              "      <td>NaN</td>\n",
              "      <td>False</td>\n",
              "      <td>live</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>78</td>\n",
              "      <td>female</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>0.7</td>\n",
              "      <td>96.0</td>\n",
              "      <td>32.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>False</td>\n",
              "      <td>live</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>31</td>\n",
              "      <td>female</td>\n",
              "      <td>NaN</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>0.7</td>\n",
              "      <td>46.0</td>\n",
              "      <td>52.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>80.0</td>\n",
              "      <td>False</td>\n",
              "      <td>live</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>34</td>\n",
              "      <td>female</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>1.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>200.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>False</td>\n",
              "      <td>live</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>34</td>\n",
              "      <td>female</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>0.9</td>\n",
              "      <td>95.0</td>\n",
              "      <td>28.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>75.0</td>\n",
              "      <td>False</td>\n",
              "      <td>live</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>51</td>\n",
              "      <td>female</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>False</td>\n",
              "      <td>die</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>23</td>\n",
              "      <td>female</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>1.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>False</td>\n",
              "      <td>live</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>39</td>\n",
              "      <td>female</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>0.7</td>\n",
              "      <td>NaN</td>\n",
              "      <td>48.0</td>\n",
              "      <td>4.4</td>\n",
              "      <td>NaN</td>\n",
              "      <td>False</td>\n",
              "      <td>live</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>30</td>\n",
              "      <td>female</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>1.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>120.0</td>\n",
              "      <td>3.9</td>\n",
              "      <td>NaN</td>\n",
              "      <td>False</td>\n",
              "      <td>live</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   age     sex steroid  antivirals fatigue malaise anorexia liver_big  \\\n",
              "0   30    male   False       False   False   False    False     False   \n",
              "1   50  female   False       False    True   False    False     False   \n",
              "2   78  female    True       False    True   False    False      True   \n",
              "3   31  female     NaN        True   False   False    False      True   \n",
              "4   34  female    True       False   False   False    False      True   \n",
              "5   34  female    True       False   False   False    False      True   \n",
              "6   51  female   False       False    True   False     True      True   \n",
              "7   23  female    True       False   False   False    False      True   \n",
              "8   39  female    True       False    True   False    False      True   \n",
              "9   30  female    True       False   False   False    False      True   \n",
              "\n",
              "  liver_firm spleen_palpable spiders ascites varices  bilirubin  \\\n",
              "0      False           False   False   False   False        1.0   \n",
              "1      False           False   False   False   False        0.9   \n",
              "2      False           False   False   False   False        0.7   \n",
              "3      False           False   False   False   False        0.7   \n",
              "4      False           False   False   False   False        1.0   \n",
              "5      False           False   False   False   False        0.9   \n",
              "6      False            True    True   False   False        NaN   \n",
              "7      False           False   False   False   False        1.0   \n",
              "8       True           False   False   False   False        0.7   \n",
              "9      False           False   False   False   False        1.0   \n",
              "\n",
              "   alk_phosphate   sgot  albumin  protime  histology class  \n",
              "0           85.0   18.0      4.0      NaN      False  live  \n",
              "1          135.0   42.0      3.5      NaN      False  live  \n",
              "2           96.0   32.0      4.0      NaN      False  live  \n",
              "3           46.0   52.0      4.0     80.0      False  live  \n",
              "4            NaN  200.0      4.0      NaN      False  live  \n",
              "5           95.0   28.0      4.0     75.0      False  live  \n",
              "6            NaN    NaN      NaN      NaN      False   die  \n",
              "7            NaN    NaN      NaN      NaN      False  live  \n",
              "8            NaN   48.0      4.4      NaN      False  live  \n",
              "9            NaN  120.0      3.9      NaN      False  live  "
            ]
          },
          "execution_count": 117,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "df.head(10)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "v6lh_G9keuC_"
      },
      "source": [
        "Now we can apply the `interpolate()` function to numeric columns, by setting also the limit direction to `forward`. This means that the linear interpolation is applied starting from the first row until the last one."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "metadata": {
        "id": "J34ucNNseuC_"
      },
      "outputs": [],
      "source": [
        "df[numeric_columns] = df[numeric_columns].interpolate(method ='linear', limit_direction ='forward')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "k7qyZda9euC_"
      },
      "source": [
        "For example, in line 6 the column `bilirubin`, which was NaN before the interpolation, now assumes the value 0.95, which is the interpolation between 0.90 (line 4) and 1.00 (line 6)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 17,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 427
        },
        "id": "xgVB656WeuDA",
        "outputId": "c7eac756-bb7b-43ff-fc20-239a667c5cad"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "   age     sex steroid  antivirals fatigue malaise anorexia liver_big  \\\n",
              "0   30    male   False       False   False   False    False     False   \n",
              "1   50  female   False       False    True   False    False     False   \n",
              "2   78  female    True       False    True   False    False      True   \n",
              "3   31  female     NaN        True   False   False    False      True   \n",
              "4   34  female    True       False   False   False    False      True   \n",
              "5   34  female    True       False   False   False    False      True   \n",
              "6   51  female   False       False    True   False     True      True   \n",
              "7   23  female    True       False   False   False    False      True   \n",
              "8   39  female    True       False    True   False    False      True   \n",
              "9   30  female    True       False   False   False    False      True   \n",
              "\n",
              "  liver_firm spleen_palpable spiders ascites varices  bilirubin  \\\n",
              "0      False           False   False   False   False       1.00   \n",
              "1      False           False   False   False   False       0.90   \n",
              "2      False           False   False   False   False       0.70   \n",
              "3      False           False   False   False   False       0.70   \n",
              "4      False           False   False   False   False       1.00   \n",
              "5      False           False   False   False   False       0.90   \n",
              "6      False            True    True   False   False       0.95   \n",
              "7      False           False   False   False   False       1.00   \n",
              "8       True           False   False   False   False       0.70   \n",
              "9      False           False   False   False   False       1.00   \n",
              "\n",
              "   alk_phosphate        sgot   albumin  protime  histology class  \n",
              "0           85.0   18.000000  4.000000      NaN      False  live  \n",
              "1          135.0   42.000000  3.500000      NaN      False  live  \n",
              "2           96.0   32.000000  4.000000      NaN      False  live  \n",
              "3           46.0   52.000000  4.000000     80.0      False  live  \n",
              "4           70.5  200.000000  4.000000     77.5      False  live  \n",
              "5           95.0   28.000000  4.000000     75.0      False  live  \n",
              "6           91.6   34.666667  4.133333     77.0      False   die  \n",
              "7           88.2   41.333333  4.266667     79.0      False  live  \n",
              "8           84.8   48.000000  4.400000     81.0      False  live  \n",
              "9           81.4  120.000000  3.900000     83.0      False  live  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-653d68da-cf6e-485d-9703-dd29a0a933e5\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>age</th>\n",
              "      <th>sex</th>\n",
              "      <th>steroid</th>\n",
              "      <th>antivirals</th>\n",
              "      <th>fatigue</th>\n",
              "      <th>malaise</th>\n",
              "      <th>anorexia</th>\n",
              "      <th>liver_big</th>\n",
              "      <th>liver_firm</th>\n",
              "      <th>spleen_palpable</th>\n",
              "      <th>spiders</th>\n",
              "      <th>ascites</th>\n",
              "      <th>varices</th>\n",
              "      <th>bilirubin</th>\n",
              "      <th>alk_phosphate</th>\n",
              "      <th>sgot</th>\n",
              "      <th>albumin</th>\n",
              "      <th>protime</th>\n",
              "      <th>histology</th>\n",
              "      <th>class</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>30</td>\n",
              "      <td>male</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>1.00</td>\n",
              "      <td>85.0</td>\n",
              "      <td>18.000000</td>\n",
              "      <td>4.000000</td>\n",
              "      <td>NaN</td>\n",
              "      <td>False</td>\n",
              "      <td>live</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>50</td>\n",
              "      <td>female</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>0.90</td>\n",
              "      <td>135.0</td>\n",
              "      <td>42.000000</td>\n",
              "      <td>3.500000</td>\n",
              "      <td>NaN</td>\n",
              "      <td>False</td>\n",
              "      <td>live</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>78</td>\n",
              "      <td>female</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>0.70</td>\n",
              "      <td>96.0</td>\n",
              "      <td>32.000000</td>\n",
              "      <td>4.000000</td>\n",
              "      <td>NaN</td>\n",
              "      <td>False</td>\n",
              "      <td>live</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>31</td>\n",
              "      <td>female</td>\n",
              "      <td>NaN</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>0.70</td>\n",
              "      <td>46.0</td>\n",
              "      <td>52.000000</td>\n",
              "      <td>4.000000</td>\n",
              "      <td>80.0</td>\n",
              "      <td>False</td>\n",
              "      <td>live</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>34</td>\n",
              "      <td>female</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>1.00</td>\n",
              "      <td>70.5</td>\n",
              "      <td>200.000000</td>\n",
              "      <td>4.000000</td>\n",
              "      <td>77.5</td>\n",
              "      <td>False</td>\n",
              "      <td>live</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>34</td>\n",
              "      <td>female</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>0.90</td>\n",
              "      <td>95.0</td>\n",
              "      <td>28.000000</td>\n",
              "      <td>4.000000</td>\n",
              "      <td>75.0</td>\n",
              "      <td>False</td>\n",
              "      <td>live</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>51</td>\n",
              "      <td>female</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>0.95</td>\n",
              "      <td>91.6</td>\n",
              "      <td>34.666667</td>\n",
              "      <td>4.133333</td>\n",
              "      <td>77.0</td>\n",
              "      <td>False</td>\n",
              "      <td>die</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>23</td>\n",
              "      <td>female</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>1.00</td>\n",
              "      <td>88.2</td>\n",
              "      <td>41.333333</td>\n",
              "      <td>4.266667</td>\n",
              "      <td>79.0</td>\n",
              "      <td>False</td>\n",
              "      <td>live</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>39</td>\n",
              "      <td>female</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>0.70</td>\n",
              "      <td>84.8</td>\n",
              "      <td>48.000000</td>\n",
              "      <td>4.400000</td>\n",
              "      <td>81.0</td>\n",
              "      <td>False</td>\n",
              "      <td>live</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>30</td>\n",
              "      <td>female</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>1.00</td>\n",
              "      <td>81.4</td>\n",
              "      <td>120.000000</td>\n",
              "      <td>3.900000</td>\n",
              "      <td>83.0</td>\n",
              "      <td>False</td>\n",
              "      <td>live</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-653d68da-cf6e-485d-9703-dd29a0a933e5')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-653d68da-cf6e-485d-9703-dd29a0a933e5 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-653d68da-cf6e-485d-9703-dd29a0a933e5');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 17
        }
      ],
      "source": [
        "df.head(10)"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Summary\n",
        "In this tutorial we have seen one of the aspects of data preprocessing, which is dealing with missing data. Missing data can alter the data analysis process, thus they must be managed.\n",
        "\n",
        "Three strategies can be used to deal with missing data:\n",
        "\n",
        "1. drop missing data: this can be done when the dataset has a small number of missing data\n",
        "2. replace missing data with other values, such as the mean or the most frequent value\n",
        "3. leave missing data as they are.\n",
        "\n",
        "If you would like to learn about the other aspects of data preprocessing, such as data standardization and data normalization, stay tuned…"
      ],
      "metadata": {
        "id": "e0Oc4g5VflTG"
      }
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.1"
    },
    "colab": {
      "provenance": [],
      "toc_visible": true,
      "include_colab_link": true
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}