machine learning/data_manipulation.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "data_summary.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# Data manipulation cheatsheet\n",
        "## Introduction\n",
        "\n",
        "this is the companion colab code for my tutorial in medium about Data manipulation cheatsheet in Python, please read the article for better understanding of the code. In the article I discuss about the theory behinds, trick, use cases and so on. Here I am providing just generic information about the algorithms and the code. \n",
        "this is a colab notebook, you can test in colab (all the code has been tested and you do not need to set your machine)\n",
        "\n",
        "[Medium article about Data manipulation cheatsheet](https://)\n",
        "\n",
        "note: the articles will be publish soon\n",
        "\n",
        "for other stories check my medium profile:\n",
        "[Salvatore Raieli](https://salvatore-raieli.medium.com/)"
      ],
      "metadata": {
        "id": "dYFlojZqu1Ul"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Observations and features\n",
        "\n",
        "we use the titanic dataset, let explore the this dataset and do the first data manipulation"
      ],
      "metadata": {
        "id": "MPMEtuSVvRxk"
      }
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2kvelx7MqsMQ"
      },
      "source": [
        "import pandas as pd\n",
        "import numpy as np\n",
        "df = pd.read_csv('https://raw.githubusercontent.com/pandas-dev/pandas/master/doc/data/titanic.csv')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "7v-UEVO_u_Ph",
        "outputId": "51178268-569e-42af-ab21-83ae0fa94954"
      },
      "source": [
        "#finding categorial and numerical variables\n",
        "numerical = df.applymap(np.isreal).all()\n",
        "#numerical is true, categorical is false\n",
        "#or more elegant\n",
        "df.info()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "<class 'pandas.core.frame.DataFrame'>\n",
            "RangeIndex: 891 entries, 0 to 890\n",
            "Data columns (total 12 columns):\n",
            " #   Column       Non-Null Count  Dtype  \n",
            "---  ------       --------------  -----  \n",
            " 0   PassengerId  891 non-null    int64  \n",
            " 1   Survived     891 non-null    int64  \n",
            " 2   Pclass       891 non-null    int64  \n",
            " 3   Name         891 non-null    object \n",
            " 4   Sex          891 non-null    object \n",
            " 5   Age          714 non-null    float64\n",
            " 6   SibSp        891 non-null    int64  \n",
            " 7   Parch        891 non-null    int64  \n",
            " 8   Ticket       891 non-null    object \n",
            " 9   Fare         891 non-null    float64\n",
            " 10  Cabin        204 non-null    object \n",
            " 11  Embarked     889 non-null    object \n",
            "dtypes: float64(2), int64(5), object(5)\n",
            "memory usage: 83.7+ KB\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 284
        },
        "id": "IfT-f9jnwsiQ",
        "outputId": "4e702f33-74af-428d-bc27-beae8b88ac6a"
      },
      "source": [
        "#summary statistics for the numerical colums\n",
        "df.describe()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>PassengerId</th>\n",
              "      <th>Survived</th>\n",
              "      <th>Pclass</th>\n",
              "      <th>Age</th>\n",
              "      <th>SibSp</th>\n",
              "      <th>Parch</th>\n",
              "      <th>Fare</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>count</th>\n",
              "      <td>891.000000</td>\n",
              "      <td>891.000000</td>\n",
              "      <td>891.000000</td>\n",
              "      <td>714.000000</td>\n",
              "      <td>891.000000</td>\n",
              "      <td>891.000000</td>\n",
              "      <td>891.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>mean</th>\n",
              "      <td>446.000000</td>\n",
              "      <td>0.383838</td>\n",
              "      <td>2.308642</td>\n",
              "      <td>29.699118</td>\n",
              "      <td>0.523008</td>\n",
              "      <td>0.381594</td>\n",
              "      <td>32.204208</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>std</th>\n",
              "      <td>257.353842</td>\n",
              "      <td>0.486592</td>\n",
              "      <td>0.836071</td>\n",
              "      <td>14.526497</td>\n",
              "      <td>1.102743</td>\n",
              "      <td>0.806057</td>\n",
              "      <td>49.693429</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>min</th>\n",
              "      <td>1.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>0.420000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>25%</th>\n",
              "      <td>223.500000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>2.000000</td>\n",
              "      <td>20.125000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>7.910400</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>50%</th>\n",
              "      <td>446.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>3.000000</td>\n",
              "      <td>28.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>14.454200</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>75%</th>\n",
              "      <td>668.500000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>3.000000</td>\n",
              "      <td>38.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>31.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>max</th>\n",
              "      <td>891.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>3.000000</td>\n",
              "      <td>80.000000</td>\n",
              "      <td>8.000000</td>\n",
              "      <td>6.000000</td>\n",
              "      <td>512.329200</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "       PassengerId    Survived      Pclass  ...       SibSp       Parch        Fare\n",
              "count   891.000000  891.000000  891.000000  ...  891.000000  891.000000  891.000000\n",
              "mean    446.000000    0.383838    2.308642  ...    0.523008    0.381594   32.204208\n",
              "std     257.353842    0.486592    0.836071  ...    1.102743    0.806057   49.693429\n",
              "min       1.000000    0.000000    1.000000  ...    0.000000    0.000000    0.000000\n",
              "25%     223.500000    0.000000    2.000000  ...    0.000000    0.000000    7.910400\n",
              "50%     446.000000    0.000000    3.000000  ...    0.000000    0.000000   14.454200\n",
              "75%     668.500000    1.000000    3.000000  ...    1.000000    0.000000   31.000000\n",
              "max     891.000000    1.000000    3.000000  ...    8.000000    6.000000  512.329200\n",
              "\n",
              "[8 rows x 7 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 30
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "19v-MW1CvKqN",
        "outputId": "8c95f886-b083-4422-c605-4d1a8e1d0e1f"
      },
      "source": [
        "#obtaining the count for a categorical variable\n",
        "df.Sex.value_counts()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "male      577\n",
              "female    314\n",
              "Name: Sex, dtype: int64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 21
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Dataset charachteristics and issues\n",
        "\n",
        "we will use pandas functions and more to find the charachteristics of our dataset (dimensionality, distribution, resolution). \n",
        "\n",
        "Moreover, we will check the presence of different issues and how to solve them (like missing values, duplicate data, inconsistent data)"
      ],
      "metadata": {
        "id": "Q76X_7ZKvqQ0"
      }
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "CudfrAmlsDvR",
        "outputId": "8cab5b55-1962-48a2-b3c8-6c0ca03a19ee"
      },
      "source": [
        "#find list column with missing values\n",
        "na_col = df.columns[df.isnull().any()].tolist()\n",
        "na_col"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['Age', 'Cabin', 'Embarked']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Wyzgdts0tUtS",
        "outputId": "e01820f5-2e86-48d5-90ec-761e5b48e6bb"
      },
      "source": [
        "#selecting the rows with missing values\n",
        "na_row = df[df.isnull().any(axis = 1)]\n",
        "na_row = na_row.index.to_list() #getting a list\n",
        "df.isnull().any(axis = 1).sum() #number of na row present"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "708"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 14
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "mYb4j27hxr-I",
        "outputId": "fe956c0f-9f3e-4535-b129-e46cdc908cab"
      },
      "source": [
        "#any remove a row/col if has any missing value, all instead if there are all missing values\n",
        "df_noNa_row = df.dropna(axis = 0, how = \"any\") #remove rows with na\n",
        "df_noNa_col = df.dropna(axis = 1, how = \"any\") #remove col with na\n",
        "df.shape, df_noNa_row.shape, df_noNa_col.shape"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "((891, 12), (183, 12), (891, 9))"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 34
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "yXHBV_Yyy4Qf",
        "outputId": "a6f82035-5766-49dd-c918-d32a9ceebe06"
      },
      "source": [
        "#remove rows with NA only in a subset of column\n",
        "df1=df.dropna(subset=['Age'])\n",
        "df2 = df.dropna(subset=['Cabin', 'Age'])\n",
        "df1.shape, df2.shape"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "((714, 12), (185, 12))"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 38
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "-UBRVTPFzUho",
        "outputId": "3c5d9a96-65cc-4301-88a0-753ff04433fa"
      },
      "source": [
        "age_fill = df[[\"Age\"]].fillna(0) #careful, it return a df of the column Age\n",
        "df1 = df.fillna(method ='pad') #substitute the NA with the previous value\n",
        "df1 = df.fillna(method ='bfill') #substitute the NA with the next value\n",
        "df1 = df.fillna(0) #this is substitute zero in each columns\n",
        "df.Age, df1.Age"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(0      22.0\n",
              " 1      38.0\n",
              " 2      26.0\n",
              " 3      35.0\n",
              " 4      35.0\n",
              "        ... \n",
              " 886    27.0\n",
              " 887    19.0\n",
              " 888     NaN\n",
              " 889    26.0\n",
              " 890    32.0\n",
              " Name: Age, Length: 891, dtype: float64, 0      22.0\n",
              " 1      38.0\n",
              " 2      26.0\n",
              " 3      35.0\n",
              " 4      35.0\n",
              "        ... \n",
              " 886    27.0\n",
              " 887    19.0\n",
              " 888     0.0\n",
              " 889    26.0\n",
              " 890    32.0\n",
              " Name: Age, Length: 891, dtype: float64)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 56
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 402
        },
        "id": "XTFNdKfsHm7y",
        "outputId": "b5faaf8b-49fb-47b6-daed-c3a3ea91e743"
      },
      "source": [
        "age_inter = df[[\"Age\"]].interpolate(method ='linear', limit_direction ='forward')\n",
        "df1 = df.loc[:, [\"Age\"]]\n",
        "df1[\"interpolated\"] = age_inter.Age\n",
        "df1"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Age</th>\n",
              "      <th>interpolated</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>22.0</td>\n",
              "      <td>22.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>38.0</td>\n",
              "      <td>38.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>26.0</td>\n",
              "      <td>26.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>35.0</td>\n",
              "      <td>35.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>35.0</td>\n",
              "      <td>35.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>886</th>\n",
              "      <td>27.0</td>\n",
              "      <td>27.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>887</th>\n",
              "      <td>19.0</td>\n",
              "      <td>19.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>888</th>\n",
              "      <td>NaN</td>\n",
              "      <td>22.5</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>889</th>\n",
              "      <td>26.0</td>\n",
              "      <td>26.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>890</th>\n",
              "      <td>32.0</td>\n",
              "      <td>32.0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>891 rows × 2 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "      Age  interpolated\n",
              "0    22.0          22.0\n",
              "1    38.0          38.0\n",
              "2    26.0          26.0\n",
              "3    35.0          35.0\n",
              "4    35.0          35.0\n",
              "..    ...           ...\n",
              "886  27.0          27.0\n",
              "887  19.0          19.0\n",
              "888   NaN          22.5\n",
              "889  26.0          26.0\n",
              "890  32.0          32.0\n",
              "\n",
              "[891 rows x 2 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 95
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Data pre-processing\n",
        "\n",
        "some common pre-processing techniques like aggregation, sampling (different type of sampling as simple, stratified), dimensionality reduction, discretization (binarization, equal width, equal frequency, k-means binarization). These all functions can be handy in many contexts.\n",
        "\n",
        "I will also discuss about variable transformation, techniques as standardization and so on"
      ],
      "metadata": {
        "id": "IRR9YQ2RwMLG"
      }
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Bfeb8VqDKlXM",
        "outputId": "d09131f0-4495-46f3-fb94-d0f81091b078"
      },
      "source": [
        "#you can use this method to remove extreme and inconsistent values\n",
        "df1 = df[(df['Age'] > 1) & (df['Age'] < 80)]\n",
        "df2 = df[(df[\"Sex\"] == \"female\")]\n",
        "adult = df['Age'] > 18 #return a boolean list\n",
        "df3 = df[adult] \n",
        "df.shape, df1.shape, df2.shape, df3.shape"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "((891, 12), (699, 12), (314, 12), (575, 12))"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 102
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 282
        },
        "id": "s1w8jRh4PpbM",
        "outputId": "da86f469-5501-4b78-d98a-8a4a1e308e4a"
      },
      "source": [
        "#To investigate distribution\n",
        "df[\"Age\"].hist()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<matplotlib.axes._subplots.AxesSubplot at 0x7fc587ccc650>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 110
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD4CAYAAAAXUaZHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAUPUlEQVR4nO3df5Dcd33f8ee7duIoOkayY7Ojym7PTIwZ8AWBbhwySZk7nARhGBwyjGONh1rBrWAGUtK5mVROOoGUYcZtEbQdWhIROyY/qjPFGDwyCXFdX2g6BXICxZIxBhtEYtWRwDZyzmhczrz7x36vLOc73+1+96v96qPnY2bn9vv57vf7fem+X71u77vf3YvMRJJUln8w6gCSpOGz3CWpQJa7JBXIcpekAlnuklSgc0cdAODCCy/M8fHxvpd7+umn2bhx4/AD1WSu/rU1m7n609Zc0N5sdXIdPHjw25l50YozM3Pkt+3bt+cg7rvvvoGWa5q5+tfWbObqT1tzZbY3W51cwHyu0quelpGkAlnuklQgy12SCmS5S1KBLHdJKpDlLkkFstwlqUCWuyQVaM1yj4hbI+JERBzpGbs9Ig5Vt6MRcagaH4+IUz3zfrfJ8JKkla3n4wduAz4E/OHSQGb+ytL9iNgLnOx5/COZuW1YAfVc43vuHmi5mYlFdg247JKjN7++1vKSTo81yz0zPxsR4yvNi4gArgVeM9xYkqQ6ItfxZ/aqcj+QmVcsG3818IHMnOx53APAV4GngH+dmf9zlXXuBnYDdDqd7bOzs32HX1hYYGxsrO/lmtZ0rsPHTq79oBV0NsDxU/W2PbF1U70VrOJs3ZeDMlf/2pqtTq7p6emDS/27XN1PhdwJ7O+Zfgz4R5n5eERsBz4ZES/LzKeWL5iZ+4B9AJOTkzk1NdX3xufm5hhkuaY1nWvQUyszE4vsPVxvlx+9fqrW8qs5W/floMzVv7ZmayrXwFfLRMS5wC8Dty+NZeYzmfl4df8g8Ajw4rohJUn9qXMp5M8DX8nMR5cGIuKiiDinuv8i4DLg6/UiSpL6tZ5LIfcD/xu4PCIejYgbq1nX8cOnZABeDdxfXRr5ceDtmfnEMANLkta2nqtldq4yvmuFsTuAO+rHkiTV4TtUJalArfgbqjpzDPoGqrWs9QYr3zwl9cdn7pJUIMtdkgpkuUtSgSx3SSqQ5S5JBbLcJalAlrskFchyl6QCWe6SVCDLXZIKZLlLUoEsd0kqkOUuSQWy3CWpQJa7JBXIcpekAlnuklQgy12SCrRmuUfErRFxIiKO9Iy9JyKORcSh6nZ1z7ybIuLhiHgoIl7bVHBJ0urW88z9NmDHCuMfzMxt1e3TABHxUuA64GXVMv8lIs4ZVlhJ0vqsWe6Z+VngiXWu7xpgNjOfycxvAA8DV9bIJ0kaQGTm2g+KGAcOZOYV1fR7gF3AU8A8MJOZT0bEh4DPZeYfV4+7BfjTzPz4CuvcDewG6HQ622dnZ/sOv7CwwNjYWN/LNa3pXIePnRxouc4GOH5qyGGGZK1sE1s3nb4wPc7WY2xQbc0F7c1WJ9f09PTBzJxcad65A+b5MPBeIKuve4G39rOCzNwH7AOYnJzMqampvkPMzc0xyHJNazrXrj13D7TczMQiew8PusubtVa2o9dPnb4wPc7WY2xQbc0F7c3WVK6BrpbJzOOZ+Wxmfh/4CD849XIMuKTnoRdXY5Kk02igco+ILT2TbwKWrqS5C7guIs6LiEuBy4Av1IsoSerXmr+jR8R+YAq4MCIeBd4NTEXENrqnZY4CbwPIzAci4mPAl4FF4B2Z+Wwz0SVJq1mz3DNz5wrDtzzP498HvK9OKElSPb5DVZIKZLlLUoEsd0kqkOUuSQWy3CWpQJa7JBXIcpekAlnuklQgy12SCmS5S1KBLHdJKpDlLkkFstwlqUCWuyQVyHKXpAJZ7pJUIMtdkgpkuUtSgSx3SSqQ5S5JBVqz3CPi1og4ERFHesb+fUR8JSLuj4g7I2JzNT4eEaci4lB1+90mw0uSVraeZ+63ATuWjd0DXJGZPwV8FbipZ94jmbmtur19ODElSf1Ys9wz87PAE8vG/jwzF6vJzwEXN5BNkjSgYZxzfyvwpz3Tl0bElyLiLyLinwxh/ZKkPkVmrv2giHHgQGZesWz8t4BJ4JczMyPiPGAsMx+PiO3AJ4GXZeZTK6xzN7AboNPpbJ+dne07/MLCAmNjY30v17Smcx0+dnKg5Tob4PipIYcZkrWyTWzddPrC9Dhbj7FBtTUXtDdbnVzT09MHM3NypXnnDhooInYBbwCuyuonRGY+AzxT3T8YEY8ALwbmly+fmfuAfQCTk5M5NTXVd4a5uTkGWa5pTefatefugZabmVhk7+GBd3mj1sp29Pqp0xemx9l6jA2qrbmgvdmayjXQaZmI2AH8BvDGzPxuz/hFEXFOdf9FwGXA14cRVJK0fms+jYuI/cAUcGFEPAq8m+7VMecB90QEwOeqK2NeDfybiPge8H3g7Zn5xIorliQ1Zs1yz8ydKwzfsspj7wDuqBtKklSP71CVpAJZ7pJUIMtdkgrUzuvipGXGB7z8s66ZiUWmRrJlqR6fuUtSgSx3SSqQ5S5JBbLcJalAlrskFchyl6QCWe6SVCDLXZIKZLlLUoEsd0kqkOUuSQWy3CWpQJa7JBXIcpekAlnuklQgy12SCmS5S1KB1lXuEXFrRJyIiCM9YxdExD0R8bXq6/nVeETEf4qIhyPi/oh4ZVPhJUkrW+8z99uAHcvG9gD3ZuZlwL3VNMDrgMuq227gw/VjSpL6sa5yz8zPAk8sG74G+Gh1/6PAL/WM/2F2fQ7YHBFbhhFWkrQ+kZnre2DEOHAgM6+opr+TmZur+wE8mZmbI+IAcHNm/mU1717gX2Xm/LL17ab7zJ5Op7N9dna27/ALCwuMjY31vVzTms51+NjJgZbrbIDjp4YcZkjamq2zAV54waZRx3iOs/XYr6Ot2erkmp6ePpiZkyvNO7dWqkpmZkSs76fED5bZB+wDmJyczKmpqb63Ozc3xyDLNa3pXLv23D3QcjMTi+w9PJRdPnRtzTYzsci1Z+ExNqi25oL2ZmsqV52rZY4vnW6pvp6oxo8Bl/Q87uJqTJJ0mtQp97uAG6r7NwCf6hn/p9VVM68CTmbmYzW2I0nq07p+D46I/cAUcGFEPAq8G7gZ+FhE3Ah8E7i2evingauBh4HvAr865MySpDWsq9wzc+cqs65a4bEJvKNOKElSPb5DVZIKZLlLUoEsd0kqkOUuSQWy3CWpQJa7JBXIcpekAlnuklQgy12SCmS5S1KBLHdJKpDlLkkFstwlqUCWuyQVyHKXpAJZ7pJUIMtdkgpkuUtSgSx3SSqQ5S5JBVrXH8heSURcDtzeM/Qi4LeBzcA/B75Vjf9mZn564ISSpL4NXO6Z+RCwDSAizgGOAXcCvwp8MDPfP5SEkqS+Deu0zFXAI5n5zSGtT5JUQ2Rm/ZVE3Ap8MTM/FBHvAXYBTwHzwExmPrnCMruB3QCdTmf77Oxs39tdWFhgbGysRvJmNJ3r8LGTAy3X2QDHTw05zJC0NVtnA7zwgk2jjvEcZ+uxX0dbs9XJNT09fTAzJ1eaV7vcI+JHgf8DvCwzj0dEB/g2kMB7gS2Z+dbnW8fk5GTOz8/3ve25uTmmpqb6D92wpnON77l7oOVmJhbZe3jgM3GNamu2mYlFfu36a0Yd4znO1mO/jrZmq5MrIlYt92Gclnkd3WftxwEy83hmPpuZ3wc+Alw5hG1IkvowjHLfCexfmoiILT3z3gQcGcI2JEl9qPV7cERsBH4BeFvP8L+LiG10T8scXTZPknQa1Cr3zHwa+IllY2+plUiSVFv7XsGSWmbQF7DrOnrz60eyXZXBjx+QpAJZ7pJUIMtdkgpkuUtSgSx3SSqQ5S5JBbLcJalAlrskFchyl6QCWe6SVCDLXZIKZLlLUoEsd0kqkOUuSQWy3CWpQJa7JBXIcpekAlnuklSgIv7Mnn8GTZJ+WO1yj4ijwN8DzwKLmTkZERcAtwPjwFHg2sx8su62JEnrM6zTMtOZuS0zJ6vpPcC9mXkZcG81LUk6TZo6534N8NHq/keBX2poO5KkFURm1ltBxDeAJ4EEfi8z90XEdzJzczU/gCeXpnuW2w3sBuh0OttnZ2f73vbCwgJjY2McPnay1r9hUBNbN604vpSrKYP+ezsb4PipIYcZkrZmG2Wu1Y4vaP4YG1Rbc0F7s9XJNT09fbDnjMkPGUa5b83MYxHxQuAe4NeAu3rLPCKezMzzV1vH5ORkzs/P973tubk5pqamWveC6lKupgz6752ZWGTv4Xa+ht7WbKPM9Xwv2Dd9jA2qrbmgvdnq5IqIVcu99mmZzDxWfT0B3AlcCRyPiC3VxrcAJ+puR5K0frXKPSI2RsQLlu4DvwgcAe4CbqgedgPwqTrbkST1p+7vmx3gzu5pdc4F/mtm/llE/BXwsYi4EfgmcG3N7UiS+lCr3DPz68DLVxh/HLiqzrrPBKud+56ZWGTXiF4HkCQo5B2qUome74Xzpp9A+O7rM5+fLSNJBbLcJalAlrskFchyl6QCWe6SVCDLXZIKZLlLUoEsd0kqkOUuSQWy3CWpQJa7JBXIcpekAlnuklQgy12SCmS5S1KBLHdJKpDlLkkFstwlqUCWuyQVaOByj4hLIuK+iPhyRDwQEe+qxt8TEcci4lB1u3p4cSVJ61HnD2QvAjOZ+cWIeAFwMCLuqeZ9MDPfXz+eJGkQA5d7Zj4GPFbd//uIeBDYOqxgkqTBDeWce0SMA68APl8NvTMi7o+IWyPi/GFsQ5K0fpGZ9VYQMQb8BfC+zPxERHSAbwMJvBfYkplvXWG53cBugE6ns312drbvbS8sLDA2NsbhYyfr/BOGrrMBjp8adYrnamsuaG+2szXXxNZNAy239H+yjdqarU6u6enpg5k5udK8WuUeET8CHAA+k5kfWGH+OHAgM694vvVMTk7m/Px839ufm5tjamqK8T13971sk2YmFtl7uM7LGc1oay5obzZz9adurqM3v36IaX7YUl+0TZ1cEbFqude5WiaAW4AHe4s9Irb0POxNwJFBtyFJGkydH/0/C7wFOBwRh6qx3wR2RsQ2uqdljgJvq5VQktS3OlfL/CUQK8z69OBxJEnD4DtUJalAlrskFchyl6QCWe6SVCDLXZIKZLlLUoEsd0kqkOUuSQWy3CWpQJa7JBXIcpekAlnuklQgy12SCtS+T/uXdNZq8g/vzEwssmuV9Tf5R0JGxWfuklQgy12SCmS5S1KBLHdJKpDlLkkFstwlqUCWuyQVqLHr3CNiB/AfgXOA38/Mm5valiTV0eT19Wu5bcfGRtbbyDP3iDgH+M/A64CXAjsj4qVNbEuS9FxNnZa5Eng4M7+emf8XmAWuaWhbkqRlIjOHv9KINwM7MvOfVdNvAX46M9/Z85jdwO5q8nLgoQE2dSHw7Zpxm2Cu/rU1m7n609Zc0N5sdXL948y8aKUZI/tsmczcB+yrs46ImM/MySFFGhpz9a+t2czVn7bmgvZmaypXU6dljgGX9ExfXI1Jkk6Dpsr9r4DLIuLSiPhR4Drgroa2JUlappHTMpm5GBHvBD5D91LIWzPzgQY2Veu0ToPM1b+2ZjNXf9qaC9qbrZFcjbygKkkaLd+hKkkFstwlqUBnZLlHxI6IeCgiHo6IPSPOcmtEnIiIIz1jF0TEPRHxterr+SPIdUlE3BcRX46IByLiXW3IFhE/FhFfiIi/rnL9TjV+aUR8vtqnt1cvxJ92EXFORHwpIg60LNfRiDgcEYciYr4aa8NxtjkiPh4RX4mIByPiZ0adKyIur75PS7enIuLXR52ryvYvq+P+SETsr/4/NHKMnXHl3sKPNrgN2LFsbA9wb2ZeBtxbTZ9ui8BMZr4UeBXwjur7NOpszwCvycyXA9uAHRHxKuDfAh/MzJ8EngRuPM25lrwLeLBnui25AKYzc1vPNdGj3pfQ/fyoP8vMlwAvp/u9G2muzHyo+j5tA7YD3wXuHHWuiNgK/AtgMjOvoHuxyXU0dYxl5hl1A34G+EzP9E3ATSPONA4c6Zl+CNhS3d8CPNSC79ungF9oUzbgx4EvAj9N9x165660j09jnovp/qd/DXAAiDbkqrZ9FLhw2dhI9yWwCfgG1YUZbcm1LMsvAv+rDbmArcDfAhfQvVLxAPDapo6xM+6ZOz/4Bi15tBprk05mPlbd/zugM8owETEOvAL4PC3IVp36OAScAO4BHgG+k5mL1UNGtU//A/AbwPer6Z9oSS6ABP48Ig5WH90Bo9+XlwLfAv6gOpX1+xGxsQW5el0H7K/ujzRXZh4D3g/8DfAYcBI4SEPH2JlY7meU7P44Htn1phExBtwB/HpmPtU7b1TZMvPZ7P7KfDHdD5l7yenOsFxEvAE4kZkHR51lFT+Xma+kezryHRHx6t6ZI9qX5wKvBD6cma8AnmbZqY5RHv/Vues3Av9t+bxR5KrO8V9D94fiPwQ28txTukNzJpb7mfDRBscjYgtA9fXEKEJExI/QLfY/ycxPtCkbQGZ+B7iP7q+imyNi6U11o9inPwu8MSKO0v0U09fQPZ886lzA/3/WR2aeoHv++EpGvy8fBR7NzM9X0x+nW/ajzrXkdcAXM/N4NT3qXD8PfCMzv5WZ3wM+Qfe4a+QYOxPL/Uz4aIO7gBuq+zfQPd99WkVEALcAD2bmB9qSLSIuiojN1f0NdF8HeJBuyb95VLky86bMvDgzx+keU/8jM68fdS6AiNgYES9Yuk/3PPIRRrwvM/PvgL+NiMuroauAL486V4+d/OCUDIw+198Ar4qIH6/+fy59v5o5xkb1QkfNFyauBr5K91ztb404y36658++R/eZzI10z9XeC3wN+O/ABSPI9XN0f+28HzhU3a4edTbgp4AvVbmOAL9djb8I+ALwMN1fo88b4T6dAg60JVeV4a+r2wNLx/yo92WVYRswX+3PTwLntyTXRuBxYFPPWBty/Q7wlerY/yPgvKaOMT9+QJIKdCaelpEkrcFyl6QCWe6SVCDLXZIKZLlLUoEsd0kqkOUuSQX6f6EjqSprEB7OAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 484
        },
        "id": "9J5vqfbRHUxM",
        "outputId": "5841a5ca-e9fd-492c-ad78-9502afb7db69"
      },
      "source": [
        "df.head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>PassengerId</th>\n",
              "      <th>Survived</th>\n",
              "      <th>Pclass</th>\n",
              "      <th>Name</th>\n",
              "      <th>Sex</th>\n",
              "      <th>Age</th>\n",
              "      <th>SibSp</th>\n",
              "      <th>Parch</th>\n",
              "      <th>Ticket</th>\n",
              "      <th>Fare</th>\n",
              "      <th>Cabin</th>\n",
              "      <th>Embarked</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>3</td>\n",
              "      <td>Braund, Mr. Owen Harris</td>\n",
              "      <td>male</td>\n",
              "      <td>22.0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>A/5 21171</td>\n",
              "      <td>7.2500</td>\n",
              "      <td>NaN</td>\n",
              "      <td>S</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>Cumings, Mrs. John Bradley (Florence Briggs Th...</td>\n",
              "      <td>female</td>\n",
              "      <td>38.0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>PC 17599</td>\n",
              "      <td>71.2833</td>\n",
              "      <td>C85</td>\n",
              "      <td>C</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>3</td>\n",
              "      <td>1</td>\n",
              "      <td>3</td>\n",
              "      <td>Heikkinen, Miss. Laina</td>\n",
              "      <td>female</td>\n",
              "      <td>26.0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>STON/O2. 3101282</td>\n",
              "      <td>7.9250</td>\n",
              "      <td>NaN</td>\n",
              "      <td>S</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>4</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>Futrelle, Mrs. Jacques Heath (Lily May Peel)</td>\n",
              "      <td>female</td>\n",
              "      <td>35.0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>113803</td>\n",
              "      <td>53.1000</td>\n",
              "      <td>C123</td>\n",
              "      <td>S</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>5</td>\n",
              "      <td>0</td>\n",
              "      <td>3</td>\n",
              "      <td>Allen, Mr. William Henry</td>\n",
              "      <td>male</td>\n",
              "      <td>35.0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>373450</td>\n",
              "      <td>8.0500</td>\n",
              "      <td>NaN</td>\n",
              "      <td>S</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   PassengerId  Survived  Pclass  ...     Fare Cabin  Embarked\n",
              "0            1         0       3  ...   7.2500   NaN         S\n",
              "1            2         1       1  ...  71.2833   C85         C\n",
              "2            3         1       3  ...   7.9250   NaN         S\n",
              "3            4         1       1  ...  53.1000  C123         S\n",
              "4            5         0       3  ...   8.0500   NaN         S\n",
              "\n",
              "[5 rows x 12 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 97
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 136
        },
        "id": "g2SJ9t6FU8FL",
        "outputId": "9f1a00c7-c48c-4666-b640-243d60781302"
      },
      "source": [
        "#selecting and grouping\n",
        "df1 = df[[\"Age\", \"Fare\", \"Sex\"]].groupby([\"Sex\"]).mean()\n",
        "df1"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Age</th>\n",
              "      <th>Fare</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Sex</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>female</th>\n",
              "      <td>27.915709</td>\n",
              "      <td>44.479818</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>male</th>\n",
              "      <td>30.726645</td>\n",
              "      <td>25.523893</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "              Age       Fare\n",
              "Sex                         \n",
              "female  27.915709  44.479818\n",
              "male    30.726645  25.523893"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 112
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "0CeUwczGV63v",
        "outputId": "71345091-0628-4809-bf6a-1979dbe155cf"
      },
      "source": [
        "df[\"Embarked\"].value_counts()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "S    644\n",
              "C    168\n",
              "Q     77\n",
              "Name: Embarked, dtype: int64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 115
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "DxK7YIazXdqC",
        "outputId": "79e8cf9c-cc45-4556-e17b-4be959ef8851"
      },
      "source": [
        "#aggregate in high category\n",
        "replacement = { \"Embarked\" : {\"S\" :  \"England\", \"C\" : \"France\", \"Q\" : \"England\"}}\n",
        "df1 = df.replace(replacement)\n",
        "df1[\"Embarked\"].value_counts()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "England    721\n",
              "France     168\n",
              "Name: Embarked, dtype: int64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 119
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Z1awMjBtY9lx",
        "outputId": "6f40a30b-3980-4128-8af9-cf54bc05f2a2"
      },
      "source": [
        "#simple sampling\n",
        "df1 = df.sample(frac = 0.3) #without replacement\n",
        "df1 = df.sample(frac = 0.3, replace= True) #without replacement\n",
        "df1.shape"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(267, 12)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 121
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "pepb0fqwaA8P",
        "outputId": "d03984cb-e034-4b7b-fe11-8bfee20fea46"
      },
      "source": [
        "#stratified sampling\n",
        "from sklearn.model_selection import train_test_split\n",
        "df_strat, _ = train_test_split(df, test_size=0.7, stratify=df[ 'Sex'])\n",
        "df[\"Embarked\"] = df.Embarked.fillna(\"Q\") #otherwise give error with Na\n",
        "df_strat, _ = train_test_split(df, test_size=0.7, stratify=df[[ 'Sex', \"Embarked\"]])\n",
        "df_strat.shape"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(267, 12)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 133
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "D7bvVp2Sb3PX",
        "outputId": "2c60d5a9-4fcc-46b0-fa12-43117ec0af90"
      },
      "source": [
        "#dimentionality reduction with PCA\n",
        "from sklearn import datasets\n",
        "from sklearn.decomposition import PCA\n",
        "\n",
        "iris = datasets.load_iris()\n",
        "X = iris.data\n",
        "y = iris.target\n",
        "pca = PCA(n_components=4) #select number components\n",
        "pca.fit(X)\n",
        "#notice only the first component show higher variance\n",
        "pca.explained_variance_ratio_ "
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([0.92461872, 0.05306648, 0.01710261, 0.00521218])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 136
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "jXJD3FVgcP0e",
        "outputId": "63474d74-fb9d-43be-a522-21073fff9c20"
      },
      "source": [
        "#dimensionality reduction with single value decomposition\n",
        "from sklearn.decomposition import TruncatedSVD\n",
        "svd = TruncatedSVD(n_components=3, n_iter=7, random_state=42)\n",
        "svd.fit(X)\n",
        "svd.explained_variance_ratio_"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([0.52875361, 0.44845576, 0.01757678])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 139
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "7muaoshKct6H",
        "outputId": "c22b801b-9174-4f27-b7a7-673f3d3200e4"
      },
      "source": [
        "#drop irrelevant feature\n",
        "df1 = df.drop(\"PassengerId\", 1)\n",
        "df1 = df.drop([\"PassengerId\", \"Name\"], 1)\n",
        "df.shape, df1.shape"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "((891, 12), (891, 10))"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 143
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "idFd_qxadEk9",
        "outputId": "0dd5d419-01ce-412b-96a1-98e8f4a5c485"
      },
      "source": [
        "#calculate correlation\n",
        "df['Age'].corr(df['Fare'])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.0960666917690389"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 144
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SHNPn3Ptd0c-"
      },
      "source": [
        "#create a new feature\n",
        "df1 = df.loc[:, [\"Age\", \"Fare\"]]\n",
        "df1[\"Ratio\"] = df1[\"Fare\"] / df1[\"Age\"]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 402
        },
        "id": "2qEdyYjzc3kP",
        "outputId": "a79a2013-258d-44eb-fe3b-375e2a8106f3"
      },
      "source": [
        "#binarization\n",
        "pd.get_dummies(y)\n",
        "df1 = df.loc[:, [\"Age\", \"Fare\", \"Sex\"]]\n",
        "df1 = pd.get_dummies(df1, columns = [\"Sex\"])\n",
        "df1"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Age</th>\n",
              "      <th>Fare</th>\n",
              "      <th>Sex_female</th>\n",
              "      <th>Sex_male</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>22.0</td>\n",
              "      <td>7.2500</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>38.0</td>\n",
              "      <td>71.2833</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>26.0</td>\n",
              "      <td>7.9250</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>35.0</td>\n",
              "      <td>53.1000</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>35.0</td>\n",
              "      <td>8.0500</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>886</th>\n",
              "      <td>27.0</td>\n",
              "      <td>13.0000</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>887</th>\n",
              "      <td>19.0</td>\n",
              "      <td>30.0000</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>888</th>\n",
              "      <td>NaN</td>\n",
              "      <td>23.4500</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>889</th>\n",
              "      <td>26.0</td>\n",
              "      <td>30.0000</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>890</th>\n",
              "      <td>32.0</td>\n",
              "      <td>7.7500</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>891 rows × 4 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "      Age     Fare  Sex_female  Sex_male\n",
              "0    22.0   7.2500           0         1\n",
              "1    38.0  71.2833           1         0\n",
              "2    26.0   7.9250           1         0\n",
              "3    35.0  53.1000           1         0\n",
              "4    35.0   8.0500           0         1\n",
              "..    ...      ...         ...       ...\n",
              "886  27.0  13.0000           0         1\n",
              "887  19.0  30.0000           1         0\n",
              "888   NaN  23.4500           1         0\n",
              "889  26.0  30.0000           0         1\n",
              "890  32.0   7.7500           0         1\n",
              "\n",
              "[891 rows x 4 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 161
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "E9qUNowZkfQY",
        "outputId": "32273d6e-a930-4a40-d3a4-37ca264efbbf"
      },
      "source": [
        "#equal width discretization\n",
        "df1 = df.loc[:, [\"Survived\", \"Age\"]]\n",
        "df1[\"age_bin\"] = pd.cut(df1[\"Age\"],5)\n",
        "df1[\"age_bin\"].value_counts()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(16.336, 32.252]    346\n",
              "(32.252, 48.168]    188\n",
              "(0.34, 16.336]      100\n",
              "(48.168, 64.084]     69\n",
              "(64.084, 80.0]       11\n",
              "Name: age_bin, dtype: int64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 168
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "zgdR_sRAlZKm",
        "outputId": "10cf5068-4485-4381-9529-e77300e20bd0"
      },
      "source": [
        "#equal frequency discretization\n",
        "df1 = df.loc[:, [\"Survived\", \"Age\"]]\n",
        "df1[\"age_bin\"] = pd.qcut(df1[\"Age\"],5)\n",
        "df1[\"age_bin\"].value_counts()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(0.419, 19.0]    164\n",
              "(31.8, 41.0]     144\n",
              "(41.0, 80.0]     142\n",
              "(19.0, 25.0]     137\n",
              "(25.0, 31.8]     127\n",
              "Name: age_bin, dtype: int64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 167
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "xg7TwRNYlquV",
        "outputId": "6f4ebd7a-e548-4127-978c-c1059dcf22f2"
      },
      "source": [
        "#equal frequency discretization with labels\n",
        "bin_label = [\"young\", \"middle_age\", \"old\"]\n",
        "df1[\"age_bin\"] = pd.qcut(df1[\"Age\"], 3, labels = bin_label)\n",
        "df1[\"age_bin\"].value_counts()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "young         246\n",
              "old           236\n",
              "middle_age    232\n",
              "Name: age_bin, dtype: int64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 169
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ptWpPuC3mPnW",
        "outputId": "13bcad82-0ef1-492e-a404-28b07d812b3b"
      },
      "source": [
        "#k-means binarization, notice scikit learn want a 2d array\n",
        "from sklearn.preprocessing import KBinsDiscretizer\n",
        "discretizer = KBinsDiscretizer(n_bins=5, encode='ordinal', strategy='kmeans')\n",
        "X = df.loc[:, [\"Age\", \"Fare\"]]\n",
        "X[\"Age\"] = X[[\"Age\"]].interpolate(method ='linear', limit_direction ='forward')\n",
        "discretizer.fit_transform(X)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/sklearn/preprocessing/_discretization.py:184: ConvergenceWarning: Number of distinct clusters (4) found smaller than n_clusters (5). Possibly due to duplicate points in X.\n",
            "  centers = km.fit(column[:, None]).cluster_centers_[:, 0]\n",
            "/usr/local/lib/python3.7/dist-packages/sklearn/preprocessing/_discretization.py:197: UserWarning: Bins whose width are too small (i.e., <= 1e-8) in feature 1 are removed. Consider decreasing the number of bins.\n",
            "  'decreasing the number of bins.' % jj)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[1., 0.],\n",
              "       [2., 1.],\n",
              "       [1., 0.],\n",
              "       ...,\n",
              "       [1., 0.],\n",
              "       [1., 0.],\n",
              "       [2., 0.]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 182
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "tYEUN5tbmRoA",
        "outputId": "27da8861-ae2e-4976-b6fc-078ebc5d09f0"
      },
      "source": [
        "#alternative method\n",
        "#!pip3 install feature_engine \n",
        "from feature_engine.discretisation import EqualWidthDiscretiser, EqualFrequencyDiscretiser\n",
        "df1 = df.loc[:, [\"Survived\", \"Age\"]]\n",
        "discretizer = EqualWidthDiscretiser(bins=3, variables = [\"Age\"])\n",
        "discretizer = EqualFrequencyDiscretiser(q=3, variables = [\"Age\"])\n",
        "df1[\"Age\"] = df1[[\"Age\"]].interpolate(method ='linear', limit_direction ='forward')\n",
        "discr = discretizer.fit_transform(df1)\n",
        "discr.Age.value_counts()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "1    447\n",
              "0    390\n",
              "2     54\n",
              "Name: Age, dtype: int64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 186
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "qao8lzQqtSLK",
        "outputId": "211b458b-caf1-40cd-9696-1c3b0325ff81"
      },
      "source": [
        "#standardization\n",
        "from scipy.stats import zscore\n",
        "df1 = df.loc[:, [\"Survived\", \"Age\"]]\n",
        "df1[\"Age\"] = zscore(df1[\"Age\"], nan_policy = \"omit\") #otherwise all NaN\n",
        "df1[\"Age\"] "
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0     -0.530377\n",
              "1      0.571831\n",
              "2     -0.254825\n",
              "3      0.365167\n",
              "4      0.365167\n",
              "         ...   \n",
              "886   -0.185937\n",
              "887   -0.737041\n",
              "888         NaN\n",
              "889   -0.254825\n",
              "890    0.158503\n",
              "Name: Age, Length: 891, dtype: float64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 192
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "eItgliDZuCMp",
        "outputId": "e01d278d-446f-44e0-eb9f-0ad6bdf69621"
      },
      "source": [
        "#scaling\n",
        "df1 = df.loc[:, [\"Survived\", \"Age\"]]\n",
        "df1[\"Age\"]  = (df1.Age - df1.Age.min())/(df1.Age.max()-df1.Age.min())\n",
        "df1[\"Age\"] "
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0      0.271174\n",
              "1      0.472229\n",
              "2      0.321438\n",
              "3      0.434531\n",
              "4      0.434531\n",
              "         ...   \n",
              "886    0.334004\n",
              "887    0.233476\n",
              "888         NaN\n",
              "889    0.321438\n",
              "890    0.396833\n",
              "Name: Age, Length: 891, dtype: float64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 193
        }
      ]
    }
  ]
}