tutorials/labelling_numerical.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "source": [
    "# YData Quality - Labelling (Numerical) Tutorial\n",
    "Time-to-Value: 4 minutes\n",
    "\n",
    "This notebook provides a tutorial for the ydata_quality package funcionality for numerical labels.\n",
    "\n",
    "**Structure:**\n",
    "\n",
    "1. Load dataset\n",
    "2. Corrupt the dataset\n",
    "2. Instantiate the Data Quality engine\n",
    "3. Run the quality checks\n",
    "4. Assess the warnings\n",
    "5. (Extra) Detailed overview"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "source": [
    "import pandas as pd\n",
    "from ydata_quality.labelling import label_inspector_dispatch"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Load the example dataset\n",
    "We will use a transformed version of the \"Guerry\" dataset available from the statsmodels package.\n",
    "Records without label and single record labels are added to the dataset."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "source": [
    "df = pd.read_csv('../datasets/transformed/guerry_num_label.csv')\n",
    "df.head()"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>dept</th>\n",
       "      <th>Region</th>\n",
       "      <th>Department</th>\n",
       "      <th>Crime_pers</th>\n",
       "      <th>Crime_prop</th>\n",
       "      <th>Literacy</th>\n",
       "      <th>Donations</th>\n",
       "      <th>Infants</th>\n",
       "      <th>Suicides</th>\n",
       "      <th>MainCity</th>\n",
       "      <th>...</th>\n",
       "      <th>Crime_parents</th>\n",
       "      <th>Infanticide</th>\n",
       "      <th>Donation_clergy</th>\n",
       "      <th>Lottery</th>\n",
       "      <th>Desertion</th>\n",
       "      <th>Instruction</th>\n",
       "      <th>Prostitutes</th>\n",
       "      <th>Distance</th>\n",
       "      <th>Area</th>\n",
       "      <th>Pop1831</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>E</td>\n",
       "      <td>Ain</td>\n",
       "      <td>28870</td>\n",
       "      <td>15890</td>\n",
       "      <td>37</td>\n",
       "      <td>NaN</td>\n",
       "      <td>33120</td>\n",
       "      <td>35039</td>\n",
       "      <td>2:Med</td>\n",
       "      <td>...</td>\n",
       "      <td>71</td>\n",
       "      <td>60</td>\n",
       "      <td>69</td>\n",
       "      <td>41</td>\n",
       "      <td>55</td>\n",
       "      <td>46</td>\n",
       "      <td>13</td>\n",
       "      <td>218.372</td>\n",
       "      <td>5762</td>\n",
       "      <td>346.03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>N</td>\n",
       "      <td>Aisne</td>\n",
       "      <td>26226</td>\n",
       "      <td>5521</td>\n",
       "      <td>51</td>\n",
       "      <td>50000.0</td>\n",
       "      <td>14572</td>\n",
       "      <td>12831</td>\n",
       "      <td>2:Med</td>\n",
       "      <td>...</td>\n",
       "      <td>4</td>\n",
       "      <td>82</td>\n",
       "      <td>36</td>\n",
       "      <td>38</td>\n",
       "      <td>82</td>\n",
       "      <td>24</td>\n",
       "      <td>327</td>\n",
       "      <td>65.945</td>\n",
       "      <td>7369</td>\n",
       "      <td>513.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>C</td>\n",
       "      <td>Allier</td>\n",
       "      <td>26747</td>\n",
       "      <td>7925</td>\n",
       "      <td>13</td>\n",
       "      <td>10973.0</td>\n",
       "      <td>17044</td>\n",
       "      <td>114121</td>\n",
       "      <td>2:Med</td>\n",
       "      <td>...</td>\n",
       "      <td>46</td>\n",
       "      <td>42</td>\n",
       "      <td>76</td>\n",
       "      <td>66</td>\n",
       "      <td>16</td>\n",
       "      <td>85</td>\n",
       "      <td>34</td>\n",
       "      <td>161.927</td>\n",
       "      <td>7340</td>\n",
       "      <td>298.26</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>E</td>\n",
       "      <td>Basses-Alpes</td>\n",
       "      <td>12935</td>\n",
       "      <td>7289</td>\n",
       "      <td>46</td>\n",
       "      <td>2733.0</td>\n",
       "      <td>23018</td>\n",
       "      <td>14238</td>\n",
       "      <td>1:Sm</td>\n",
       "      <td>...</td>\n",
       "      <td>70</td>\n",
       "      <td>12</td>\n",
       "      <td>37</td>\n",
       "      <td>80</td>\n",
       "      <td>32</td>\n",
       "      <td>29</td>\n",
       "      <td>2</td>\n",
       "      <td>351.399</td>\n",
       "      <td>6925</td>\n",
       "      <td>155.90</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>E</td>\n",
       "      <td>Hautes-Alpes</td>\n",
       "      <td>17488</td>\n",
       "      <td>8174</td>\n",
       "      <td>69</td>\n",
       "      <td>6962.0</td>\n",
       "      <td>23076</td>\n",
       "      <td>16171</td>\n",
       "      <td>1:Sm</td>\n",
       "      <td>...</td>\n",
       "      <td>22</td>\n",
       "      <td>23</td>\n",
       "      <td>64</td>\n",
       "      <td>79</td>\n",
       "      <td>35</td>\n",
       "      <td>7</td>\n",
       "      <td>1</td>\n",
       "      <td>320.280</td>\n",
       "      <td>5549</td>\n",
       "      <td>129.10</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 23 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   dept Region    Department  Crime_pers  Crime_prop  Literacy  Donations  \\\n",
       "0     1      E           Ain       28870       15890        37        NaN   \n",
       "1     2      N         Aisne       26226        5521        51    50000.0   \n",
       "2     3      C        Allier       26747        7925        13    10973.0   \n",
       "3     4      E  Basses-Alpes       12935        7289        46     2733.0   \n",
       "4     5      E  Hautes-Alpes       17488        8174        69     6962.0   \n",
       "\n",
       "   Infants  Suicides MainCity  ...  Crime_parents  Infanticide  \\\n",
       "0    33120     35039    2:Med  ...             71           60   \n",
       "1    14572     12831    2:Med  ...              4           82   \n",
       "2    17044    114121    2:Med  ...             46           42   \n",
       "3    23018     14238     1:Sm  ...             70           12   \n",
       "4    23076     16171     1:Sm  ...             22           23   \n",
       "\n",
       "   Donation_clergy  Lottery  Desertion  Instruction  Prostitutes  Distance  \\\n",
       "0               69       41         55           46           13   218.372   \n",
       "1               36       38         82           24          327    65.945   \n",
       "2               76       66         16           85           34   161.927   \n",
       "3               37       80         32           29            2   351.399   \n",
       "4               64       79         35            7            1   320.280   \n",
       "\n",
       "   Area  Pop1831  \n",
       "0  5762   346.03  \n",
       "1  7369   513.00  \n",
       "2  7340   298.26  \n",
       "3  6925   155.90  \n",
       "4  5549   129.10  \n",
       "\n",
       "[5 rows x 23 columns]"
      ]
     },
     "metadata": {},
     "execution_count": 2
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Create the engine\n",
    "Each engine contains the checks and tests for each suite. To create a Label Inspector, you provide:\n",
    "- df: target DataFrame, for which we will run the test suite\n",
    "- label: name of the column to be used as label, in this case it points to a numerical label!"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "source": [
    "li = label_inspector_dispatch(df=df, label='Donations', random_state=24)"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Full Evaluation\n",
    "The easiest way to assess the data quality analysis is to run `.evaluate()` which returns the results of the tests, stores the warnings and prints a summary of the data quality warnings found during the analysis."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "source": [
    "results = li.evaluate()"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "WARNING | It was not possible to normalize the label values. See the data quality warning message for additional context.\n",
      "\u001b[1mWarnings:\u001b[0m\n",
      "\tTOTAL: 3 warning(s)\n",
      "\t\u001b[1m\u001b[38;5;209mPriority 1\u001b[0m: 2 warning(s)\n",
      "\t\u001b[1m\u001b[38;5;11mPriority 2\u001b[0m: 1 warning(s)\n",
      "\n",
      "\u001b[38;5;209m\u001b[1mPriority 1\u001b[0m - \u001b[1mheavy impact expected\u001b[0m:\n",
      "\t\u001b[38;5;209m*\u001b[0m \u001b[1m[LABELS\u001b[0m - \u001b[4mMISSING LABELS]\u001b[0m Found 1 instances with missing labels.\n",
      "\t\u001b[38;5;209m*\u001b[0m \u001b[1m[LABELS\u001b[0m - \u001b[4mTEST NORMALITY]\u001b[0m The label distribution failed to pass a normality test as-is and following a battery of transforms. It is possible that the data originates from an exotic distribution, there is heavy outlier presence or it is multimodal. Addressing this issue might prove critical for regressor performance.\n",
      "\u001b[38;5;11m\u001b[1mPriority 2\u001b[0m - \u001b[1musage allowed, limited human intelligibility\u001b[0m:\n",
      "\t\u001b[38;5;11m*\u001b[0m \u001b[1m[LABELS\u001b[0m - \u001b[4mOUTLIER DETECTION]\u001b[0m Found 3 potential outliers across the full dataset. A distance bigger than 3.0 standard deviations of intra-cluster distances to the respective centroids was used to define the potential outliers.\n",
      "\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Full Test Suite\n",
    "In this section, you will find a detailed overview of the available tests in the labelling module of ydata_quality."
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Missing Labels\n",
    "\n",
    "Return records with a missing label."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "source": [
    "li.missing_labels()"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>dept</th>\n",
       "      <th>Region</th>\n",
       "      <th>Department</th>\n",
       "      <th>Crime_pers</th>\n",
       "      <th>Crime_prop</th>\n",
       "      <th>Literacy</th>\n",
       "      <th>Donations</th>\n",
       "      <th>Infants</th>\n",
       "      <th>Suicides</th>\n",
       "      <th>MainCity</th>\n",
       "      <th>...</th>\n",
       "      <th>Crime_parents</th>\n",
       "      <th>Infanticide</th>\n",
       "      <th>Donation_clergy</th>\n",
       "      <th>Lottery</th>\n",
       "      <th>Desertion</th>\n",
       "      <th>Instruction</th>\n",
       "      <th>Prostitutes</th>\n",
       "      <th>Distance</th>\n",
       "      <th>Area</th>\n",
       "      <th>Pop1831</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>E</td>\n",
       "      <td>Ain</td>\n",
       "      <td>28870</td>\n",
       "      <td>15890</td>\n",
       "      <td>37</td>\n",
       "      <td>NaN</td>\n",
       "      <td>33120</td>\n",
       "      <td>35039</td>\n",
       "      <td>2:Med</td>\n",
       "      <td>...</td>\n",
       "      <td>71</td>\n",
       "      <td>60</td>\n",
       "      <td>69</td>\n",
       "      <td>41</td>\n",
       "      <td>55</td>\n",
       "      <td>46</td>\n",
       "      <td>13</td>\n",
       "      <td>218.372</td>\n",
       "      <td>5762</td>\n",
       "      <td>346.03</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1 rows × 23 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   dept Region Department  Crime_pers  Crime_prop  Literacy  Donations  \\\n",
       "0     1      E        Ain       28870       15890        37        NaN   \n",
       "\n",
       "   Infants  Suicides MainCity  ...  Crime_parents  Infanticide  \\\n",
       "0    33120     35039    2:Med  ...             71           60   \n",
       "\n",
       "   Donation_clergy  Lottery  Desertion  Instruction  Prostitutes  Distance  \\\n",
       "0               69       41         55           46           13   218.372   \n",
       "\n",
       "   Area  Pop1831  \n",
       "0  5762   346.03  \n",
       "\n",
       "[1 rows x 23 columns]"
      ]
     },
     "metadata": {},
     "execution_count": 5
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Outlier Detection\n",
    "\n",
    "Get a dictionary of all potential outliers obtained over the full dataset (see example 1) or across clusters on the label feature (see example 2 below). Outliers are defined based on a threshold of standard deviations of distances to the median of the dataset or of the cluster (see example 2)."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "source": [
    "# Example 1\n",
    "li.outlier_detection(th=3)  # The top 3 valued labels were flagged as potential outliers"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "{'full_dataset':     dept Region Department  Crime_pers  Crime_prop  Literacy  Donations  \\\n",
       " 1      2      N      Aisne       26226        5521        51    50000.0   \n",
       " 12    14      N   Calvados       17577        4500        52    27830.0   \n",
       " 85   200    NaN      Corse        2199        4589        49    37015.0   \n",
       " \n",
       "     Infants  Suicides MainCity  ...  Crime_parents  Infanticide  \\\n",
       " 1     14572     12831    2:Med  ...              4           82   \n",
       " 12     8983     31807    2:Med  ...             57           56   \n",
       " 85    24743     37016    2:Med  ...             81            2   \n",
       " \n",
       "     Donation_clergy  Lottery  Desertion  Instruction  Prostitutes  Distance  \\\n",
       " 1                36       38         82           24          327    65.945   \n",
       " 12               11       13         12           22          194   117.487   \n",
       " 85               84       83          9           25            1   539.213   \n",
       " \n",
       "     Area  Pop1831  \n",
       " 1   7369   513.00  \n",
       " 12  5548   494.70  \n",
       " 85  8680   195.41  \n",
       " \n",
       " [3 rows x 23 columns]}"
      ]
     },
     "metadata": {},
     "execution_count": 6
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "source": [
    "# Example 2\n",
    "li.outlier_detection(th=3, use_clusters=True)  # Only the highest donation was flagged as a potential outlier"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "{}"
      ]
     },
     "metadata": {},
     "execution_count": 7
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Normality Test\n",
    "\n",
    "Test the labels for normality (test if the data might originate from a normal distribution with a pre-specified confidence).\n",
    "If the test fails apply a battery of transforms to the label and repeat the test after each transform.\n",
    "Raise a low priority warning if a transform is required, raise a high priority warning if the transforms did not result in positive normality test."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "source": [
    "li.test_normality()"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "WARNING | It was not possible to normalize the label values. See the data quality warning message for additional context.\n"
     ]
    }
   ],
   "metadata": {}
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "e255f3ac955330aecee05fff6b7b15b68f4bd4cf0e9481cf0822c8a2e5228d43"
  },
  "kernelspec": {
   "name": "python3",
   "display_name": "Python 3.8.11 64-bit ('DQ': conda)"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.11"
  },
  "metadata": {
   "interpreter": {
    "hash": "cdc2bce73c2a9ac283f602628cabf735dbe06c4ee87a7849fc5f3d1177c8f304"
   }
  },
  "orig_nbformat": 2
 },
 "nbformat": 4,
 "nbformat_minor": 2
}