4_Missing.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/theheking/intro-to-python/blob/gh-pages/4_Missing.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Dealing with missing data\n",
        "\n",
        "\n",
        "Most real-world data has lots of missing values, and this can be an issue; in this notebook, we'll go through how to deal with missing data.\n",
        "\n",
        "Objectives\n",
        "1. What NaN values are\n",
        "2. How they might be represented\n",
        "3. What this means for your work\n",
        "4. How to replace NaN values, if desired\n",
        "5. How to use to_csv to write manipulated data to a file.\n",
        "\n"
      ],
      "metadata": {
        "id": "UIOSx0n2APCV"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ATmp4cjkAIgB"
      },
      "outputs": [],
      "source": [
        "!pip install pandas matplotlib # install the packages directly into the notebook\n"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Make sure the libraries are loaded and we have the data we need.\n"
      ],
      "metadata": {
        "id": "6oivEcqFAb6D"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Download a file using python\n",
        "import urllib.request # this is the library we need \n",
        "\n",
        "url = 'https://raw.githubusercontent.com/theheking/intro-to-python/gh-pages/docs/patient_data.csv'\n",
        "#retrieve the file\n",
        "\n",
        "urllib.request.urlretrieve(url, 'patient_data.csv')\n",
        "\n",
        "#import pandas as a package\n",
        "import pandas as pd \n",
        "\n",
        "#read in the patient data \n",
        "patient_df=pd.read_csv('patient_data.csv', sep=',')"
      ],
      "metadata": {
        "id": "Ygfrp8jeAe8w"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Finding Missing Values\n",
        "Let's identify all locations in the data that have null (missing or NaN) data values. \n",
        "\n",
        "\n",
        "We can use the `isnull` method to do this. The `isnull` method will compare each cell with a null value. If an element has a null value, it will be assigned a value of `True` in the output object."
      ],
      "metadata": {
        "id": "FQl1xotNA93R"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# check what is null using isnull function \n"
      ],
      "metadata": {
        "id": "-Lf-YQcvBDYO"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## How to select rows with missing data\n",
        "\n",
        "To select the rows where there are null values, we can use the mask as an index to subset our data as follows:\n",
        "\n"
      ],
      "metadata": {
        "id": "F38CWPMPBIjv"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#  select only the rows with NaN values, using the 'any(axis=1)' method"
      ],
      "metadata": {
        "id": "L-Xpy3LcBHE8"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Explanation\n",
        "Notice that we have 444 observations/rows that contain one or more missing values. \n",
        "\n",
        "That's roughly **37%** of data contains missing values.\n",
        "\n",
        "We have used `[]` conversion to select a subset of data.\n",
        "\n",
        "More information about slicing and indexing can be found out here.\n",
        "\n",
        "`(axis=1)` is a numpy convention to specify columns.\n",
        "\n",
        "\n",
        "\n",
        "Note that the weight column of our DataFrame contains many null or NaN values. Next, we will explore ways of dealing with this.\n",
        "\n",
        "If we look at the weight column in the patient data, we notice that there are NaN (Not a Number) values. NaN values are undefined values that are not represented mathematically. \n",
        "\n",
        "Pandas, for example, will read an empty cell in a CSV or Excel sheet as a NaN. However, NaNs have some desirable properties: if we were to average the weight column without replacing our NaNs, Python would know to skip over those cells."
      ],
      "metadata": {
        "id": "ckl-RYpQBQzu"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "## Dealing with missing values.\n",
        "\n",
        "Discuss: \n",
        "\n",
        "1. What are some reasons why there might be missing data?\n",
        "2. How would you deal with missing values?\n",
        "3. Is it OK to ignore missing values when calculating the mean?\n",
        "4. What effect do missing values have when you multiply 2 columns (either test this out or think about what would happen)?"
      ],
      "metadata": {
        "id": "_nt-a7JkBg1V"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "## how many missing values are there in weight column"
      ],
      "metadata": {
        "id": "risIAelFBXVd"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# what are the number of rows have weight values len(patient_df[patient_df.weight> 0])\n"
      ],
      "metadata": {
        "id": "q1gh_ieaB5CL"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "We can replace all NaN values with zeroes using the .fillna() method (after making a copy of the data so we don't lose our work): "
      ],
      "metadata": {
        "id": "lLU4lmY_CDcn"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# create a new DataFrame using copy function\n",
        "\n",
        "# fill all NaN values of the weight comun with fillna(0)\n",
        "\n",
        "#print the new mean of weight\n"
      ],
      "metadata": {
        "id": "QYgCq9jhCBpw"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Writing Out Data to CSV\n",
        "\n",
        "Great, you've filled out all your data, but now you want to share it with your collaborators. To do that, you can just write to a CSV as usual."
      ],
      "metadata": {
        "id": "ldhMeqrKCljK"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Write DataFrame to CSV use index=False"
      ],
      "metadata": {
        "id": "ADJ1_NWYCxc5"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n"
      ],
      "metadata": {
        "id": "kMy7wWRlC6hO"
      }
    }
  ]
}