world-data-visualization.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "b9d29384-3b4e-4ef8-87b2-24603b27818c",
   "metadata": {},
   "source": [
    "# This is a notebook\n",
    "\n",
    "A notebook is made up of cells. Those cells can either:\n",
    "* contain text and images (a markdown cell)\n",
    "* code that can be executred (code cell)\n",
    "\n",
    "You can select which type of cell a cell should be by:\n",
    "1. selecting a cell by clicking onto/into it\n",
    "2. using the drop drop down menu at the top of the notebook"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8d7488db-cbe6-4184-8995-aeca7b7e4fee",
   "metadata": {},
   "source": [
    "## Python uses libraries\n",
    "\n",
    "Most programming languages come with a standard set of features and functions. People can add onto to the standard library of functions using libraries or modules. Some common libraries for working with data are:\n",
    "* `numpy`\n",
    "* `pandas`\n",
    "* `matplotlib`\n",
    "* `plotly`\n",
    "\n",
    "These allow for statistical analysis, visualization, and more.\n",
    "\n",
    "These libraries are designed for folks in industry and in research to conduct high-level data science work. As such, they are not always the most user friendly for beginners. UC-Berkeley has created a library called `datascience` that attempts to simplify the syntax for writing code, and hides a lot of the technical bits from the user/student. We'll make use of this notebook throughout the week. If you're interested in learning more about how the library works, you can visit the website for the library at [https://www.data8.org/datascience/](https://www.data8.org/datascience/).\n",
    "\n",
    "The code cell below will load up the neccesary features and functions from the `datascience` library for the code found later on in the notebook to correctly execute. Run the cell by clicking into it, then either pressing the \"play\" button near the top of the notebook, or, pressing `shift+enter` on your keyboard."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0b7740ab-0617-45e6-b015-3b979d27d52f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from datascience import *\n",
    "import numpy as np\n",
    "\n",
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plots\n",
    "plots.style.use('fivethirtyeight')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "27e76d1e-7b9c-4a6e-bcc5-fead95ea8f0e",
   "metadata": {},
   "source": [
    "## Loading the data\n",
    "To load the data from a `.csv` file into a Table, you can use the `read_table` function as shown below. Run the code below to store the entire dataset into a Table named `helicopter`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2c63e34a-c532-42b2-bf69-d8d06242a78d",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "world_data = Table.read_table(\"data/world_data.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "42a6c281-cfc5-4ec8-9a29-26aa83590428",
   "metadata": {},
   "source": [
    "## Investigating the data\n",
    "\n",
    "You will likely want to learn a bit of summary information regarding your data. There are several commands to help you do this quickly.\n",
    "\n",
    "\n",
    "### Inspecting the data\n",
    "You can start by looking at the first 10 rows of the Table just by running a code well with the Table's name."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "222290e1-0256-4c2e-8c33-3bd68dace223",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "world_data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "de6b7f9e-9edd-4fbe-9548-0c4c07ea72d8",
   "metadata": {},
   "source": [
    "This allows you to see the variable names, and the types of data they contain.\n",
    "\n",
    "You can see additional rows by appending the `show()` function to the Table reference after a period `.` and specifying the number of rows to display between the parentheses. The code below will show 30 rows of the Table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "abf9ae34-c902-4b2f-97eb-c2cacddb5f6f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "world_data.show(30)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "38d8130a-e153-4b41-925d-9de06986498e",
   "metadata": {},
   "source": [
    "### Size of your data\n",
    "You can determine the dimensions of your data using the `.num_rows` and `.num_columns` commands."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9c42fb6e-a26c-4e67-9126-0b4c604b7269",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "world_data.num_rows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9a3d6787-eb10-428f-9d4b-87a342d03d44",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "world_data.num_columns"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c32ad6fe-733c-43fc-9e6c-6568cbc6e9f6",
   "metadata": {},
   "source": [
    "### Summary statistics\n",
    "\n",
    "You may wish to calculate some summary statistics on your data. You can use the `numpy` library which contains many statistical functions. First, select the function you wish to use, then, select column you wish to use in your computation using the `.column()` command. \n",
    "\n",
    "For example, if I wished to compute the mean value of the data in the column labeled `\"life_expectancy_years\"`, I would select the `np.nanmean` function (compute the mean ignoring missing, or nan, values) and then provide the `\"life_expectancy_years\"` column from the `world_data` Table as the input."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7c0850d4-ff64-4827-a4ee-5679a019c81f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "np.nanmean( world_data.column(\"life_expectancy_years\") )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d7f33fa2-68c2-4af0-befd-470497d10e7c",
   "metadata": {},
   "source": [
    "Common `numpy` statistics functions are:\n",
    "\n",
    "* **Arithmetic Mean**: `np.mean` / `np.nanmean`\n",
    "* **Median**: `np.median` / `np.nanmedian`\n",
    "* **Standard Deviation**: `np.std` / `np.nanstd`\n",
    "* **Variance**: `np.var` / `np.nanvar`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "381f9150-28dc-4687-9616-ae7a9473e263",
   "metadata": {},
   "source": [
    "### Grouping your data\n",
    "\n",
    "You'll often want to perform calculations on a particular subgroupd of your dataset. You can use the `.group` function to help you perform such a task.\n",
    "\n",
    "The `.group` function takes as its argument the label of the column that contains the categories. By default it returns a table of counts of rows in each category. `group` creates a distribution table that shows how the individuals are distributed among the categories found in the indicated column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "66be1a39-c959-41ad-8a9c-c1c6bf0f37b9",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "world_data.group(\"region\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "767d8f08-4a4c-4746-bd22-f1ea236915c8",
   "metadata": {},
   "source": [
    "You can optionally provide a second input to the `group` function that will apply a summary function to the remaining variables within each group. For example, specifying the `np.nanmean` function will apply `np.nanmean` to all other numerical columns in the dataset. If there is non-numerical data in those columns, the result will be blank."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "13e17ef9-d970-4c5b-895b-9a2079361b45",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "world_data.group(\"region\", np.nanmean)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fc8bf8d2-8256-4517-aaa0-61bf9f65627f",
   "metadata": {},
   "source": [
    "## Filtering the data\n",
    "\n",
    "You may what to only use part of your dataset at a time. You can use `where` functions to specify how to filter down to just the rows/observations you're interested in using. **Note:** This is not modifying the original table at all, it just creates a new table that contains the requested rows.\n",
    "\n",
    "The code below will only retain rows of `world_data` where the value in the \"year\" column is not equal to 2021."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "25c76470-014f-463b-a7d0-1f6311945a84",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "world_data.where('year', are.equal_to(2021))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "120aa76c-d90a-4a1a-8bba-341a0a8f2cee",
   "metadata": {},
   "source": [
    "You can chain multiple filters together to fine tune your selection process."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8b5f5c6b-91aa-4672-bc7b-536221ae0775",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "world_data.where('year', are.equal_to(2021)).where('region', are.equal_to(\"asia\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f258ab4-98d0-462b-84d0-bf45e0459147",
   "metadata": {},
   "source": [
    "Other common actions used in filtering are:\n",
    "\n",
    "* `are.not_equal_to(x)`\n",
    "* `are.above(x)`\n",
    "* `are.above_or_equal_to(x)`\n",
    "* `are.below(x)`\n",
    "* `are.below_or_equal_to(x)`\n",
    "* `are.between(x, y)`\n",
    "* `are.between_or_equal_to(x,y)`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f58f9d85-e016-423a-8d4c-71aabe4ccf21",
   "metadata": {},
   "source": [
    "If you want to save the result of a filter, you need to assign it a new name. It is a best practice not to overwrite any tables in a notebook, but instead create a new table with a new name to store any filtered or otherwise modified data that you intend to use again later in the notebook.\n",
    "\n",
    "The code below will create a new Table `world_data_2021` that only contains the observations that occurred in the year 2021."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8e915497-33c3-46a3-948a-7a4f12956a91",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "world_data_2021 = world_data.where('year', are.equal_to(2021))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b0e92448-f3db-4307-a065-3ea7828cbce0",
   "metadata": {},
   "source": [
    "## Visualizing the data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c839c027-7fce-4f25-b5ac-05f2c7bae7e0",
   "metadata": {},
   "source": [
    "You can easily create bar charts, scatter plots, line plots, and histograms depending on the data you are hoping to visualize."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d907ff0-18c2-4693-917f-c34576f5703b",
   "metadata": {},
   "source": [
    "### Bar charts\n",
    "\n",
    "To create a bar chart you need a Table that contains a frequency count of categorical varibles. We can use the `.group` function to create such a table, and then generate the bar chart using the `bar()` function. \n",
    "\n",
    "Recall that the `group` function creates a new two-column table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0276868d-2495-416f-836e-08163354f32d",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "world_data_2021.group(\"region\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4cf1376f-8caa-47ae-b61c-39cb5d88bb01",
   "metadata": {},
   "source": [
    "After group, you can chain the `.barh()` function to the resulting table, specifying the column that contains the categorical variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cabb35bf-7616-4c9b-a477-74e717beda15",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "world_data_2021.group(\"region\").barh(\"region\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "20ec4843-d723-4ce0-9563-685b51f2793c",
   "metadata": {},
   "source": [
    "**Note:** When creating frequency bar charts, its usually best to sort the data before creating the chart. The code below wil use the `.sort` function to sort by the frequency in descending order before creating the bar chart."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5398e7d3-ff4a-4500-828c-5e9918780d5d",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "world_data_2021.group(\"region\").sort(\"count\", descending=True).barh(\"region\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2e3d8458-c3cd-48b1-81b7-9cb084e3172c",
   "metadata": {},
   "source": [
    "### Scatter plot\n",
    "\n",
    "When investigating pairs of quantitative data, a scatter plot is a great tool. You can use `.scatter()` on a Table to create one quickly. Specify the labels of the columns that contain the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "58d1339c-a998-40a7-b606-914dc556b95b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "world_data_2021.scatter(\"fertility_cpw\", \"child_mortality_dpk\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a688bc2-35de-411e-a359-f5a18630522d",
   "metadata": {},
   "source": [
    "You may wish to see how your data differents based by category. You can assign each categorical group a different color on your scatterplot by including an optional `group` argument to the function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bf8b6e7a-8331-47d3-870a-84cea0a81aa3",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "world_data_2021.scatter(\"fertility_cpw\", \"child_mortality_dpk\", group=\"region\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "82e642a2-6d35-421d-8d37-b52eab8e248f",
   "metadata": {},
   "source": [
    "### Line plots\n",
    "\n",
    "For time-series data, a line plot helps to illustrate changes over time. Use the `.plot()` function to create a line plot.\n",
    "\n",
    "First, let's create a Table that has all the years of information just for the United States:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "774afcb8-1296-4cd1-91db-e5d0abc393b0",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "united_states = world_data.where(\"name\", are.equal_to(\"United States\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7aec9843-0d9a-466f-b2bf-457ffef859a4",
   "metadata": {},
   "source": [
    "Then, let's look at how child mortality has changed over time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c67a7620-d171-4ebb-8a09-876bd2609f49",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "united_states.plot(\"year\", \"child_mortality_dpk\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f994499-18ae-4983-9a05-53e85a71b02b",
   "metadata": {},
   "source": [
    "### Histograms\n",
    "\n",
    "To investigate a distribution of numerical values, use a histograms! The `.hist()` function can handle this quickly. Specify the numerical column you wish to create a histogram from."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "47634a95-924e-4bc7-a03c-80ba377c2b13",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "world_data_2021.hist(\"child_mortality_dpk\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1238799b-e5f9-4e0e-a455-e08271d246e6",
   "metadata": {},
   "source": [
    "You might wish to see how the distribution of the variable differs among groups. You can specify an optional input to the `.hist()` function that specifies which column contains your grouping variable. \n",
    "\n",
    "For example, the code below will group by \"region\" and then show the distribution of the \"life_expectancy_years\" variable for each group, with each histogram in a different color."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9600fe5e-2f57-443a-a943-a66e4fd2e893",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "world_data_2021.hist(\"life_expectancy_years\", group=\"region\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}