Ex.Data analysis and visualisation-solution.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "<a href=\"https://colab.research.google.com/github/ML-Challenge/week2-data-analysis/blob/master/Ex.Data%20analysis%20and%20visualisation-solution.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Download lesson datasets\n",
    "# Required if you're using Google Colab\n",
    "!wget \"https://github.com/ML-Challenge/week2-data-analysis/raw/master/datasets.zip\"\n",
    "!unzip -o datasets.zip"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {
    "dc": {
     "key": "3"
    },
    "deletable": false,
    "editable": false,
    "run_control": {
     "frozen": true
    },
    "tags": [
     "context"
    ]
   },
   "outputs": [],
   "source": [
    "## 1. The Nobel Prizes\n",
    "<p>The Nobel Prize is perhaps the world's most well known scientific award. Except for the honor, prestige and substantial prize money the recipient also gets a gold medal showing Alfred Nobel (1833 - 1896) who established the prize. Every year it's given to scientists and scholars in the categories chemistry, literature, physics, physiology or medicine, economics, and peace. </p>\n",
    "<p>The Nobel Foundation has made a dataset available of all prize winners from the start of the prize, in 1901, to 2016. Let's load it in and take a look.</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Task 1:\n",
    "\n",
    "* Import the required libraries\n",
    "* Read the dataset from `data/nobel.csv`\n",
    "* Show the first six entries of `nobel`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Loading in required libraries\n",
    "import pandas as pd\n",
    "import seaborn as sns\n",
    "import numpy as np\n",
    "\n",
    "# Reading in the Nobel Prize data\n",
    "nobel = pd.read_csv('data/nobel.csv')\n",
    "\n",
    "# Taking a look at the first several winners\n",
    "nobel.head(6)"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {
    "dc": {
     "key": "10"
    },
    "deletable": false,
    "editable": false,
    "run_control": {
     "frozen": true
    },
    "tags": [
     "context"
    ]
   },
   "outputs": [],
   "source": [
    "## 2. So, who gets the Nobel Prize?\n",
    "<p>Just looking at the first couple of prize winners, or Nobel laureates as they are also called, we already see a celebrity: Wilhelm Conrad Röntgen, who discovered X-rays. And actually, we see that all of the winners in 1901 were European men. But that was back in 1901, looking at all winners in the dataset, from 1901 to 2016, which sex and which country is the most commonly represented? </p>\n",
    "<p>(For <em>country</em>, we will use the <code>birth_country</code> of the winner, as the <code>organization_country</code> is <code>NaN</code> for all shared Nobel Prizes.)</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Task 2:\n",
    "Count up the Nobel Prizes. \n",
    "* Count the number of rows/prized. Use the `display()` function to display the result\n",
    "* Count and display the number of prizes for each sex using the `value_counts()` method.\n",
    "* Count the number of prizes for each `birth_country` using `value_counts()` and show the top 10. **Do not use display()**\n",
    "\n",
    "Why `display()` over `print()`? Try them both out for yourself."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Display the number of (possibly shared) Nobel Prizes handed\n",
    "# out between 1901 and 2016\n",
    "display(len(nobel))\n",
    "\n",
    "# Display the number of prizes won by male and female recipients.\n",
    "display(nobel['sex'].value_counts())\n",
    "\n",
    "# Display the number of prizes won by the top 10 nationalities.\n",
    "nobel['birth_country'].value_counts().head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {
    "dc": {
     "key": "17"
    },
    "deletable": false,
    "editable": false,
    "run_control": {
     "frozen": true
    },
    "tags": [
     "context"
    ]
   },
   "outputs": [],
   "source": [
    "## 3. USA dominance\n",
    "<p>Not so surprising perhaps: the most common Nobel laureate between 1901 and 2016 was a man born in the United States of America. But in 1901 all the winners were European. When did the USA start to dominate the Nobel Prize charts?</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Task 3:\n",
    "Create a DataFrame with two columns: decade and proportion of USA-born Nobel Prize winners that decade.\n",
    "* Add a `usa_born_winner` column to `nobel`, where the value is True when `birth_country` is `\"United States of America\"`.\n",
    "* Add a `decade` column to `nobel` for the decade each prize was awarded. Here, `np.floor()` will come in handy. Ensure the decade column is of type `int64`.\n",
    "* Use `groupby` to group by `decade`, setting `as_index=False`. Then isolate the `usa_born_winner` column and take the `mean()`. Assign the resulting DataFrame to `prop_usa_winners`.\n",
    "* Display `prop_usa_winners`.\n",
    "\n",
    "For the `decade column`, 1953 should become 1950, for example. Calculating this column is a bit tricky, but try to see if you can solve it using the np.floor function.\n",
    "\n",
    "By setting `as_index=False`, you make sure that both the grouping variable and the calculated mean are included in the resulting DataFrame. You can find the documentation for `groupby` [here](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.groupby.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Calculating the proportion of USA born winners per decade\n",
    "nobel['usa_born_winner'] = nobel['birth_country'] == 'United States of America'\n",
    "nobel['decade'] = (np.floor(nobel['year']/10)*10).astype('int64')\n",
    "prop_usa_winners = nobel.groupby('decade', as_index=False)['usa_born_winner'].mean()\n",
    "\n",
    "# Display the proportions of USA born winners per decade\n",
    "prop_usa_winners"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {
    "dc": {
     "key": "24"
    },
    "deletable": false,
    "editable": false,
    "run_control": {
     "frozen": true
    },
    "tags": [
     "context"
    ]
   },
   "outputs": [],
   "source": [
    "## 4. USA dominance, visualized\n",
    "<p>A table is OK, but to <em>see</em> when the USA started to dominate the Nobel charts we need a plot!</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Task 4:\n",
    "Plot the proportion of USA born winners per decade\n",
    "\n",
    "* Use seaborn to plot `prop_usa_winners` with `decade` on the x-axis and `usa_born_winner` on the y-axis as an `sns.lineplot`. Asign the plot to `ax`\n",
    "* Fix the y-scale so that the is shows percentages using `PercentFormatter`\n",
    "\n",
    "See [here](https://stackoverflow.com/questions/31357611/format-y-axis-as-percent/36319915#36319915) for a Stack Overflow answer on how `PercentFormatter` works and [here](https://seaborn.pydata.org/generated/seaborn.lineplot.html) for the documentation of `lineplot`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Setting the plotting theme\n",
    "sns.set()\n",
    "# and setting the size of all plots.\n",
    "import matplotlib.pyplot as plt\n",
    "plt.rcParams['figure.figsize'] = [11, 7]\n",
    "\n",
    "# Plotting USA born winners \n",
    "ax = sns.lineplot(x='decade', y='usa_born_winner', data=prop_usa_winners)\n",
    "\n",
    "# Adding %-formatting to the y-axis\n",
    "from matplotlib.ticker import PercentFormatter\n",
    "ax.yaxis.set_major_formatter(PercentFormatter(1.0))"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {
    "dc": {
     "key": "31"
    },
    "deletable": false,
    "editable": false,
    "run_control": {
     "frozen": true
    },
    "tags": [
     "context"
    ]
   },
   "outputs": [],
   "source": [
    "## 5. What is the gender of a typical Nobel Prize winner?\n",
    "<p>So the USA became the dominating winner of the Nobel Prize first in the 1930s and had kept the leading position ever since. But one group that was in the lead from the start, and never seems to let go, are <em>men</em>. Maybe it shouldn't come as a shock that there is some imbalance between how many male and female prize winners there are, but how significant is this imbalance? And is it better or worse within specific prize categories like physics, medicine, literature, etc.?</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Task 5:\n",
    "Plot the proportion of female laureates by decade split by prize category.\n",
    "\n",
    "* Add the `female_winner` column to `nobel`, where the value is `True` when `sex` is `\"Female\"`.\n",
    "* Use `groupby` to group by both `decade` and `category`, setting `as_index=False`. Then isolate the `female_winner` column and take the `mean()`. Assign the resulting DataFrame to prop_female_winners.\n",
    "* Copy and paste your `seaborn` plot from task 4 (including axis formatting code), but plot `prop_female_winners` and map the `category` variable to the `hue` parameter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Calculating the proportion of female laureates per decade\n",
    "nobel['female_winner'] = nobel['sex']  == 'Female'\n",
    "prop_female_winners = nobel.groupby(['decade', 'category'], as_index=False)['female_winner'].mean()\n",
    "\n",
    "# Plotting female winners \n",
    "ax = sns.lineplot(x='decade', y='female_winner', data=prop_female_winners, hue='category')\n",
    "\n",
    "# Adding %-formatting to the y-axis\n",
    "ax.yaxis.set_major_formatter(PercentFormatter(1.0))"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {
    "dc": {
     "key": "38"
    },
    "deletable": false,
    "editable": false,
    "run_control": {
     "frozen": true
    },
    "tags": [
     "context"
    ]
   },
   "outputs": [],
   "source": [
    "## 6. The first woman to win the Nobel Prize\n",
    "<p>The plot above is a bit messy as the lines are overplotting. But it does show some interesting trends and patterns. Overall the imbalance is pretty large with physics, economics, and chemistry having the largest imbalance. Medicine has a somewhat positive trend, and since the 1990s the literature prize is also now more balanced. The big outlier is the peace prize during the 2010s, but keep in mind that this just covers the years 2010 to 2016.</p>\n",
    "<p>Given this imbalance, who was the first woman to receive a Nobel Prize? And in what category?</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Task 6:\n",
    "Extract and display the row showing the first woman to win a Nobel Prize.\n",
    "* Select only the rows of `'Female'` winners in `nobel`.\n",
    "* Using the `nsmallest()` method with its `n` and `columns` parameters, pick out the first woman to get a Nobel Prize.\n",
    "\n",
    "See [here](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.nsmallest.html) for the documentation of nsmallest()."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Picking out the first woman to win a Nobel Prize\n",
    "nobel[nobel['sex'] == 'Female'].nsmallest(n=1, columns='year')"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {
    "dc": {
     "key": "45"
    },
    "deletable": false,
    "editable": false,
    "run_control": {
     "frozen": true
    },
    "tags": [
     "context"
    ]
   },
   "outputs": [],
   "source": [
    "## 7. Repeat laureates\n",
    "<p>For most scientists/writers/activists a Nobel Prize would be the crowning achievement of a long career. But for some people, one is just not enough, and few have gotten it more than once. Who are these lucky few? </p>"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Task 7:\n",
    "Extract and display the rows of repeat Nobel Prize winners.\n",
    "\n",
    "* Use `groupby` to group `nobel` by `'full_name'`.\n",
    "* Use the `filter` method to keep only those rows in `nobel` with winners with 2 or more prizes.\n",
    "\n",
    "See [here](https://pandas.pydata.org/pandas-docs/stable/user_guide/groupby.html#filtration) for how to use the filter method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Selecting the laureates that have received 2 or more prizes.\n",
    "nobel.groupby('full_name').filter(lambda x: len(x) >= 2)"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {
    "dc": {
     "key": "52"
    },
    "deletable": false,
    "editable": false,
    "run_control": {
     "frozen": true
    },
    "tags": [
     "context"
    ]
   },
   "outputs": [],
   "source": [
    "## 8. Nobel Laureates by age\n",
    "<p>The list of repeat winners contains some illustrious names! We again meet Marie Curie, who got the prize in physics for discovering radiation and in chemistry for isolating radium and polonium. John Bardeen got it twice in physics for transistors and superconductivity, Frederick Sanger got it twice in chemistry, and Linus Carl Pauling got it first in chemistry and later in peace for his work in promoting nuclear disarmament. We also learn that organizations also get the prize as both the Red Cross and the UNHCR have gotten it twice.</p>\n",
    "<p>But, generally, how old were the prize winners?</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Task 8:\n",
    "Calculate and plot the age of each winner when they won their Nobel Prize.\n",
    "\n",
    "* Convert the `nobel['birth_date']` column to datetime using `pd.to_datetime`.\n",
    "* Add a new column `nobel['age']` that contains the age of each winner when they got the prize. That is, year of prize win minus birth year.\n",
    "* Use `sns.lmplot` (not `sns.lineplot`) to make a plot with `year` on the x-axis and `age` on the y-axis.\n",
    "\n",
    "To get the year from a datetime column you need to use access the `dt.year` value. Here is an example:\n",
    "```\n",
    "a_data_frame['a_datatime_column'].dt.year\n",
    "```\n",
    "\n",
    "Seaborn's `lmplot` is a 2D scatterplot with an optional overlaid regression line. This type of plot is useful for [visualizing linear relationships](https://seaborn.pydata.org/tutorial/regression.html).\n",
    "\n",
    "To make the plot prettier, add the arguments `lowess=True`, `aspect=2`, and `line_kws={'color' : 'black'}`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Converting birth_date from String to datetime\n",
    "nobel['birth_date'] = pd.to_datetime(nobel['birth_date'])\n",
    "\n",
    "# Calculating the age of Nobel Prize winners\n",
    "nobel['age'] = nobel['year'] - nobel['birth_date'].dt.year\n",
    "\n",
    "# Plotting the age of Nobel Prize winners\n",
    "sns.lmplot(x=\"year\", y=\"age\", data=nobel, lowess=True, aspect=2,line_kws={'color' : 'black'})"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {
    "dc": {
     "key": "59"
    },
    "deletable": false,
    "editable": false,
    "run_control": {
     "frozen": true
    },
    "tags": [
     "context"
    ]
   },
   "outputs": [],
   "source": [
    "## 9. Age differences between prize categories\n",
    "<p>The plot above shows us a lot! We see that people use to be around 55 when they received the price, but nowadays the average is closer to 65. But there is a large spread in the laureates' ages, and while most are 50+, some are very young.</p>\n",
    "<p>We also see that the density of points is much high nowadays than in the early 1900s -- nowadays many more of the prizes are shared, and so there are many more winners. We also see that there was a disruption in awarded prizes around the Second World War (1939 - 1945). </p>\n",
    "<p>Let's look at age trends within different prize categories.</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Task 9:\n",
    "Plot how old winners are within the different price categories.\n",
    "\n",
    "* As before, use `sns.lmplot` to make a plot with `year` on the x-axis and `age` on the `y-axis`. But this time, make one plot per prize category by setting the `row` argument to `'category'`.\n",
    "\n",
    "This is the same plot as in task 8, except with the added row= argument (examples in the official Seaborn documentation [here](https://seaborn.pydata.org/generated/seaborn.lmplot.html))."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Same plot as above, but separate plots for each type of Nobel Prize\n",
    "sns.lmplot(x=\"year\", y=\"age\", data=nobel, lowess=True, aspect=2,line_kws={'color' : 'black'}, row='category')"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {
    "dc": {
     "key": "66"
    },
    "deletable": false,
    "editable": false,
    "run_control": {
     "frozen": true
    },
    "tags": [
     "context"
    ]
   },
   "outputs": [],
   "source": [
    "## 10. Oldest and youngest winners\n",
    "<p>More plots with lots of exciting stuff going on! We see that both winners of the chemistry, medicine, and physics prize have gotten older over time. The trend is strongest for physics: the average age used to be below 50, and now it's almost 70. Literature and economics are more stable. We also see that economics is a newer category. But peace shows an opposite trend where winners are getting younger! </p>\n",
    "<p>In the peace category we also a winner around 2010 that seems exceptionally young. This begs the questions, who are the oldest and youngest people ever to have won a Nobel Prize?</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Task 10:\n",
    "Pick out the rows of the oldest and the youngest winner of a Nobel Prize.\n",
    "\n",
    "* Use nlargest() to pick out and display the row of the oldest winner.\n",
    "* Use nsmallest() to pick out and display the row of the youngest winner.\n",
    "\n",
    "As before, you will need to use `display()` to display more than the last output of the cell. Here is the documentation for [nsmallest](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.nsmallest.html) and [nlargest](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.nlargest.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The oldest winner of a Nobel Prize as of 2016\n",
    "display(nobel.nlargest(n=1, columns='age'))\n",
    "\n",
    "# The youngest winner of a Nobel Prize as of 2016\n",
    "nobel.nsmallest(n=1, columns='age')"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {
    "dc": {
     "key": "73"
    },
    "deletable": false,
    "editable": false,
    "run_control": {
     "frozen": true
    },
    "tags": [
     "context"
    ]
   },
   "outputs": [],
   "source": [
    "## 11. Name of youngest winner!\n",
    "<p>What was the name of the youngest winner ever who in 2014 got the prize for \"[her] struggle against the suppression of children and young people and for the right of all children to education\"?</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Task  11:\n",
    "Assign the name of the youngest winner of a Nobel Prize to `youngest_winner`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The name of the youngest winner of the Nobel Prize as of 2016\n",
    "youngest_winner = nobel.nsmallest(n=1, columns='age')['full_name'].iloc[0]\n",
    "youngest_winner"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "---\n",
    "**[Week 2 - Data Analysis and Visualisation](https://radu-enuca.gitbook.io/ml-challenge/data-analysis-and-visualisation)**\n",
    "\n",
    "*Have questions or comments? Visit the ML Challenge Mattermost Channel.*"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3-final"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}