Population_Data_Download.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<script>\n",
       "    // AUTORUN ALL CELLS ON NOTEBOOK-LOAD!\n",
       "    require(\n",
       "        ['base/js/namespace', 'jquery'], \n",
       "        function(jupyter, $) {\n",
       "            $(jupyter.events).on(\"kernel_ready.Kernel\", function () {\n",
       "                console.log(\"Auto-running all cells-below...\");\n",
       "                jupyter.actions.call('jupyter-notebook:run-all-cells-below');\n",
       "                jupyter.notebook.scroll_to_top();\n",
       "                jupyter.actions.call('jupyter-notebook:save-notebook');                \n",
       "                \n",
       "            });\n",
       "        });\n",
       "        \n",
       "        $( document ).ready(function(){\n",
       "        code_shown=false;\n",
       "        $('div.input').hide()});\n",
       "    \n",
       "    \n",
       "</script>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%html\n",
    "<script>\n",
    "    // AUTORUN ALL CELLS ON NOTEBOOK-LOAD!\n",
    "    require(\n",
    "        ['base/js/namespace', 'jquery'], \n",
    "        function(jupyter, $) {\n",
    "            $(jupyter.events).on(\"kernel_ready.Kernel\", function () {\n",
    "                console.log(\"Auto-running all cells-below...\");\n",
    "                jupyter.actions.call('jupyter-notebook:run-all-cells-below');\n",
    "                jupyter.notebook.scroll_to_top();\n",
    "                jupyter.actions.call('jupyter-notebook:save-notebook');                \n",
    "                \n",
    "            });\n",
    "        });\n",
    "        \n",
    "        $( document ).ready(function(){\n",
    "        code_shown=false;\n",
    "        $('div.input').hide()});\n",
    "    \n",
    "    \n",
    "</script>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: Above this cell is a hidden cell that hides and runs all code in the file. This is intended for those who do not want to see or interact with the code. It can be seen by converting the cell to markdown(see toolbar above) and then back to code. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Overview \n",
    "\n",
    "This program is intended to obtain population and demographic by country from Worldpop.org. It is designed to explore a country or specific location within that country's population and demographic information and produce a table that can be used to generate a realistic syntehtic population. It comes in two parts. \n",
    "\n",
    "* Data Download of Population and Demographic Information\n",
    "* Population and Demogrpahic Data Exploration and Conversion\n",
    "\n",
    "\n",
    "**Citation Information**\n",
    "WorldPop (www.worldpop.org - School of Geography and Environmental Science, University of Southampton; Department of Geography and Geosciences, University of Louisville; Departement de Geographie, Universite de Namur) and Center for International Earth Science Information Network (CIESIN), Columbia University (2018). Global High Resolution Population Denominators Project - Funded by The Bill and Melinda Gates Foundation (OPP1134076). https://dx.doi.org/10.5258/SOTON/WP00645\n",
    "\n",
    "**License Information**\n",
    "License is located at https://www.worldpop.org/data/licence.txt\n",
    "\n",
    "\n",
    "### User Instructions\n",
    "\n",
    "This program is designed to be skill scalable so users can either just retrieve the data with no coding or jump into the code at desired spots to make their own changes.\n",
    "\n",
    "Some user tips: \n",
    "* The interaction widgets are live, they update as soon as you select something. \n",
    "* Hitting return or run cell after a selection is not necessary.\n",
    "* The workflow is sequential. Unless you want to change something you should only move down the workflow. If you move up the workflow and change something it is important to run every cell below it to update all the information. \n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 0: The Dependencies\n",
    "\n",
    "This cell imports the python dependencies we use to download the data. It has already run. If you want to know which dependencies this code is using click Hide/Show Code. If you click the run cell button, the buttons for this cell will disappear. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <script>\n",
       "    function code_toggle() {\n",
       "        if ($('div.cell.code_cell.rendered.selected div.input').css('display') !='none'){\n",
       "            $('div.cell.code_cell.rendered.selected div.input').hide('500');\n",
       "        } else {\n",
       "            $('div.cell.code_cell.rendered.selected div.input').show('500');\n",
       "            \n",
       "        }\n",
       "        \n",
       "    }\n",
       "    </script>\n",
       "\n",
       "\n",
       "<form action=\"javascript:code_toggle()\"><input type=\"submit\" id=\"toggleButton\" value=\"Hide/Show Code\"></form>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<form action=\"javascript:Jupyter.notebook.execute_cell()\"><input type =\"submit\" id=\"runButton\" value=\"Run Cell\"></form>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from toggle_code import toggle_code as hide_code\n",
    "from toggle_code import run_code as run_code\n",
    "import requests\n",
    "import pandas as pd\n",
    "import ipywidgets as widgets\n",
    "from ipywidgets import interact\n",
    "import urllib.request as request\n",
    "import os"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1: Country Selection\n",
    "\n",
    "Select **one** country to download its information"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "<form action=\"javascript:code_toggle()\"><input type=\"submit\" id=\"toggleButton\" value=\"Hide/Show Code\"></form>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<form action=\"javascript:Jupyter.notebook.execute_cell()\"><input type =\"submit\" id=\"runButton\" value=\"Run Cell\"></form>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "aa16fa3c39184724846859b0e0458fc2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "interactive(children=(Select(description='Countries', options=('Select A Country', 'Afghanistan', 'Albania', '…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "hide_code()\n",
    "run_code()\n",
    "#Country Look Up table\n",
    "\n",
    "countries= {\"Select A Country\": \"AAA\", 'Afghanistan': 'AFG',\n",
    " 'Albania': 'ALB',\n",
    " 'Algeria': 'DZA',\n",
    " 'American Samoa': 'ASM',\n",
    " 'Andorra': 'AND',\n",
    " 'Angola': 'AGO',\n",
    " 'Anguilla': 'AIA',\n",
    " 'Antarctica': 'ATA',\n",
    " 'Antigua and Barbuda': 'ATG',\n",
    " 'Argentina': 'ARG',\n",
    " 'Armenia': 'ARM',\n",
    " 'Aruba': 'ABW',\n",
    " 'Australia': 'AUS',\n",
    " 'Austria': 'AUT',\n",
    " 'Azerbaijan': 'AZE',\n",
    " 'Bahamas': 'BHS',\n",
    " 'Bahrain': 'BHR',\n",
    " 'Bangladesh': 'BGD',\n",
    " 'Barbados': 'BRB',\n",
    " 'Belarus': 'BLR',\n",
    " 'Belgium': 'BEL',\n",
    " 'Belize': 'BLZ',\n",
    " 'Benin': 'BEN',\n",
    " 'Bermuda': 'BMU',\n",
    " 'Bhutan': 'BTN',\n",
    " 'Bolivia, Plurinational State of': 'BOL',\n",
    " 'Bolivia': 'BOL',\n",
    " 'Bosnia and Herzegovina': 'BIH',\n",
    " 'Botswana': 'BWA',\n",
    " 'Bouvet Island': 'BVT',\n",
    " 'Brazil': 'BRA',\n",
    " 'British Indian Ocean Territory': 'IOT',\n",
    " 'Brunei Darussalam': 'BRN',\n",
    " 'Brunei': 'BRN',\n",
    " 'Bulgaria': 'BGR',\n",
    " 'Burkina Faso': 'BFA',\n",
    " 'Burundi': 'BDI',\n",
    " 'Cambodia': 'KHM',\n",
    " 'Cameroon': 'CMR',\n",
    " 'Canada': 'CAN',\n",
    " 'Cape Verde': 'CPV',\n",
    " 'Cayman Islands': 'CYM',\n",
    " 'Central African Republic': 'CAF',\n",
    " 'Chad': 'TCD',\n",
    " 'Chile': 'CHL',\n",
    " 'China': 'CHN',\n",
    " 'Christmas Island': 'CXR',\n",
    " 'Cocos (Keeling) Islands': 'CCK',\n",
    " 'Colombia': 'COL',\n",
    " 'Comoros': 'COM',\n",
    " 'Congo': 'COG',\n",
    " 'Congo, the Democratic Republic of the': 'COD',\n",
    " 'Cook Islands': 'COK',\n",
    " 'Costa Rica': 'CRI',\n",
    " \"Côte d'Ivoire\": 'CIV',\n",
    " 'Ivory Coast': 'CIV',\n",
    " 'Croatia': 'HRV',\n",
    " 'Cuba': 'CUB',\n",
    " 'Cyprus': 'CYP',\n",
    " 'Czech Republic': 'CZE',\n",
    " 'Denmark': 'DNK',\n",
    " 'Djibouti': 'DJI',\n",
    " 'Dominica': 'DMA',\n",
    " 'Dominican Republic': 'DOM',\n",
    " 'Ecuador': 'ECU',\n",
    " 'Egypt': 'EGY',\n",
    " 'El Salvador': 'SLV',\n",
    " 'Equatorial Guinea': 'GNQ',\n",
    " 'Eritrea': 'ERI',\n",
    " 'Estonia': 'EST',\n",
    " 'Ethiopia': 'ETH',\n",
    " 'Falkland Islands (Malvinas)': 'FLK',\n",
    " 'Faroe Islands': 'FRO',\n",
    " 'Fiji': 'FJI',\n",
    " 'Finland': 'FIN',\n",
    " 'France': 'FRA',\n",
    " 'French Guiana': 'GUF',\n",
    " 'French Polynesia': 'PYF',\n",
    " 'French Southern Territories': 'ATF',\n",
    " 'Gabon': 'GAB',\n",
    " 'Gambia': 'GMB',\n",
    " 'Georgia': 'GEO',\n",
    " 'Germany': 'DEU',\n",
    " 'Ghana': 'GHA',\n",
    " 'Gibraltar': 'GIB',\n",
    " 'Greece': 'GRC',\n",
    " 'Greenland': 'GRL',\n",
    " 'Grenada': 'GRD',\n",
    " 'Guadeloupe': 'GLP',\n",
    " 'Guam': 'GUM',\n",
    " 'Guatemala': 'GTM',\n",
    " 'Guernsey': 'GGY',\n",
    " 'Guinea': 'GIN',\n",
    " 'Guinea-Bissau': 'GNB',\n",
    " 'Guyana': 'GUY',\n",
    " 'Haiti': 'HTI',\n",
    " 'Heard Island and McDonald Islands': 'HMD',\n",
    " 'Holy See (Vatican City State)': 'VAT',\n",
    " 'Honduras': 'HND',\n",
    " 'Hong Kong': 'HKG',\n",
    " 'Hungary': 'HUN',\n",
    " 'Iceland': 'ISL',\n",
    " 'India': 'IND',\n",
    " 'Indonesia': 'IDN',\n",
    " 'Iran, Islamic Republic of': 'IRN',\n",
    " 'Iraq': 'IRQ',\n",
    " 'Ireland': 'IRL',\n",
    " 'Isle of Man': 'IMN',\n",
    " 'Israel': 'ISR',\n",
    " 'Italy': 'ITA',\n",
    " 'Jamaica': 'JAM',\n",
    " 'Japan': 'JPN',\n",
    " 'Jersey': 'JEY',\n",
    " 'Jordan': 'JOR',\n",
    " 'Kazakhstan': 'KAZ',\n",
    " 'Kenya': 'KEN',\n",
    " 'Kiribati': 'KIR',\n",
    " \"Korea, Democratic People's Republic of\": 'PRK',\n",
    " 'Korea, Republic of': 'KOR',\n",
    " 'South Korea': 'KOR',\n",
    " 'Kuwait': 'KWT',\n",
    " 'Kyrgyzstan': 'KGZ',\n",
    " \"Lao People's Democratic Republic\": 'LAO',\n",
    " 'Latvia': 'LVA',\n",
    " 'Lebanon': 'LBN',\n",
    " 'Lesotho': 'LSO',\n",
    " 'Liberia': 'LBR',\n",
    " 'Libyan Arab Jamahiriya': 'LBY',\n",
    " 'Libya': 'LBY',\n",
    " 'Liechtenstein': 'LIE',\n",
    " 'Lithuania': 'LTU',\n",
    " 'Luxembourg': 'LUX',\n",
    " 'Macao': 'MAC',\n",
    " 'Macedonia, the former Yugoslav Republic of': 'MKD',\n",
    " 'Madagascar': 'MDG',\n",
    " 'Malawi': 'MWI',\n",
    " 'Malaysia': 'MYS',\n",
    " 'Maldives': 'MDV',\n",
    " 'Mali': 'MLI',\n",
    " 'Malta': 'MLT',\n",
    " 'Marshall Islands': 'MHL',\n",
    " 'Martinique': 'MTQ',\n",
    " 'Mauritania': 'MRT',\n",
    " 'Mauritius': 'MUS',\n",
    " 'Mayotte': 'MYT',\n",
    " 'Mexico': 'MEX',\n",
    " 'Micronesia, Federated States of': 'FSM',\n",
    " 'Moldova, Republic of': 'MDA',\n",
    " 'Monaco': 'MCO',\n",
    " 'Mongolia': 'MNG',\n",
    " 'Montenegro': 'MNE',\n",
    " 'Montserrat': 'MSR',\n",
    " 'Morocco': 'MAR',\n",
    " 'Mozambique': 'MOZ',\n",
    " 'Myanmar': 'MMR',\n",
    " 'Burma': 'MMR',\n",
    " 'Namibia': 'NAM',\n",
    " 'Nauru': 'NRU',\n",
    " 'Nepal': 'NPL',\n",
    " 'Netherlands': 'NLD',\n",
    " 'Netherlands Antilles': 'ANT',\n",
    " 'New Caledonia': 'NCL',\n",
    " 'New Zealand': 'NZL',\n",
    " 'Nicaragua': 'NIC',\n",
    " 'Niger': 'NER',\n",
    " 'Nigeria': 'NGA',\n",
    " 'Niue': 'NIU',\n",
    " 'Norfolk Island': 'NFK',\n",
    " 'Northern Mariana Islands': 'MNP',\n",
    " 'Norway': 'NOR',\n",
    " 'Oman': 'OMN',\n",
    " 'Pakistan': 'PAK',\n",
    " 'Palau': 'PLW',\n",
    " 'Palestinian Territory, Occupied': 'PSE',\n",
    " 'Panama': 'PAN',\n",
    " 'Papua New Guinea': 'PNG',\n",
    " 'Paraguay': 'PRY',\n",
    " 'Peru': 'PER',\n",
    " 'Philippines': 'PHL',\n",
    " 'Pitcairn': 'PCN',\n",
    " 'Poland': 'POL',\n",
    " 'Portugal': 'PRT',\n",
    " 'Puerto Rico': 'PRI',\n",
    " 'Qatar': 'QAT',\n",
    " 'Réunion': 'REU',\n",
    " 'Romania': 'ROU',\n",
    " 'Russian Federation': 'RUS',\n",
    " 'Russia': 'RUS',\n",
    " 'Rwanda': 'RWA',\n",
    " 'Saint Helena, Ascension and Tristan da Cunha': 'SHN',\n",
    " 'Saint Kitts and Nevis': 'KNA',\n",
    " 'Saint Lucia': 'LCA',\n",
    " 'Saint Pierre and Miquelon': 'SPM',\n",
    " 'Saint Vincent and the Grenadines': 'VCT',\n",
    " 'Saint Vincent & the Grenadines': 'VCT',\n",
    " 'St. Vincent and the Grenadines': 'VCT',\n",
    " 'Samoa': 'WSM',\n",
    " 'San Marino': 'SMR',\n",
    " 'Sao Tome and Principe': 'STP',\n",
    " 'Saudi Arabia': 'SAU',\n",
    " 'Senegal': 'SEN',\n",
    " 'Serbia': 'SRB',\n",
    " 'Seychelles': 'SYC',\n",
    " 'Sierra Leone': 'SLE',\n",
    " 'Singapore': 'SGP',\n",
    " 'Slovakia': 'SVK',\n",
    " 'Slovenia': 'SVN',\n",
    " 'Solomon Islands': 'SLB',\n",
    " 'Somalia': 'SOM',\n",
    " 'South Africa': 'ZAF',\n",
    " 'South Georgia and the South Sandwich Islands': 'SGS',\n",
    " 'South Sudan': 'SSD',\n",
    " 'Spain': 'ESP',\n",
    " 'Sri Lanka': 'LKA',\n",
    " 'Sudan': 'SDN',\n",
    " 'Suriname': 'SUR',\n",
    " 'Svalbard and Jan Mayen': 'SJM',\n",
    " 'Swaziland': 'SWZ',\n",
    " 'Sweden': 'SWE',\n",
    " 'Switzerland': 'CHE',\n",
    " 'Syrian Arab Republic': 'SYR',\n",
    " 'Taiwan, Province of China': 'TWN',\n",
    " 'Taiwan': 'TWN',\n",
    " 'Tajikistan': 'TJK',\n",
    " 'Tanzania, United Republic of': 'TZA',\n",
    " 'Thailand': 'THA',\n",
    " 'Timor-Leste': 'TLS',\n",
    " 'Togo': 'TGO',\n",
    " 'Tokelau': 'TKL',\n",
    " 'Tonga': 'TON',\n",
    " 'Trinidad and Tobago': 'TTO',\n",
    " 'Tunisia': 'TUN',\n",
    " 'Turkey': 'TUR',\n",
    " 'Turkmenistan': 'TKM',\n",
    " 'Turks and Caicos Islands': 'TCA',\n",
    " 'Tuvalu': 'TUV',\n",
    " 'Uganda': 'UGA',\n",
    " 'Ukraine': 'UKR',\n",
    " 'United Arab Emirates': 'ARE',\n",
    " 'United Kingdom': 'GBR',\n",
    " 'United States': 'USA',\n",
    " 'United States Minor Outlying Islands': 'UMI',\n",
    " 'Uruguay': 'URY',\n",
    " 'Uzbekistan': 'UZB',\n",
    " 'Vanuatu': 'VUT',\n",
    " 'Venezuela, Bolivarian Republic of': 'VEN',\n",
    " 'Venezuela': 'VEN',\n",
    " 'Viet Nam': 'VNM',\n",
    " 'Vietnam': 'VNM',\n",
    " 'Virgin Islands, British': 'VGB',\n",
    " 'Virgin Islands, U.S.': 'VIR',\n",
    " 'Wallis and Futuna': 'WLF',\n",
    " 'Western Sahara': 'ESH',\n",
    " 'Yemen': 'YEM',\n",
    " 'Zambia': 'ZMB',\n",
    " 'Zimbabwe': 'ZWE'}\n",
    "\n",
    "select_country = widgets.Select(options = list(countries.keys()),\n",
    "                                        description = \"Countries\",\n",
    "                                        disbaled = False, \n",
    "                                      )\n",
    "\n",
    "def country_select(country_data):\n",
    "    select = country_data\n",
    "    return select\n",
    "    \n",
    "select_country_out = interact(country_select, country_data = select_country)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Available Population Density Information\n",
    "\n",
    "The following outputs a table of available population density information. Each hyper link can take you to a summary of the information. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "<form action=\"javascript:code_toggle()\"><input type=\"submit\" id=\"toggleButton\" value=\"Hide/Show Code\"></form>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<form action=\"javascript:Jupyter.notebook.execute_cell()\"><input type =\"submit\" id=\"runButton\" value=\"Run Cell\"></form>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Waiting for inputs.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>Year</th>\n",
       "      <th>Date of File</th>\n",
       "      <th>File Details</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: [ID, Year, Date of File, File Details]\n",
       "Index: []"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hide_code()\n",
    "run_code()\n",
    "\n",
    "#get country code\n",
    "selection=select_country.value\n",
    "cntry = countries[selection]\n",
    "\n",
    "if cntry == \"AAA\":\n",
    "    print(\"Waiting for inputs.\")\n",
    "    available = {\"ID\":[],\"Year\": [],\"Date of File\":[],\"File Details\":[]}\n",
    "\n",
    "else:\n",
    "\n",
    "    url = 'https://worldpop.org/rest/data/pop/wpgp?iso3='+ cntry\n",
    "    r = requests.get(url)\n",
    "    print(\"Requested {} Population Data\".format(selection))\n",
    "    print()\n",
    "    print (\"Request url is \" + url)\n",
    "    data = r.json()\n",
    "\n",
    "    # Table for User\n",
    "    available = {\"ID\":[],\"Year\": [],\"Date of File\":[],\"File Details\":[]}\n",
    "\n",
    "    #Table for retrieval\n",
    "    prep = {\"ID\":[], \"data_file\":[]}\n",
    "\n",
    "    for d in data[\"data\"]: \n",
    "        available[\"ID\"].append(d[\"id\"])\n",
    "        available[\"Year\"].append(d['popyear'])\n",
    "        available[\"Date of File\"].append(d['date'])\n",
    "        available[\"File Details\"].append(d['url_summary'])\n",
    "        prep[\"ID\"].append(d['id'])\n",
    "        prep[\"data_file\"].append(d[\"data_file\"])\n",
    "\n",
    "\n",
    "pd.DataFrame.from_dict(available)   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Select Year(s)\n",
    "\n",
    "In the following you select the yer or years yuou want from the avialable data.\n",
    "\n",
    "You can select mutliple non-consecutive years with the **ctrl/cmd** key or consecutive years with the **shift* key\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "<form action=\"javascript:code_toggle()\"><input type=\"submit\" id=\"toggleButton\" value=\"Hide/Show Code\"></form>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<form action=\"javascript:Jupyter.notebook.execute_cell()\"><input type =\"submit\" id=\"runButton\" value=\"Run Cell\"></form>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Waiting for inputs.\n"
     ]
    }
   ],
   "source": [
    "hide_code()\n",
    "run_code()\n",
    "print()\n",
    "\n",
    "if cntry == \"AAA\":\n",
    "    print(\"Waiting for inputs.\")\n",
    "\n",
    "else:\n",
    "    select_year = widgets.SelectMultiple(options = list(available[\"Year\"]),\n",
    "                                            description = \"Year\",\n",
    "                                            disbaled = False, \n",
    "                                          )\n",
    "\n",
    "    def year_select(year):\n",
    "        select = year\n",
    "        return select\n",
    "\n",
    "    select_year_out = interact(year_select, year = select_year)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Download the Population Density Data\n",
    "\n",
    "Thoe following downloads the population Density data in a geotiff file that can be explored in the other *Population Exploration And Conversion* file. \n",
    "\n",
    "It saves all files to a direcotry called \"data\" in the same directory as this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "<form action=\"javascript:code_toggle()\"><input type=\"submit\" id=\"toggleButton\" value=\"Hide/Show Code\"></form>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<form action=\"javascript:Jupyter.notebook.execute_cell()\"><input type =\"submit\" id=\"runButton\" value=\"Run Cell\"></form>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Waiting for Inputs\n"
     ]
    }
   ],
   "source": [
    "hide_code()\n",
    "run_code()\n",
    "\n",
    "\n",
    "\n",
    "if cntry== \"AAA\":\n",
    "    print(\"Waiting for Inputs\")\n",
    "else:\n",
    "    select=select_year.value \n",
    "    os.makedirs(\"data\", exist_ok=True) \n",
    "    ftp_base = r'ftp://ftp.worldpop.org.uk'\n",
    "    for year in select: \n",
    "        print(\"Retrieving year {}...\".format(year))\n",
    "        #Get the index\n",
    "        idx = available[\"Year\"].index(year)\n",
    "        #Use to retrieve id\n",
    "        file = prep[\"data_file\"][idx]\n",
    "        name_prep = file.split('/')\n",
    "        filename=name_prep[-1]\n",
    "        #get file\n",
    "        ftp = ftp_base+'/'+file\n",
    "        data = request.Request(ftp)\n",
    "        with request.urlopen(data) as response:\n",
    "            DataBody = response.read() \n",
    "        filepath = os.path.join(\"data\", filename) \n",
    "        file_ = open(filepath, 'wb')\n",
    "        file_.write(DataBody)\n",
    "        file_.close()\n",
    "        print (filepath, \"has downloaded\")\n",
    "    \n",
    "    \n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 5. Available Demographic Data\n",
    "\n",
    "The following code shows what demographic data is available from Wolrdpop.org. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "<form action=\"javascript:code_toggle()\"><input type=\"submit\" id=\"toggleButton\" value=\"Hide/Show Code\"></form>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<form action=\"javascript:Jupyter.notebook.execute_cell()\"><input type =\"submit\" id=\"runButton\" value=\"Run Cell\"></form>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Waiting for Inputs.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>Year</th>\n",
       "      <th>Date of File</th>\n",
       "      <th>File Details</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: [ID, Year, Date of File, File Details]\n",
       "Index: []"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hide_code()\n",
    "run_code()\n",
    "\n",
    "#Allows user to skip population density\n",
    "selection=select_country.value\n",
    "cntry = countries[selection]\n",
    "\n",
    "\n",
    "if cntry == 'AAA':\n",
    "    print(\"Waiting for Inputs.\")\n",
    "\n",
    "else: \n",
    "    url = 'https://www.worldpop.org/rest/data/age_structures/aswpgp?iso3='+ cntry\n",
    "    r = requests.get(url)\n",
    "    print(\"Requested {} Age_Sex Data\".format(selection))\n",
    "    print()\n",
    "    print (\"Request url is \" + url)\n",
    "    data = r.json()\n",
    "\n",
    "    # Table for User\n",
    "    available = {\"ID\":[],\"Year\": [],\"Date of File\":[],\"File Details\":[]}\n",
    "\n",
    "    #Table for retrieval\n",
    "    prep = {\"ID\":[], \"data_file\":[]}\n",
    "\n",
    "    for d in data[\"data\"]: \n",
    "        available[\"ID\"].append(d[\"id\"])\n",
    "        available[\"Year\"].append(d['popyear'])\n",
    "        available[\"Date of File\"].append(d['date'])\n",
    "        available[\"File Details\"].append(d['url_summary'])\n",
    "        prep[\"ID\"].append(d['id'])\n",
    "        prep[\"data_file\"].append(d[\"data_file\"])\n",
    "    \n",
    "\n",
    "pd.DataFrame.from_dict(available)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 6. Download Demographic Data\n",
    "\n",
    "The following downloads all the demographic data for the select country and years."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "<form action=\"javascript:code_toggle()\"><input type=\"submit\" id=\"toggleButton\" value=\"Hide/Show Code\"></form>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<form action=\"javascript:Jupyter.notebook.execute_cell()\"><input type =\"submit\" id=\"runButton\" value=\"Run Cell\"></form>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Waiting for Inputs.\n"
     ]
    }
   ],
   "source": [
    "hide_code()\n",
    "run_code()\n",
    "\n",
    "def convert(databody):\n",
    "    out = []\n",
    "    body = databody.decode('utf-8')\n",
    "    body = body.split(\" \")\n",
    "    for b in body:\n",
    "        if \"tif\" in b:\n",
    "            b =b.split(\"\\r\")\n",
    "            out.append(b[0])\n",
    "    return out\n",
    "\n",
    "\n",
    "if cntry == 'AAA':\n",
    "    print(\"Waiting for Inputs.\")\n",
    "else:\n",
    "    os.makedirs(\"data\", exist_ok=True) \n",
    "    ftp_base = r'ftp://ftp.worldpop.org.uk'\n",
    "    for year in select: \n",
    "        print(\"Retrieving year {}...\".format(year))\n",
    "        #Get the index\n",
    "        idx = available[\"Year\"].index(year)\n",
    "        #Use to retrieve id\n",
    "        file = prep[\"data_file\"][idx]\n",
    "        name_prep = file.split('/')\n",
    "        filename=name_prep[-1]\n",
    "        #get file list\n",
    "        ftp = ftp_base+'/'+file\n",
    "        data = request.Request(ftp)\n",
    "        \n",
    "        with request.urlopen(data) as response:\n",
    "            DataBody = response.read() \n",
    "        files = convert(DataBody)\n",
    "        #get filenames\n",
    "    \n",
    "        #download demogrpahic files\n",
    "        for f in files: \n",
    "            demo_req = ftp + \"/\" + f\n",
    "            dem = request.Request(demo_req)\n",
    "            with request.urlopen(dem) as response:\n",
    "                dem_data = response.read()\n",
    "            filepath = os.path.join(\"data\", f)\n",
    "            file_ = open(filepath, 'wb')\n",
    "            file_.write(dem_data)\n",
    "            file_.close()\n",
    "            print (filepath, \"has downloaded\")\n",
    "    print(\"Download Complete\")    \n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}