Web Scraping.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As a part of my current job, I had to migrate data from a HTML page to a different format and it contained 112 rows with 9 different components that had to be extracted from each row. So I decided that this would be an excellent opportunity to learn web page crawling and use it to get the job done. I used Beautiful Soup library for parsing the HTML tags and their content and the library is quite easy to use and extensive. And I also utilised Pandas library for saving the data in a dataframe for further preprocessing as you will see later in the script.\n",
    "\n",
    "As there is a severe lack of good tutorials on Web crawling(Most of them only cover the basics and simple pages), I had to rely solely on Beautiful Soup's documentation (https://www.crummy.com/software/BeautifulSoup/bs4/doc/). Luckily, the documentation is quite user friendly and easy to understand and I was able to learn the library on the go as well as finish the script in a matter of a few hours. So here goes.\n",
    "\n",
    "Since the page will eventually be taken down from the server after the migration, I have added the original HTML page to the repository as well.\n",
    "\n",
    "Disclaimer: I am fairly new to pandas and Beautiful Soup library so a lot of things done in this notebook may not necessarily be the definitive way to do things. Any feedback is welcome. And the data I am using for this is available publicly on a website so there is no breach of privacy in any way."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Import the necessary libraries\n",
    "# requests library is used to send a GET request to the url specified to extract the required information\n",
    "import requests\n",
    "from bs4 import BeautifulSoup\n",
    "import pandas as pd\n",
    "import re\n",
    "\n",
    "# Extract the required information from the given url\n",
    "response = requests.get(\"http://www.chemeng.unimelb.edu.au/people/rhd.html#Abdellah\")\n",
    "content = response.content\n",
    "\n",
    "# Parse the extracted information using Beautiful Soup library\n",
    "parser = BeautifulSoup(content,'html.parser')\n",
    "\n",
    "# Extract the content with <body></body> tags from the HTML page\n",
    "body = parser.body\n",
    "\n",
    "#Uncomment the below variable and run this cell to view the output\n",
    "#body "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Extract the 'h2' tags from the body to get all the names and ids of the students from the page \n",
    "h2 = body.find_all(\"h2\")\n",
    "ids = []\n",
    "names = []\n",
    "\n",
    "for h in h2:\n",
    "    ids.append(h['id'])\n",
    "    names.append(h.text)\n",
    "    \n",
    "#ids\n",
    "#names"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A dataframe is created containing the ids of all users present within the 'ids' list and saved under the 'ID' column in the dataframe. We will keep adding more columns to this as we extract more information from the page."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "df = pd.DataFrame(ids,columns=['ID'])\n",
    "\n",
    "# Add 'Names' column to the dataframe and pass the names list to it containing the names of all the students present on the page.\n",
    "df['Names'] = names"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All the other information about the students is present within divs with class 'col-2'. So we will find all of these divs and then extract the info from them one by one. \n",
    "\n",
    "As can be seen in the HTML page given, the images present in the page are saved in a div which not only has the class 'col-2' but also class 'first', so we will use the second class to further narrow down our search and extract the image sources from the page."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Extract all the divisions with the class col-2\n",
    "divs = body.find_all(\"div\",class_=\"col-2\")\n",
    "\n",
    "img_srcs = []\n",
    "\n",
    "# Extracting divs with class 'first' \n",
    "img_divs = body.find_all(\"div\",class_=\"first\") \n",
    "\n",
    "\n",
    "for imgs in img_divs:\n",
    "    # Check whether the divs retrieved have <img> tags within them as some users may not have uploaded their images\n",
    "    if (imgs.img != None):\n",
    "        # Extract the 'src' value from with the <img> tags\n",
    "        img_srcs.append(imgs.img['src'])\n",
    "    else:\n",
    "        img_srcs.append(\"None\")\n",
    "        \n",
    "# Save the image-sources to 'Image Sources' column in the dataframe.\n",
    "df['Image Sources'] = img_srcs[:-1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You will observe in the HTML page that all the information for each user is contained with four divs. The first div contains images, the second contains thesis titles, the third contains academic info such as Supervisors and the fourth contains the contact information. Thus, every fourth div contains the same kind of information but for different users.\n",
    "\n",
    "So, we will try and extract each piece of information for all the users collectively using the div value. I admit that this may not be the most efficient way to do it but it was the one I decided to go with at that moment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "thesis_titles = []\n",
    "i = 1\n",
    "\n",
    "while(i<447):\n",
    "    # Check whether the student has a thesis title info\n",
    "    if(divs[i].p != None):\n",
    "        thesis_titles.append(divs[i].p.text)\n",
    "    else:\n",
    "        thesis_titles.append(\"None\")\n",
    "    i += 4\n",
    "\n",
    "# Add the thesis titles to the dataframe\n",
    "df['Thesis Title'] = thesis_titles\n",
    "# df[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>Names</th>\n",
       "      <th>Image Sources</th>\n",
       "      <th>Thesis Title</th>\n",
       "      <th>Telephone</th>\n",
       "      <th>Email</th>\n",
       "      <th>Room</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Abdellah</td>\n",
       "      <td>Mohamed Hussein Ali Abdellah</td>\n",
       "      <td>/people/images/mohamed-abdellah.jpg</td>\n",
       "      <td>Solvent resistant nano-filtration for recovery...</td>\n",
       "      <td>T: 8344 8863</td>\n",
       "      <td>E: sendstudentmail(\"mabdellah\");</td>\n",
       "      <td>Room 215, Building 165</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Ali</td>\n",
       "      <td>Mr Suhaib  Ali</td>\n",
       "      <td>images/suhaib-ali.jpg</td>\n",
       "      <td>None</td>\n",
       "      <td>T: 8344 6640</td>\n",
       "      <td>E: sendstudentmail(\"s.ali4\");</td>\n",
       "      <td>Room B.07, C&amp;BE Building 167</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Allison</td>\n",
       "      <td>Stephanie Allison</td>\n",
       "      <td>/people/images/stephanie-allison.jpg</td>\n",
       "      <td>None</td>\n",
       "      <td>T: 8344 8678</td>\n",
       "      <td>E: sendstudentmail(\"s.allison\");</td>\n",
       "      <td>Room 5.01, Chemistry  East Building 154</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Bai</td>\n",
       "      <td>Tianyi (Alisa) Bai</td>\n",
       "      <td>/people/images/tianyi-bai.jpg</td>\n",
       "      <td>Polymer Surfactant Interactions in Emulsions</td>\n",
       "      <td>T: 8344 6640</td>\n",
       "      <td>E: sendstudentmail(\"t.bai2\");</td>\n",
       "      <td>Room B07, Desk 4, C&amp;BE Building 167</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Bhaskaran</td>\n",
       "      <td>Ayana Bhaskaran</td>\n",
       "      <td>/people/images/ayana-bhaskaran.jpg</td>\n",
       "      <td>Design of new artificial active sites</td>\n",
       "      <td></td>\n",
       "      <td>E: sendstudentmail(\"abhaskaran\");</td>\n",
       "      <td>Room B07, Desk 8, C&amp;BE Building 167</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          ID                         Names  \\\n",
       "0   Abdellah  Mohamed Hussein Ali Abdellah   \n",
       "1        Ali                Mr Suhaib  Ali   \n",
       "2    Allison             Stephanie Allison   \n",
       "3        Bai            Tianyi (Alisa) Bai   \n",
       "4  Bhaskaran               Ayana Bhaskaran   \n",
       "\n",
       "                          Image Sources  \\\n",
       "0   /people/images/mohamed-abdellah.jpg   \n",
       "1                 images/suhaib-ali.jpg   \n",
       "2  /people/images/stephanie-allison.jpg   \n",
       "3         /people/images/tianyi-bai.jpg   \n",
       "4    /people/images/ayana-bhaskaran.jpg   \n",
       "\n",
       "                                        Thesis Title     Telephone  \\\n",
       "0  Solvent resistant nano-filtration for recovery...  T: 8344 8863   \n",
       "1                                               None  T: 8344 6640   \n",
       "2                                               None  T: 8344 8678   \n",
       "3       Polymer Surfactant Interactions in Emulsions  T: 8344 6640   \n",
       "4              Design of new artificial active sites                 \n",
       "\n",
       "                               Email                                     Room  \n",
       "0   E: sendstudentmail(\"mabdellah\");                   Room 215, Building 165  \n",
       "1      E: sendstudentmail(\"s.ali4\");             Room B.07, C&BE Building 167  \n",
       "2   E: sendstudentmail(\"s.allison\");  Room 5.01, Chemistry  East Building 154  \n",
       "3      E: sendstudentmail(\"t.bai2\");      Room B07, Desk 4, C&BE Building 167  \n",
       "4  E: sendstudentmail(\"abhaskaran\");      Room B07, Desk 8, C&BE Building 167  "
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "telephone = []\n",
    "email = []\n",
    "room = []\n",
    "i = 3\n",
    "\n",
    "while(i<451):\n",
    "    # Check whether the div has contact info\n",
    "    if(divs[i].p != None):\n",
    "        # If the <p> tag contains a string starting with 'T:',add it to 'telephone' list\n",
    "        temp = divs[i].find_all(\"p\",string = re.compile(\"T:\"))\n",
    "        if temp:\n",
    "            telephone.append(temp[0].text)\n",
    "        else:\n",
    "            telephone.append(' ')\n",
    "        # If the <p> tag contains a string starting with 'R' or 'O',add it to 'room' list\n",
    "        temp_room = divs[i].find_all(\"p\",string = re.compile(\"[RO]\"))\n",
    "        if temp_room:\n",
    "            room.append(temp_room[0].text)\n",
    "        else:\n",
    "            room.append(' ')\n",
    "        # I tried using the same pattern as the one for telephone numbers, by finding the 'E:' string. \n",
    "        # But for some reason it didn't work, so I had to work out another way\n",
    "        # Finds all the <p> tags\n",
    "        temp_email = divs[i].find_all(\"p\")\n",
    "        for t in temp_email:\n",
    "            # If the tag's text contains \"E:\",append it to 'email' list\n",
    "            if \"E:\" in t.text:\n",
    "                email.append(t.text)\n",
    "            # If the tag's text is a telephone number or a room number,then it is not an emailid & append ' ' to 'email' list\n",
    "            elif t.text == temp and t.text == temp_room:\n",
    "                email.append(' ')\n",
    "    i += 4\n",
    "\n",
    "df['Telephone'] = telephone \n",
    "df['Email'] = email\n",
    "df['Room'] = room\n",
    "df[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "supervisors = []\n",
    "groups= []\n",
    "siblings=[]\n",
    "i = 2\n",
    "\n",
    "while (i<449):\n",
    "    # Check whether the div has supervisor or group info\n",
    "    if(divs[i].p!=None):\n",
    "        # If the <h3> tag contains a string starting with 'Supervisor',add it to 'supervisor' list\n",
    "        temp = divs[i].find_all(\"h3\",string=re.compile(\"Supervisor\"))\n",
    "        for t in temp:\n",
    "            # Get all the siblings of the given tag found above and add it to 'siblings' list\n",
    "            # t.next_siblings produces a list of type 'generator', which contains variables of type 'tag'\n",
    "            siblings.append(t.next_siblings)\n",
    "    i += 4\n",
    "for items in siblings:\n",
    "    temp=[]\n",
    "    for item in items:\n",
    "        temp.append(item)\n",
    "    supervisors.append(temp)\n",
    "# supervisors[:3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Remove all the '\\n' from the list of lists\n",
    "supervisors_mod = []\n",
    "\n",
    "for lists in supervisors:\n",
    "    temp = []\n",
    "    for item in lists:\n",
    "        if item != '\\n':\n",
    "            temp.append(item)\n",
    "    supervisors_mod.append(temp)\n",
    "# supervisors_mod[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we need to split the Supervisors and Groups into two separate lists so we can save it into separate columns. This can be done by splitting on the h3 tag with the heading 'Discipline Group'. Everything before the h3 tag goes into supervisors_new list and after the tag goes into the group list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "supervisors_new = []\n",
    "groups= []\n",
    "\n",
    "for lists in supervisors_mod:\n",
    "    index = 0\n",
    "    for item in lists:\n",
    "        if item.name == 'h3':\n",
    "            # Get the index of the h3 tag from 'supervisors_mod' list for the split\n",
    "            index = lists.index(item)\n",
    "    groups.append(lists[index+1:])\n",
    "    supervisors_new.append(lists[:index])\n",
    "# supervisors_new[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "\n",
    "supervisors_final = []\n",
    "\n",
    "for lists in supervisors_new:\n",
    "    temp = []\n",
    "    for item in lists:\n",
    "        # Get the content within the <p> tags and add it to 'supervisors_final' list\n",
    "        # We keep the <a> tags as we need them in the new html file as it is\n",
    "        temp.append(item.contents[0])\n",
    "    supervisors_final.append(temp)\n",
    "\n",
    "df['Supervisors'] = supervisors_final\n",
    "# df\n",
    "# supervisors_final[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "groups_final = []\n",
    "\n",
    "for lists in groups:\n",
    "    temp = []\n",
    "    for item in lists:\n",
    "        # Get the content within the <p> tags and add it to 'groups_final' list\n",
    "        # We keep the <a> tags as we need them in the new html file as it is\n",
    "        temp.append(item.contents[0])\n",
    "    groups_final.append(temp)\n",
    "    \n",
    "df['Groups'] = groups_final\n",
    "# df[:5]\n",
    "# groups_final"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Add +61 to the telephone numbers\n",
    "df['Telephone'] = df['Telephone'].str.replace(\"T:\\s+\",\"+61 \")\n",
    "\n",
    "# Get the image sources url in the proper format\n",
    "df['Image Sources'] = df['Image Sources'].str.replace(\"/people/\",\"\")\n",
    "df['Image Sources'] = df['Image Sources'].str.replace(\"images\",\"/people/images\")\n",
    "# df[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Steps to extract emailids in the proper format\n",
    "df['Email'] = df['Email'].str.replace(\"E: sendstudentmail\\(\\\"\",\"\")\n",
    "df['Email'] = df['Email'].str.replace(\"E: sendpgradmail\\(\\\"\",\"\")\n",
    "df['Email'] = df['Email'].str.replace(\"\\\"\\);\",\"\")\n",
    "df['Email'] = df['Email'].str.replace(\";\",\"\")\n",
    "# df[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Errors found in the original html file on exploring the csv file generated later in the script\n",
    "df.loc[8]['Email'] = df.loc[8]['Email'].replace(\"d.biswas\",\"d.biviano\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>Names</th>\n",
       "      <th>Image Sources</th>\n",
       "      <th>Thesis Title</th>\n",
       "      <th>Telephone</th>\n",
       "      <th>Email</th>\n",
       "      <th>Room</th>\n",
       "      <th>Supervisors</th>\n",
       "      <th>Groups</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Abdellah</td>\n",
       "      <td>Mohamed Hussein Ali Abdellah</td>\n",
       "      <td>/people/images/mohamed-abdellah.jpg</td>\n",
       "      <td>Solvent resistant nano-filtration for recovery...</td>\n",
       "      <td>+61 8344 8863</td>\n",
       "      <td>mabdellah</td>\n",
       "      <td>Room 215, Building 165</td>\n",
       "      <td>[&lt;a href=\"staff.php?person_ID=14055\"&gt;Sandra Ke...</td>\n",
       "      <td>[&lt;a href=\"http://www.co2crc.com.au/\"&gt;CO2CRC&lt;/a&gt;]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Ali</td>\n",
       "      <td>Mr Suhaib  Ali</td>\n",
       "      <td>/people/images/suhaib-ali.jpg</td>\n",
       "      <td></td>\n",
       "      <td>+61 8344 6640</td>\n",
       "      <td>s.ali4</td>\n",
       "      <td>Room B.07, C&amp;BE Building 167</td>\n",
       "      <td>[&lt;a href=\"staff.php?person_ID=456615\"&gt;Paul Web...</td>\n",
       "      <td>[&lt;a href=\"/webley/\"&gt;Clean Energy Laboratory&lt;/a&gt;]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Allison</td>\n",
       "      <td>Stephanie Allison</td>\n",
       "      <td>/people/images/stephanie-allison.jpg</td>\n",
       "      <td></td>\n",
       "      <td>+61 8344 8678</td>\n",
       "      <td>s.allison</td>\n",
       "      <td>Room 5.01, Chemistry  East Building 154</td>\n",
       "      <td>[&lt;a href=\"staff.php?person_ID=1972\"&gt;Greg Qiao&lt;...</td>\n",
       "      <td>[&lt;a href=\"/polymer-science/\"&gt;Polymer Science&lt;/a&gt;]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Bai</td>\n",
       "      <td>Tianyi (Alisa) Bai</td>\n",
       "      <td>/people/images/tianyi-bai.jpg</td>\n",
       "      <td>Polymer Surfactant Interactions in Emulsions</td>\n",
       "      <td>+61 8344 6640</td>\n",
       "      <td>t.bai2</td>\n",
       "      <td>Room B07, Desk 4, C&amp;BE Building 167</td>\n",
       "      <td>[&lt;a href=\"staff.php?person_ID=11455\"&gt;Ray Dagas...</td>\n",
       "      <td>[&lt;a href=\"/dagastine/\"&gt;Dagastine Group&lt;/a&gt;]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Bhaskaran</td>\n",
       "      <td>Ayana Bhaskaran</td>\n",
       "      <td>/people/images/ayana-bhaskaran.jpg</td>\n",
       "      <td>Design of new artificial active sites</td>\n",
       "      <td></td>\n",
       "      <td>abhaskaran</td>\n",
       "      <td>Room B07, Desk 8, C&amp;BE Building 167</td>\n",
       "      <td>[&lt;a href=\"staff.php?person_ID=18324\"&gt;Luke Conn...</td>\n",
       "      <td>[&lt;a href=\"/connal/\"&gt;Connal Group&lt;/a&gt;]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          ID                         Names  \\\n",
       "0   Abdellah  Mohamed Hussein Ali Abdellah   \n",
       "1        Ali                Mr Suhaib  Ali   \n",
       "2    Allison             Stephanie Allison   \n",
       "3        Bai            Tianyi (Alisa) Bai   \n",
       "4  Bhaskaran               Ayana Bhaskaran   \n",
       "\n",
       "                          Image Sources  \\\n",
       "0   /people/images/mohamed-abdellah.jpg   \n",
       "1         /people/images/suhaib-ali.jpg   \n",
       "2  /people/images/stephanie-allison.jpg   \n",
       "3         /people/images/tianyi-bai.jpg   \n",
       "4    /people/images/ayana-bhaskaran.jpg   \n",
       "\n",
       "                                        Thesis Title      Telephone  \\\n",
       "0  Solvent resistant nano-filtration for recovery...  +61 8344 8863   \n",
       "1                                                     +61 8344 6640   \n",
       "2                                                     +61 8344 8678   \n",
       "3       Polymer Surfactant Interactions in Emulsions  +61 8344 6640   \n",
       "4              Design of new artificial active sites                  \n",
       "\n",
       "        Email                                     Room  \\\n",
       "0   mabdellah                   Room 215, Building 165   \n",
       "1      s.ali4             Room B.07, C&BE Building 167   \n",
       "2   s.allison  Room 5.01, Chemistry  East Building 154   \n",
       "3      t.bai2      Room B07, Desk 4, C&BE Building 167   \n",
       "4  abhaskaran      Room B07, Desk 8, C&BE Building 167   \n",
       "\n",
       "                                         Supervisors  \\\n",
       "0  [<a href=\"staff.php?person_ID=14055\">Sandra Ke...   \n",
       "1  [<a href=\"staff.php?person_ID=456615\">Paul Web...   \n",
       "2  [<a href=\"staff.php?person_ID=1972\">Greg Qiao<...   \n",
       "3  [<a href=\"staff.php?person_ID=11455\">Ray Dagas...   \n",
       "4  [<a href=\"staff.php?person_ID=18324\">Luke Conn...   \n",
       "\n",
       "                                              Groups  \n",
       "0   [<a href=\"http://www.co2crc.com.au/\">CO2CRC</a>]  \n",
       "1   [<a href=\"/webley/\">Clean Energy Laboratory</a>]  \n",
       "2  [<a href=\"/polymer-science/\">Polymer Science</a>]  \n",
       "3        [<a href=\"/dagastine/\">Dagastine Group</a>]  \n",
       "4              [<a href=\"/connal/\">Connal Group</a>]  "
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Replace the 'None' values with empty spaces\n",
    "df = df.replace('None','')\n",
    "df[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Write the data in a html file in the new format\n",
    "filename = \"rhd-new.html\"\n",
    "f = open(filename,'w')\n",
    "strs = \"\"\n",
    "complete = \"\"\n",
    "\n",
    "strs=\"\"\"<li class=\"person\">\n",
    "            <div class=\"person__photo\" style=\"background-image: url(http://www.chemeng.unimelb.edu.au%s\");\"></div>\n",
    "            <div class=\"person__info\">\n",
    "              <div class=\"person__profile\">\n",
    "                <h3><a data-bound=\"true\" id=\"%s\" href=\"#\">%s</a></h3>\n",
    "                <dl class=\"chemeng-def-list\">\n",
    "                  <dt><strong>Thesis title:</strong></dt>\n",
    "                  <dd>%s</dd>\\n\n",
    "                  <dt><strong>Supervisor:</strong></dt>\n",
    "                  %s\n",
    "                  <dt><strong>Discipline/Group:</strong></dt>                     \n",
    "                  %s\n",
    "                  <dt><strong>Location:</strong></dt>\n",
    "                  <dd>%s</dd>\n",
    "                </dl>\n",
    "              </div>\n",
    "              <div class=\"person__contact\">\n",
    "                <p class=\"person__phone\"><a href=\"tel:%s\">%s</a></p>\n",
    "                <p class=\"person__email\"><a href=\"mailto:%s@student.unimelb.edu.au\">%s@student.unimelb.edu.au</a></p>\n",
    "              </div>\n",
    "            </div>\n",
    "          </li>\\n\"\"\"\n",
    "\n",
    "supers = \"\"\"<dd>%s</dd>\\n\"\"\"\n",
    "\n",
    "for index in range(df.shape[0]):\n",
    "    supers_all = \"\"\n",
    "    groups_all = \"\"\n",
    "    for supervisor in df.iloc[index]['Supervisors']:\n",
    "        supers_all += supers % str(supervisor.encode('utf-8'))\n",
    "    for group in df.iloc[index]['Groups']:\n",
    "        groups_all += supers % str(group)\n",
    "    complete += strs %(df.iloc[index]['Image Sources'],df.iloc[index]['ID'],df.iloc[index]['Names'],\\\n",
    "                      df.iloc[index]['Thesis Title'],supers_all.decode('utf-8'),\\\n",
    "                      groups_all.decode('utf-8'),df.iloc[index]['Room'],\\\n",
    "                      df.iloc[index]['Telephone'],df.iloc[index]['Telephone'],\\\n",
    "                      df.iloc[index]['Email'],df.iloc[index]['Email'])\n",
    "\n",
    "f.write(complete.encode('utf-8'))\n",
    "f.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Write the dataframe to a csv file to ensure everything is correct and in the proper format\n",
    "df.to_csv(\"rhd.csv\",encoding='utf-8')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Outliers\n",
    "\n",
    "This script extracted most of the content out of the page, but there were a few exceptions:\n",
    "* 4 users had a '@pgrad.unimelb.edu.au' emailids instead of the '@student.unimelb.edu.au' emailids\n",
    "* 3 users had links to their online profiles in the original html file with the \"View Online Profile\" link\n",
    "\n",
    "Although these issues could be fixed with a little more code, but I decided against it as these were minor and rare anomalies and could be fixed faster manually."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}