Td-00-added_files.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "The Beautiful Soup code from last time is at the bottom of this notebook so that it's available for re-use and also as a reminder that we'll need two functions, **`parse`** and **`to_csv`**, at minimum. We may *want* a function that scrapes the TED website directly, using the CSV from the Google Doc. \n",
    "\n",
    "What follows is an attempt to build the **`BeautifulSoup`** code from scratch because TED has completely re-written the HTML for the transcript pages. Most importantly, and a minor annoyance, they have dispensed with embedding the relative times for the talks. All that is available now is the total time, which is not placed in a `<meta>` tag in the head of the document. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "collapsed": true,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "from bs4 import BeautifulSoup\n",
    "import re"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "collapsed": true,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "# NB: no need to read() the file: BS4 reads it into its own kind of object\n",
    "thesoup = BeautifulSoup(open(\"./test/transcript.1.html\"), \"html5lib\")\n",
    "\n",
    "# Talk metadata is in <meta> tags in the <head>. \n",
    "# This finds all <meta> tags\n",
    "metas = thesoup.find_all(\"meta\")\n",
    "\n",
    "# Let's see what this object is...\n",
    "print(type(metas))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "collapsed": true,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "# ... and what's inside of it:\n",
    "print([meta for meta in metas])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Some early work focused on parsing the data as the list seen above:\n",
    "\n",
    "    print(metas[0])\n",
    "    >>> <meta charset=\"utf-8\"/>\n",
    "    \n",
    "    print(type(metas), type(metas[0]))\n",
    "    >>> <class 'bs4.element.ResultSet'> <class 'bs4.element.Tag'>\n",
    "\n",
    "Even used a list comprehension based on some BS 4 functionality:\n",
    "\n",
    "    metalist = [meta.attrs for meta in metas]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "The problem was getting the value of one attribute based on the value of another attribute, which seemed an impossibility, until I found a helpful thread on [SO][]. That resulted in the code below that produces very clean output. The `tag.get(\"\", None)` syntax is the BS4 `Tag.find()` function that returns None if there is no child tag within the `<meta>` tag. \n",
    "\n",
    "[SO]: https://stackoverflow.com/questions/36768068/get-meta-tag-content-property-with-beautifulsoup-and-python"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "collapsed": true,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "for tag in thesoup.find_all(\"meta\"):\n",
    "    if tag.get(\"name\", None) == \"author\":\n",
    "        speaker = tag.get(\"content\", None)\n",
    "    if tag.get(\"itemprop\", None) == \"duration\":\n",
    "        length = tag.get(\"content\", None)\n",
    "    if tag.get(\"itemprop\", None) == \"uploadDate\":\n",
    "        published = tag.get(\"content\", None)\n",
    "    if tag.get(\"itemprop\", None) == \"interactionCount\":\n",
    "        views = tag.get(\"content\", None)\n",
    "    if tag.get(\"itemprop\", None) == \"description\":\n",
    "        description = tag.get(\"content\", None)\n",
    "\n",
    "print(speaker, length, published, views, description)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "## Solution 1: Between 2 Comments\n",
    "\n",
    "The first block of code below returns all the transcropt, which is indeed conveniently housed in `<p>` tags, but so is some extraneous information in the footer. Fortunately, the paragraphs we need occur between two comments:\n",
    "\n",
    "```html\n",
    "    <!-- Transcript text -->\n",
    "        <p> All the text we want.</p>\n",
    "    <!-- /Transcript text -->\n",
    "```\n",
    "\n",
    "Parsing between two comments seems to be possible, according to these two discussions: [\"Extracting Text Between HTML Comments with BeautifulSoup\"][SO1] and [\"How do I parse just html between two comments using Python 3 and Beautiful Soup\"][SO2], but in the case of the former discussion, the solution is to grab the line after a comment and in the case of the second, the discussion focuses on using other tags. Because of that, I am trying Solution 2 below, but I still think there might be a way to build a better function that focused on grabbing all the paragraphs between the two comments.\n",
    "\n",
    "[SO1]: https://stackoverflow.com/questions/34673851/extracting-text-between-html-comments-with-beautifulsoup\n",
    "[SO2]: https://stackoverflow.com/questions/48794294/how-do-i-parse-just-html-between-two-comments-using-python-3-and-beautiful-soup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "collapsed": true,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "# Returns transcript, but also some footer information which is in paragraph tags\n",
    "text = thesoup.find_all(\"p\")\n",
    "print(text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "collapsed": true,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "for comment in thesoup.findAll(text=lambda text:isinstance(text, Comment)):\n",
    "    if comment in ['Transcript text']:\n",
    "        print(comment.next_element.strip())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "## Solution 2: Use the Div Class\n",
    "\n",
    "All the transcript paragraphs appear to be formatted like this:\n",
    "\n",
    "```html\n",
    "    <!-- Transcript text -->\n",
    "        <div class=\"Grid Grid--with-gutter d:f@md p-b:4\">\n",
    "\t\t\t<div class=\"Grid__cell d:f h:full m-b:.5 m-b:0@md w:12\"></div>\n",
    "                <div class=\"Grid__cell flx-s:1 p-r:4\">\n",
    "\t\t\t\t\t<p> By raising your hand, how many of you know at least \n",
    "                        ...\n",
    "```\n",
    "\n",
    "It's not semantic, and I suspect that focusing on **`Grid__cell flx-s:1 p-r:4`** we have only a temporary solution, but if that works in this moment, then I'll take it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "collapsed": true,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "# This produces a BS resultset which is 12 <div>s long\n",
    "text = thesoup.findAll(\"div\", {\"class\": \"Grid__cell flx-s:1 p-r:4\"})\n",
    "print(text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "collapsed": true,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "for div in thesoup.findAll(\"div\", {\"class\": \"Grid__cell flx-s:1 p-r:4\"}):\n",
    "        print(div.text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "collapsed": true,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "strung = ' '.join([div.text for div in \n",
    "            thesoup.findAll(\"div\", {\"class\": \"Grid__cell flx-s:1 p-r:4\"})])\n",
    "text   = re.sub(r\"[\\t]\", \"\", strung).replace(\"\\n\", \" \")\n",
    "print(text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "## Assembling the Functions\n",
    "\n",
    "Now that we have a working solution to get the data out of the html files, we can re-assemble the functions we need. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "button": false,
    "collapsed": true,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "import re\n",
    "import csv\n",
    "import os\n",
    "from bs4 import BeautifulSoup\n",
    "import pandas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "button": false,
    "collapsed": true,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "def parse(thesoup):\n",
    "    for tag in thesoup.find_all(\"meta\"):\n",
    "        if tag.get(\"name\", None) == \"author\":\n",
    "            speaker = tag.get(\"content\", None)\n",
    "        if tag.get(\"itemprop\", None) == \"duration\":\n",
    "            length = tag.get(\"content\", None)\n",
    "        if tag.get(\"itemprop\", None) == \"uploadDate\":\n",
    "            published = tag.get(\"content\", None)\n",
    "        if tag.get(\"itemprop\", None) == \"interactionCount\":\n",
    "            views = tag.get(\"content\", None)\n",
    "        if tag.get(\"itemprop\", None) == \"description\":\n",
    "            description = tag.get(\"content\", None)\n",
    "    strung = ''.join([div.text for div in \n",
    "            thesoup.findAll(\"div\", {\"class\": \"Grid__cell flx-s:1 p-r:4\"})])\n",
    "    text   = re.sub(r\"[\\t]\", \"\", strung).replace(\"\\n\", \" \")\n",
    "    return speaker, length, published, views, description, text\n",
    "\n",
    "def to_csv(pth, out):\n",
    "    # open file to write to.\n",
    "    with open(out, \"w\") as out:\n",
    "        # create csv.writer. \n",
    "        wr = csv.writer(out)\n",
    "        # write our headers.\n",
    "        wr.writerow([\"author\", \"length\", \"published\", \"views\", \"description\", \"text\"])\n",
    "        # get all our html files.\n",
    "        for html in os.listdir(pth):\n",
    "            with open(os.path.join(pth, html)) as f:\n",
    "                # parse the file are write the data to a row.\n",
    "                wr.writerow(parse(BeautifulSoup(f, \"html5lib\")))\n",
    "                \n",
    "# to_csv(\"./test\",\"test.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "button": false,
    "collapsed": true,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "to_csv(\"./test\",\"test7.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "button": false,
    "collapsed": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>author</th>\n",
       "      <th>length</th>\n",
       "      <th>published</th>\n",
       "      <th>views</th>\n",
       "      <th>description</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>author</td>\n",
       "      <td>length</td>\n",
       "      <td>published</td>\n",
       "      <td>views</td>\n",
       "      <td>description</td>\n",
       "      <td>text</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Laura Indolfi</td>\n",
       "      <td>PT6M3S</td>\n",
       "      <td>2016-05-17T14:46:20+00:00</td>\n",
       "      <td>1246654</td>\n",
       "      <td>Anyone who has lost a loved one to pancreatic ...</td>\n",
       "      <td>By raising your hand, how many of you know a...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Sebastian Junger</td>\n",
       "      <td>PT13M28S</td>\n",
       "      <td>2016-05-18T14:22:09+00:00</td>\n",
       "      <td>866459</td>\n",
       "      <td>Sebastian Junger has seen war up close, and he...</td>\n",
       "      <td>I worked as a war reporter for 15 years befo...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Juno Mac</td>\n",
       "      <td>PT17M50S</td>\n",
       "      <td>2016-05-19T15:09:27+00:00</td>\n",
       "      <td>1964067</td>\n",
       "      <td>Everyone has an opinion about how to legislate...</td>\n",
       "      <td>I'm not like most of the people you'll have ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             author    length                  published    views  \\\n",
       "0            author    length                  published    views   \n",
       "1     Laura Indolfi    PT6M3S  2016-05-17T14:46:20+00:00  1246654   \n",
       "2  Sebastian Junger  PT13M28S  2016-05-18T14:22:09+00:00   866459   \n",
       "3          Juno Mac  PT17M50S  2016-05-19T15:09:27+00:00  1964067   \n",
       "\n",
       "                                         description  \\\n",
       "0                                        description   \n",
       "1  Anyone who has lost a loved one to pancreatic ...   \n",
       "2  Sebastian Junger has seen war up close, and he...   \n",
       "3  Everyone has an opinion about how to legislate...   \n",
       "\n",
       "                                                text  \n",
       "0                                               text  \n",
       "1    By raising your hand, how many of you know a...  \n",
       "2    I worked as a war reporter for 15 years befo...  \n",
       "3    I'm not like most of the people you'll have ...  "
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "with open('./test7.csv') as f:\n",
    "    colnames = f.readline().strip().split(\",\")\n",
    "df = pandas.read_csv('./test7.csv', names=colnames)\n",
    "\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "collapsed": true,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}