snooty.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Snooty\n",
    "\n",
    "_A domain-specific language for scraping HTML._\n",
    "\n",
    "![snooty mascots](snooty.jpg)\n",
    "\n",
    "I've been thinking about how to streamline the process of creating a scraper. (By \"scraper\", I only mean a program that converts HTML to structured data, not a program that crawls the web.) One of the hassles is that there are multiple languages for traversing HTML documents (e.g. CSS and XPath) that have non-overlapping features, and neither language is really designed for scraping. \n",
    "\n",
    "CSS selectors are designed for stylesheets (obviously!) and lack some important features like selecting text or attribute nodes. It has other irritating limitations like it can select a node that is the next sibling of some selector, but it cannot select a node that is the previous sibling of some selector.\n",
    "\n",
    "XPaths are more powerful but the syntax is quite confusing and it is a language designed for XML and this leads to awkward constructions for parsing HTML, like `a[contains(@href,\"image\")]` which selects anchors matching a certain `href`.\n",
    "\n",
    "The [Parsel library](https://parsel.readthedocs.io/) makes it pretty easy to go back and forth between XPath and CSS selectors, and so in a sense this solves a part of the problem. \n",
    "\n",
    "But another part of the problem is how to safely run scraping code. Imagine that we wanted to add scraping capabilities to [Starbelly](https://starbelly.readthedocs.io/en/latest/). If Starbelly runs user-provided Python code, then the user can take full control of the server, because Python is basically impossible to Sandbox.\n",
    "\n",
    "The Splash project ran into this same dilemma. As a result, they chose Lua as their scripting language, specifically because it is a powerful language that is also easy to sandbox. Lua is a pretty unusual language. It takes time to learn, and it does not have the same library support as Python, i.e. libraries like Parsel.\n",
    "\n",
    "Another drawback to Python code is that it is difficult to debug. It is helpful to visualize what a scraper is doing or debug parts of it inside a web browser, but the web browser can't run Python code. In practice, I end up converting from Python to JavaScript, debugging in a browser, then converting back to Python and putting it into my scraper.\n",
    "\n",
    "One solution to this problem may be to write the scrapers in JavaScript. These would be easy to debug in a browser. Furthermore, JavaScript is obviously capable of running in a sandbox (that's what it does inside the browser, after all) but we still wouldn't have access to libraries like Parsel or `dateutil.parser`.\n",
    "\n",
    "Snooty is a possible middle ground: a scripting language specifically written for scraping HTML. Untrusted code can safely be executed, and the language can be designed to maximize productivity for scraping tasks. This notebook contains a simple prototype."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import locale\n",
    "from lxml import etree\n",
    "from pypeg2 import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Comment:\n",
    "    grammar = '#', attr('text', restline), endl\n",
    "\n",
    "class LocaleStatement:\n",
    "    grammar = 'locale', attr('locale', word)\n",
    "    \n",
    "    def __repr__(self):\n",
    "        return 'LocaleStatement[{}]'.format(self.locale)\n",
    "\n",
    "class Selector:\n",
    "    grammar = attr('parent', word), '->', attr('child', word)\n",
    "    \n",
    "    def __str__(self):\n",
    "        return 'Selector[parent={} child={}]'.format(self.parent, self.child)\n",
    "    \n",
    "class SelectorStatement:\n",
    "    grammar = 'selector', name(), '=', attr('selector', Selector)\n",
    "    \n",
    "    def __str__(self):\n",
    "        return 'SelectorStatement[{} = {}]'.format(self.name, self.selector)\n",
    "\n",
    "class DottedName(List):\n",
    "    grammar = csl(str, separator='.')\n",
    "    \n",
    "class ExportStatement:\n",
    "    grammar = 'export', attr('name', DottedName), '=', attr('selector', str)\n",
    "    \n",
    "class Script(List):\n",
    "    grammar = maybe_some([\n",
    "        Comment, \n",
    "        ExportStatement, \n",
    "        LocaleStatement, \n",
    "        SelectorStatement\n",
    "    ])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "def run(code, html):\n",
    "    ast = parse(scraper, Script)\n",
    "    doc = etree.fromstring(html)\n",
    "    selectors = {}\n",
    "    export = {}\n",
    "\n",
    "    for stmt in ast:\n",
    "        if isinstance(stmt, LocaleStatement):\n",
    "            locale.setlocale(locale.LC_ALL, stmt.locale)\n",
    "        elif isinstance(stmt, SelectorStatement):\n",
    "            selectors[stmt.name] = stmt.selector\n",
    "        elif isinstance(stmt, Comment):\n",
    "            pass\n",
    "        elif isinstance(stmt, ExportStatement):\n",
    "            exp = export\n",
    "            for name in stmt.name[:-1]:\n",
    "                if name not in exp:\n",
    "                    exp[name] = dict()\n",
    "                exp = exp[name]\n",
    "            sel = selectors[stmt.selector]\n",
    "            exp[stmt.name[-1]] = [etree.tostring(e) for e in \n",
    "                doc.xpath('//{}/{}'.format(sel.parent, sel.child))]\n",
    "        else:\n",
    "            print(stmt)\n",
    "\n",
    "    print(export)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "doc = r'''<!DOCTYPE html>\n",
    "<html>\n",
    "<body>\n",
    "    <h1>A Heading</h1>\n",
    "    <p>A paragraph. It contains <a href='/foo'>a link.</a></p>\n",
    "    <ol>\n",
    "        <li>Link <a href='/1.html'>one</a></li>\n",
    "        <li>Link <a href='/2.html'>two</a></li>\n",
    "        <li>Link <a href='/3.html'>three</a></li>\n",
    "    </ol>\n",
    "</body>\n",
    "</html>'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "scraper = r'''\n",
    "# You can define a locale used for parsing numbers and dates:\n",
    "locale en_US\n",
    "\n",
    "# The \"selector\" command defines a selection of HTML\n",
    "# nodes.\n",
    "selector paragraph = p -> a\n",
    "selector list_items = li -> a\n",
    "\n",
    "# The export command defines data elements that should\n",
    "# be produced as the result of scraping.\n",
    "export foo.paragraph = paragraph\n",
    "export foo.list = list_items\n",
    "'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'foo': {'paragraph': [b'<a href=\"/foo\">a link.</a>'], 'list': [b'<a href=\"/1.html\">one</a>', b'<a href=\"/2.html\">two</a>', b'<a href=\"/3.html\">three</a>']}}\n"
     ]
    }
   ],
   "source": [
    "run(scraper, doc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}