JSON_parser_objectpath.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import zipfile\n",
    "import json\n",
    "import tqdm\n",
    "import requests\n",
    "import errno\n",
    "import os\n",
    "import objectpath"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# TODO\n",
    "Extract Data cell is one long and complex loop - essentially illegible. Use functions to isolate some lines and put them in a separate cell. Check occurence of `l` at the end of references - preventing int() when creating the `line` field. Problem appears in `labels_df` in particular in blms (bilingual texts with interlinears). It has been solved previously in other parsers.\n",
    "\n",
    "## 0 Create Directories, if Necessary\n",
    "The two directories needed for this script are `jsonzip` and `output`. If they do not exist they are created, else: do nothing.\n",
    "\n",
    "For the code, see [Stack Overflow](http://stackoverflow.com/questions/18973418/os-mkdirpath-returns-oserror-when-directory-does-not-exist)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "directories = ['jsonzip', 'output']\n",
    "for d in directories:\n",
    "    try:\n",
    "        os.mkdir(d)\n",
    "    except OSError as exc:\n",
    "        if exc.errno !=errno.EEXIST:\n",
    "            raise\n",
    "        pass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.1 Input Project Names\n",
    "Provide a list of one or more project names, separated by commas. Note that subprojects must be listed separately, they are not included in the main project. For instance:\n",
    "\n",
    "`saao/saa01,saao/saa02,blms`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Project(s): blms\n"
     ]
    }
   ],
   "source": [
    "projects = input('Project(s): ').lower()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.2 Split the List of Projects\n",
    "Split the list of projects and create a list of project names."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "p = projects.split(',')               # split at each comma and make a list called `p`\n",
    "p = [x.strip() for x in p]        # strip spaces left and right of each entry in `p`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.3 Download the ZIP files\n",
    "For each project in the list download all the `json` files from `http://build-oracc.museum.upenn.edu/json/`. The file is called `PROJECT.zip` (for instance: `dcclt.zip`). For subprojects the file is called `PROJECT-SUBPROJECT.zip` (for instance `cams-gkab.zip`). \n",
    "\n",
    "For larger projects (such as [DCCLT](http://oracc.org/dcclt)) the `zip` file may be 25Mb or more. Downloading may take some time and it may be necessary to chunk the downloading process. The `iter_content()` function in the `requests` library takes care of that. For the chunking code see [this page](https://www.smallsurething.com/how-to-read-a-file-properly-in-python/) by Evan Dempsey.\n",
    "\n",
    "If you have downloaded the files by hand (and put them in the `jsonzip` directory) you may skip this cell and jump directly to section ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|                                                                                            | 0/1 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading http://build-oracc.museum.upenn.edu/json/blms.zip saving as jsonzip/blms.zip\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|████████████████████████████████████████████████████████████████████████████████████| 1/1 [01:18<00:00, 78.77s/it]\n"
     ]
    }
   ],
   "source": [
    "CHUNK = 16 * 1024\n",
    "for project in tqdm.tqdm(p):\n",
    "    project = project.replace('/', '-')\n",
    "    url = \"http://build-oracc.museum.upenn.edu/json/\" + project + \".zip\"\n",
    "    file = 'jsonzip/' + project + '.zip'\n",
    "    r = requests.get(url)\n",
    "    if r.status_code == 200:\n",
    "        print(\"Downloading \" + url + \" saving as \" + file)\n",
    "        continue\n",
    "    else:\n",
    "        print(url + \" does not exist.\")\n",
    "    with open(file, 'wb') as f:\n",
    "        for c in r.iter_content(chunk_size=CHUNK):\n",
    "            f.write(c)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.2 Extract Data from `JSON` files\n",
    "The code in this cell will iterate through the list of projects entered above (1.1). For each project the `JSON` zip file is located in the directory `jsonzip`, named PROJECT.zip. The function `namelist()` (from the zipfile package) will give all file names in a `zip` object.  The files we need are in the directory `corpusjson.` The list of names is reduced to the names in that directory that end in `.json` (this way, the directory itself is omitted). The list of files is used to parse each individual text file.\n",
    "\n",
    "Each of these files is extracted from the `zip` file and read with the command `json.loads()`, which reads the json data and transforms it into a JSON object - a sequence of names and values.\n",
    "\n",
    "This JSON object is parsed with the `objectpath` package. Four different sets of data are extracted:\n",
    "- 'f' This node contains all the lemmatization information, plus the grapheme data. Currently the code extracts only the fields Citation Form, Guide Word, Part of Speech, and Language, but other fields may be extracted in the same way.\n",
    "\n",
    "- 'ref' This node contains the reference of a single word (the word lemmatized in 'f'). Word references have the format P######.[line][no], e.g. P245342.12.1 (the first word of the twelth line). The project name is added to each word reference.\n",
    "\n",
    "- 'label' This node contains line numbers in the format 'o ii 6', plus a line reference. The line references have the format P######.[line], for instance P245342.12. Note that the line number is an abstract internal line number that does not correspond with line numbers on the tablet (breaks, columns,  and horizontal rulings also receive line numbers).\n",
    "\n",
    "- `extent`, `scope`, and `state` These nodes contain information about broken/missing lines and about horizontal rulings on the tablet (separating sections).\n",
    "\n",
    "These four sets of data are collected in the lists `lemm_l`, `ref_l`, `label_l`, and `gaps_l`.\n",
    "\n",
    "### Note\n",
    "In `objectpath` expressions, quotation marks around a name are usually not necessary: the expressions `$..cdl` and `$..'cdl'` are equivalent (all `cdl` nodes that descend from the root). In some cases, however, a name may have a special function. This is the case for `f` that (without quotation marks) refers to boolean `False`. In the `JSON` files parsed here the `f` nodes subsume all lemmatization data - this `f` needs to be in quotation marks as in\n",
    "```python\n",
    "result = tree.execute(\"$...'f'.(cf, gw, pos, lang, form)\")\n",
    "```\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "lemm_l = []  # initiate the lists that will capture the data\n",
    "ref_l = []\n",
    "label_l = []\n",
    "gaps_l = []\n",
    "for project in p:\n",
    "    file = \"jsonzip/\" + project.replace(\"/\", \"-\") + \".zip\"\n",
    "    try:\n",
    "        z = zipfile.ZipFile(file)       # create a Zipfile object\n",
    "    except:\n",
    "        print(file + \" does not exist or is not a proper ZIP file\")\n",
    "        continue\n",
    "    files = z.namelist()     # list of all the files in the ZIP\n",
    "    files = [name for name in files if \"corpusjson\" in name and name[-5:] == '.json']                                                                                                  #that holds all the P, Q, and X numbers.\n",
    "    for filename in files:                            #iterate over the file names\n",
    "        try:\n",
    "            text = z.read(filename).decode('utf-8')         #read and decode the json file of one particular text\n",
    "            data_json = json.loads(text)                # make it into a json object (essentially a dictionary)\n",
    "        except:\n",
    "            print(filename + ' is not available or not complete')\n",
    "            continue\n",
    "        tree = objectpath.Tree(data_json)\n",
    "        refs = list(tree.execute(\"$..cdl[@.node is 'l'].ref\"))\n",
    "        refs = [[ref , project] for ref in refs]    # add project name to each entry in the refs list\n",
    "        ref_l.extend(refs)\n",
    "        lemms = list(tree.execute(\"$..'f'.(cf, gw, pos, lang, form)\")) # select lemmatization fields\n",
    "        lemm_l.extend(lemms)\n",
    "        labels = list(tree.execute(\"$..cdl[@.type is 'line-start'].(ref, label)\"))\n",
    "        label_l.extend(labels)\n",
    "        gaps = list(tree.execute(\"$..cdl[@.type is 'nonx' and (@.state is 'missing' or @.state is 'ruling')].(ref, extent, scope, state)\"))\n",
    "        for item in gaps:\n",
    "            item.update({\"project\" : project})  # add project name to each entry in the gaps list\n",
    "        gaps_l.extend(gaps)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Turn the four lists into DataFrames"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "ref_df = pd.DataFrame(ref_l)\n",
    "lemm_df = pd.DataFrame(lemm_l)\n",
    "labels_df = pd.DataFrame(label_l)\n",
    "gaps_df = pd.DataFrame(gaps_l)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Provide the ref_df with proper column names."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "ref_df.columns = [\"ref\", \"project\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In `ref_df`, `labels_df`, and `gaps_df` extract P number (text ID) and line number into separate fields. These are used for matching and proper ordering later on. Note that the field `line` is integer, so that '5' will be followed by '6' (not by '51')."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "ename": "ValueError",
     "evalue": "invalid literal for int() with base 10: '3l'",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-9-49542cab5396>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m()\u001b[0m\n\u001b[0;32m      1\u001b[0m \u001b[0mdataframes\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;33m[\u001b[0m\u001b[0mlabels_df\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mref_df\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mgaps_df\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      2\u001b[0m \u001b[1;32mfor\u001b[0m \u001b[0mdataframe\u001b[0m \u001b[1;32min\u001b[0m \u001b[0mdataframes\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 3\u001b[1;33m     \u001b[0mdataframe\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;34m'line'\u001b[0m\u001b[1;33m]\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;33m[\u001b[0m\u001b[0mint\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mref\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0msplit\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m'.'\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;36m1\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;32mfor\u001b[0m \u001b[0mref\u001b[0m \u001b[1;32min\u001b[0m \u001b[0mdataframe\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;34m'ref'\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m      4\u001b[0m     \u001b[0mdataframe\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;34m\"text_id\"\u001b[0m\u001b[1;33m]\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;33m[\u001b[0m\u001b[0mref\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;36m7\u001b[0m\u001b[1;33m]\u001b[0m \u001b[1;32mfor\u001b[0m \u001b[0mref\u001b[0m \u001b[1;32min\u001b[0m \u001b[0mdataframe\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;34m\"ref\"\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m<ipython-input-9-49542cab5396>\u001b[0m in \u001b[0;36m<listcomp>\u001b[1;34m(.0)\u001b[0m\n\u001b[0;32m      1\u001b[0m \u001b[0mdataframes\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;33m[\u001b[0m\u001b[0mlabels_df\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mref_df\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mgaps_df\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      2\u001b[0m \u001b[1;32mfor\u001b[0m \u001b[0mdataframe\u001b[0m \u001b[1;32min\u001b[0m \u001b[0mdataframes\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 3\u001b[1;33m     \u001b[0mdataframe\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;34m'line'\u001b[0m\u001b[1;33m]\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;33m[\u001b[0m\u001b[0mint\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mref\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0msplit\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m'.'\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;36m1\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;32mfor\u001b[0m \u001b[0mref\u001b[0m \u001b[1;32min\u001b[0m \u001b[0mdataframe\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;34m'ref'\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m      4\u001b[0m     \u001b[0mdataframe\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;34m\"text_id\"\u001b[0m\u001b[1;33m]\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;33m[\u001b[0m\u001b[0mref\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;36m7\u001b[0m\u001b[1;33m]\u001b[0m \u001b[1;32mfor\u001b[0m \u001b[0mref\u001b[0m \u001b[1;32min\u001b[0m \u001b[0mdataframe\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;34m\"ref\"\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mValueError\u001b[0m: invalid literal for int() with base 10: '3l'"
     ]
    }
   ],
   "source": [
    "dataframes = [labels_df, ref_df, gaps_df]\n",
    "for dataframe in dataframes: \n",
    "    dataframe['line'] = [int(ref.split('.')[1]) for ref in dataframe['ref']]\n",
    "    dataframe[\"text_id\"] = [ref[:7] for ref in dataframe[\"ref\"]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "for ref in labels_df['ref']:\n",
    "    try:\n",
    "        int(ref.split[1])\n",
    "    except:\n",
    "        print(ref)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Join `ref_df` and `lemm_df` (horizontally). Since both collect data on the word level, they are of the same length. The `gaps_df`, on the other hand, collects data on missing lines, etc - and those lines are per definition not represented in `ref_df` or `lemm_df`. This dataframe is joined vertically, so that each line (or statement of missing line(s)) now has one row. Sort the rows by `text_id` and by `line` to reconstruct the proper line order."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df = ref_df.join(lemm_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Replace `NaN` (\"Not a Numer\" or missing value) with the empty string."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "df = df.fillna('')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a `lemma` field by adding up Citation Form, Guide Word, and Part of Speech. If there is no Citation Form (word has not been lemmatized - for instance because it is broken), use Form and `[NA]NA` as Guideword and Part of Speech."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "df[\"lemma\"] = df.apply(lambda x: (x[\"cf\"] + '[' + x[\"gw\"] + ']' + x[\"pos\"]) \n",
    "                            if x[\"cf\"] != '' else x['form'] + '[NA]NA', axis=1)\n",
    "#df['lemma'] = [lemma if not lemma == '[NA]NA' else '' for lemma in df['lemma'] ]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Group data with the `groupby` function (from Pandas) by `project`, `text_id`, and `line`, so that all lemmas of a single line are in one row."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "lines = df.groupby([df[\"project\"], df['text_id'], df['line']]).agg({\n",
    "        'lemma': ' '.join\n",
    "    }).reset_index()    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Add the labels by merging `labels_df` with `lines`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "new = pd.merge(lines, labels_df, how='left', left_on=['text_id', 'line'], right_on=['text_id', 'line'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "new = pd.concat([new, gaps_df], sort=False)\n",
    "new = new.sort_values(by = ['text_id', 'line'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "new"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}