dataset.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>Loading and Inspecting the Full Dataset</h2>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import cPickle as pickle"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are loading all the `pickle` files that reside in the `Full/data/` directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Successfully loaded dataset file: ./Full/data/25.p\n",
      "Successfully loaded dataset file: ./Full/data/9.p\n",
      "Successfully loaded dataset file: ./Full/data/39.p\n",
      "Successfully loaded dataset file: ./Full/data/29.p\n",
      "Successfully loaded dataset file: ./Full/data/28.p\n",
      "Successfully loaded dataset file: ./Full/data/38.p\n",
      "Successfully loaded dataset file: ./Full/data/34.p\n",
      "Successfully loaded dataset file: ./Full/data/8.p\n",
      "Successfully loaded dataset file: ./Full/data/20.p\n",
      "Successfully loaded dataset file: ./Full/data/44.p\n",
      "Successfully loaded dataset file: ./Full/data/37.p\n",
      "Successfully loaded dataset file: ./Full/data/23.p\n",
      "Successfully loaded dataset file: ./Full/data/40.p\n",
      "Successfully loaded dataset file: ./Full/data/42.p\n",
      "Successfully loaded dataset file: ./Full/data/32.p\n",
      "Successfully loaded dataset file: ./Full/data/4.p\n",
      "Successfully loaded dataset file: ./Full/data/30.p\n",
      "Successfully loaded dataset file: ./Full/data/11.p\n",
      "Successfully loaded dataset file: ./Full/data/35.p\n",
      "Successfully loaded dataset file: ./Full/data/33.p\n",
      "Successfully loaded dataset file: ./Full/data/36.p\n",
      "Successfully loaded dataset file: ./Full/data/22.p\n",
      "Successfully loaded dataset file: ./Full/data/5.p\n",
      "Successfully loaded dataset file: ./Full/data/3.p\n",
      "Successfully loaded dataset file: ./Full/data/21.p\n",
      "Successfully loaded dataset file: ./Full/data/26.p\n",
      "Successfully loaded dataset file: ./Full/data/14.p\n",
      "Successfully loaded dataset file: ./Full/data/43.p\n",
      "Successfully loaded dataset file: ./Full/data/7.p\n",
      "Successfully loaded dataset file: ./Full/data/41.p\n",
      "Successfully loaded dataset file: ./Full/data/15.p\n",
      "Successfully loaded dataset file: ./Full/data/13.p\n",
      "Successfully loaded dataset file: ./Full/data/18.p\n",
      "Successfully loaded dataset file: ./Full/data/12.p\n",
      "Successfully loaded dataset file: ./Full/data/31.p\n",
      "Successfully loaded dataset file: ./Full/data/19.p\n",
      "Successfully loaded dataset file: ./Full/data/1.p\n",
      "Successfully loaded dataset file: ./Full/data/2.p\n",
      "Successfully loaded dataset file: ./Full/data/6.p\n",
      "Successfully loaded dataset file: ./Full/data/17.p\n",
      "Successfully loaded dataset file: ./Full/data/16.p\n",
      "Successfully loaded dataset file: ./Full/data/27.p\n",
      "Successfully loaded dataset file: ./Full/data/10.p\n",
      "Successfully loaded dataset file: ./Full/data/24.p\n",
      "Total items that have been loaded in the dataset: 864862\n"
     ]
    }
   ],
   "source": [
    "dataset_dir = './Full/data/'\n",
    "dataset = {'item': [], 'original_summary': [], 'summary_with_surf_forms': [], 'triples': []}\n",
    "\n",
    "for file in os.listdir(dataset_dir):\n",
    "    if file.endswith(\".p\"):\n",
    "        tempDatasetFileLocation = os.path.join(dataset_dir, file)\n",
    "        with open(tempDatasetFileLocation, 'rb') as tempDatasetFile:\n",
    "            tempDataset = pickle.load(tempDatasetFile)\n",
    "            dataset['item'].extend(tempDataset['item'])\n",
    "            dataset['original_summary'].extend(tempDataset['original_summary'])\n",
    "            dataset['summary_with_surf_forms'].extend(tempDataset['summary_with_surf_forms'])\n",
    "            dataset['triples'].extend(tempDataset['triples'])\n",
    "            print('Successfully loaded dataset file: %s' % (tempDatasetFileLocation))\n",
    "assert(len(dataset['item']) == len(dataset['original_summary']))\n",
    "assert(len(dataset['item']) == len(dataset['triples']))\n",
    "assert(len(dataset['item']) == len(dataset['summary_with_surf_forms']))\n",
    "print('Total items that have been loaded in the dataset: %d' % (len(dataset['item'])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The selected dataset is loaded as a dictionary of lists. The lists are aligned to each other. For example, in order to print all entries about the item in the $254$th position, we run the following:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "original_summary\n",
      "The cisterna magna (or cerebellomedullaris cistern) is one of three principal openings in the subarachnoid space between the arachnoid and pia mater layers of the meninges surrounding the brain. The openings are collectively referred to as cisterns.\n",
      "item\n",
      "http://dbpedia.org/resource/Cisterna_magna\n",
      "triples\n",
      "<item> http://dbpedia.org/ontology/brainInfoNumber 0\n",
      "<item> http://dbpedia.org/ontology/grayPage 0\n",
      "<item> http://dbpedia.org/ontology/graySubject 0\n",
      "summary_with_surf_forms\n",
      "The <item> ( or cerebellomedullaris #surFormToken823661 ) is one of three principal openings in the #surFormToken490331 between the #surFormToken953899 and #surFormToken416940 layers of the #surFormToken177334 surrounding the #surFormToken35668 . The openings are collectively referred to as #surFormToken1359894 .\n"
     ]
    }
   ],
   "source": [
    "index = 253\n",
    "for key in dataset:\n",
    "    print(key)\n",
    "    # Print the aligned triples properly.\n",
    "    if key == 'triples':\n",
    "        for triple in dataset[key][index]:\n",
    "            print triple\n",
    "    else:\n",
    "        print(dataset[key][index])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The keys of the dictionary are described below:\n",
    "* `item`: refers to the main entity of each Wikipedia summary.\n",
    "* `original_summary`: refers to the original Wikipedia summary, prior to any pre-processing.\n",
    "* `triples`: refers to the list of triples that associated with the Wikipedia summary.\n",
    "* `summary_with_surf_forms`: refers to the Wikipedia summary after the realisations of the identified entities have been replaced by their corresponding *surface form tuple* tokens. The realisations of the entities are identified with [DBpedia Spotlight](https://github.com/dbpedia-spotlight/dbpedia-spotlight-model).\n",
    "\n",
    "Any reference to the main entity both in the `triples` and the `summary_with_surf_forms` is replaced by the `<item>` token.\n",
    "\n",
    "Tokens, such as `#surFormToken101` and `#surFormToken103` are used as placeholders of the entities' surface form tuples."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>Loading the Supporting Dictionaries</h2>\n",
    "\n",
    "We are loading below the supporting dictionaries that reside in `Full/utils/`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Successfully loaded the surface form tokens file at: ./Full/utils/Surface-Form-Tokens.p\n"
     ]
    }
   ],
   "source": [
    "surf_forms_tokens_location = './Full/utils/Surface-Form-Tokens.p'\n",
    "with open(surf_forms_tokens_location, 'rb') as f:\n",
    "    surf_forms_tokens = pickle.load(f)\n",
    "    print('Successfully loaded the surface form tokens file at: %s' % surf_forms_tokens_location)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An example of the structure of the dictionary that maps surface form tuples to their corresponding tokens in `summary_with_surf_forms` (e.g. `#surFormToken103`) is presented below.\n",
    "```python\n",
    "surf_forms_tokens = {(u'http://dbpedia.org/resource/Science_fiction', u'science fiction'): '#surFormToken5740',\n",
    "                     (u'http://dbpedia.org/resource/Science_fiction', u'sci-fi'): '#surFormToken22979',\n",
    "                     (u'http://dbpedia.org/resource/Science_fiction', u'science-fiction'): '#surFormToken109715',\n",
    "                     (u'http://dbpedia.org/resource/United_States', u'American'): '#surFormToken212',\n",
    "                     ...,\n",
    "                     (u'http://dbpedia.org/resource/United_States', u'U.S.'): '#surFormToken1416'}\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Successfully loaded surface form counts file at: ./Full/utils/Surface-Forms-Counts.p\n"
     ]
    }
   ],
   "source": [
    "surf_form_counts_location = './Full/utils/Surface-Forms-Counts.p'\n",
    "with open(surf_form_counts_location, 'rb') as f:\n",
    "    surf_form_counts = pickle.load(f)\n",
    "    print('Successfully loaded surface form counts file at: %s' % surf_form_counts_location)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An example of the structure of the dictionary that logs the frequency with which realisations have been associated with entity URIs in the Full dataset is displayed below:\n",
    "```python\n",
    "surf_form_counts = {'http://dbpedia.org/resource/Albert_Einstein': {'Albert Einstein': 142, 'Einstein': 108},\n",
    "                    'http://dbpedia.org/resource/Actor': {'actor': 21638, 'artists': 16688}, \n",
    "                    ...}\n",
    "```\n",
    "According to the above example, for each entity $k_d \\in K$ (e.g. `dbr:Albert_Einstein`), we get a dictionary that maps all its relevant realisations in the text $g_{1 \\ldots R}^{k_d}$ (e.g. \"Albert Einstein\") to their corresponding frequency of occurrence $z_{1 \\ldots R}^{k_d}$ (e.g. the number 142)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3>Loading the English Labels and Instance Types Dictionaries</h3>\n",
    "\n",
    "Please be advised that a system with at least 24 GB of memory is required in order for the dictionaries below to be loaded together (along with the above dataset files). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Successfully loaded the English labels file at: ./Full/utils/Labels.p\n"
     ]
    }
   ],
   "source": [
    "labels_location = './Full/utils/Labels.p'\n",
    "with open(labels_location, 'rb') as f:\n",
    "    entity2label = pickle.load(f)\n",
    "    print('Successfully loaded the English labels file at: %s' % labels_location)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An example of the structure of the dictionary that maps entities to a single label (realisation) is presented below:\n",
    "```python\n",
    "entity2label = {'http://dbpedia.org/resource/Albert_Einstein': [u'Albert Einstein'],\n",
    "                'http://dbpedia.org/resource/John_Galt': [u'John Galt'],\n",
    "                ...}\n",
    "```\n",
    "\n",
    "The entities' labels are provided by DBpedia at: [http://downloads.dbpedia.org/2016-10/core-i18n/en/labels_en.ttl.bz2](http://downloads.dbpedia.org/2016-10/core-i18n/en/labels_en.ttl.bz2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Successfully loaded the instance types file at: ./Full/utils/Instance-Types.p\n"
     ]
    }
   ],
   "source": [
    "instance_types_location = './Full/utils/Instance-Types.p'\n",
    "with open(instance_types_location, 'rb') as f:\n",
    "    entity2type = pickle.load(f)\n",
    "    print('Successfully loaded the instance types file at: %s' % instance_types_location)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An example of the structure of the dictionary that maps entities to their corresponding instance type is presented below:\n",
    "```python\n",
    "entity2label = {'http://dbpedia.org/resource/Wetter_(Ruhr)': u'http://dbpedia.org/ontology/Town'\n",
    "                'http://dbpedia.org/resource/London': u'http://dbpedia.org/ontology/Settlement',\n",
    "                'http://dbpedia.org/resource/Lionel_Messi': u'http://dbpedia.org/ontology/SoccerPlayer',\n",
    "                ...}\n",
    "```\n",
    "\n",
    "The instance types of the entities are provided by DBpedia at: [http://downloads.dbpedia.org/2016-10/core-i18n/en/instance_types_en.ttl.bz2](http://downloads.dbpedia.org/2016-10/core-i18n/en/instance_types_en.ttl.bz2)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.18"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}