examples/storage_formats/hdf5_reporter.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Why do we need this? \n",
    "\n",
    "\n",
    "We are trying to solve several problems by proposing a new file format. Specifically: \n",
    "\n",
    "* Saving each conformation as individual file is producing too many files\n",
    "* Using pickle-based approaches (joblib) makes format python-specific and not backwards compatible; text is clumsy\n",
    "* Would be nice to save metadata, such as starting conformation, forces, or initial parameters. \n",
    "\n",
    "\n",
    "Are there alternatives?  I considered MDTraj compatible binary formats. They are PDB centered, and are usually one-file-per-trajectory. It looked hacky. \n",
    "\n",
    "### one file vs  many files  vs  several files \n",
    "\n",
    "Saving each conformation as an individual file is undesirable because it will produce too many files: filesystem check or backup on 30,000,000 files takes hours/days. \n",
    "\n",
    "Saving all trajectory as a single files is undesirable because 1. backup software will back up a new copy of the file every day as it grows; and 2. if the last write fails, the file will end up in the corrupted state and would need to be recovered. \n",
    "\n",
    "Solution is: save groups of conformations as individual files. E.g. save conformations 1-50 as one file, conformations 51-100 as a second file etc. \n",
    "\n",
    "This way, we are not risking to lose anything if the power goes out at the end. This way, we are not screwing with backup solutions, And we also have partial trajectories that can be analyzed. \n",
    "\n",
    "\n",
    "### Storage format - what did I choose? \n",
    "\n",
    "I chose the HDF5-based storage that roughly mimics the MDTraj HDF5 format. It does not have MDTraj topology because it seemed a little too complicated, and not compatible with nglview anyways. Maybe one day we will write something that fully converts it to MDTraj if necessary. \n",
    "\n",
    "\n",
    "### Overall design of the format\n",
    "\n",
    "\n",
    "I decided to separate two entitys: a simulation object and a reporter. When a simulation object is initialized, a reporter (actually, a list of reporters in case you want to use several) is passed to the simulation object. Simulation object would attempt to save several things: __init__ arguments, starting conformation, energy minimization results (TODO), serialized forces, and blocks of conformations together with time, Ek, Ep. \n",
    "\n",
    "Each time a simulation object wants to save something, it calls reporter.report(...) for each of the reporters. It passes a string indicating what is being reported, and a dictionary to save. Reporter will have to interpret this and save the data. Reporter is also keeping appropriate counts. NOTE: generic Python objects are not supported. It has to be HDF5-compatible, meaning an array of numbers/strings, or a number/string. \n",
    "\n",
    "The HDF5 reporter used here saves everything into an HDF5 file. For anything except for a conformation, it would immmediately save the data into a single HDF5 file: numpy array compatible structures would be saved as datasets, and regular types (strings, numbers) would be saved as attributes. For conformations, it would wait until a certain number of conformations is received. It will then save them all at once into an HDF5 file under groups /1, /2, /3... /50 for blocks 1,2,3...50 respectively, and save them to `blocks_1-50.h5` file\n",
    "\n",
    "\n",
    "### Multi-stage simulations or loop extrusion\n",
    "\n",
    "We frequently have simulations in which a simulation object changes. One example would be changing forces or parameters throughout the simulation. Another example would be loop extrusion simulations. \n",
    "\n",
    "In this design, a reporter object can be reused and passed to a new simulation. This would keep counter of conformations, and also save applied forces etc. again. The reporter would create a file \"applied_forces_0.h5\" the first time it receives forces, and \"applied_forces_1.h5\" the second time it receives forces from a simulation. \n",
    "\n",
    "\n",
    "### URIs to identify individual conformations\n",
    "\n",
    "Because we're saving several conformations into one file, we designed an URI format to quickly fetch a conformation by a unique identifyer. \n",
    "\n",
    "URIs are like that: `/path/to/the/trajectory/blocks_1-50.h5::42` \n",
    "\n",
    "This URI will fetch block #42 from a file blocks_1-50.h5, which contains blocks 1 through 50 including 1 and 50\n",
    "\n",
    "polymerutils.load are compatible with URIs \n",
    "\n",
    "Also, to make it easy to load both old-style filenames and new-style URIs, there is a function polychrom.polymerutils.fetch_block\n",
    "\n",
    "fetch_block will autodetermine the type of a trajectory folder. \n",
    "\n",
    "So it will fetch both `/path/to/the/trajectory/block42.dat` and  `/path/to/the/trajectory/blocks_x-y.h5::42` automatically \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import polychrom\n",
    "import numpy as np \n",
    "import warnings\n",
    "import h5py \n",
    "import glob\n",
    "from polychrom.simulation import Simulation\n",
    "import polychrom.starting_conformations\n",
    "import polychrom.forces, polychrom.forcekits\n",
    "import simtk.openmm \n",
    "import os \n",
    "import shutil\n",
    "import polychrom.polymerutils"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loading reporter and utils from a hdf5_format module \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from polychrom.hdf5_format import HDF5Reporter, list_URIs, load_URI, load_hdf5_file, save_hdf5_file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Making a simulation and passing a reporter "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:adding force harmonic_bonds 0\n",
      "INFO:root:adding force angle 1\n",
      "INFO:root:adding force polynomial_repulsive 2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Exclude neighbouring chain particles from polynomial_repulsive\n",
      "Number of exceptions: 999\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Particles loaded. Potential energy is 0.051754\n",
      "INFO:root:block    0 pos[1]=[13.9 14.3 14.0] dr=0.21 t=0.9ps kin=8.35 pot=3.84 Rg=5.650 dt=23.9fs dx=15.43pm \n",
      "INFO:root:block    1 pos[1]=[13.9 14.4 14.0] dr=0.17 t=1.1ps kin=3.68 pot=9.10 Rg=5.650 dt=20.3fs dx=8.69pm \n",
      "INFO:root:block    2 pos[1]=[13.8 14.4 14.0] dr=0.09 t=1.3ps kin=7.75 pot=5.08 Rg=5.652 dt=24.1fs dx=14.96pm \n",
      "INFO:root:block    3 pos[1]=[13.6 14.5 13.9] dr=0.14 t=1.6ps kin=6.50 pot=6.34 Rg=5.658 dt=22.0fs dx=12.52pm \n",
      "INFO:root:block    4 pos[1]=[13.5 14.5 13.9] dr=0.11 t=1.8ps kin=6.81 pot=5.99 Rg=5.663 dt=21.7fs dx=12.62pm \n",
      "INFO:root:block    5 pos[1]=[13.5 14.5 13.9] dr=0.12 t=2.0ps kin=7.02 pot=5.77 Rg=5.669 dt=21.7fs dx=12.81pm \n",
      "INFO:root:block    6 pos[1]=[13.6 14.6 13.8] dr=0.12 t=2.2ps kin=7.02 pot=5.74 Rg=5.675 dt=21.7fs dx=12.82pm \n",
      "INFO:root:block    7 pos[1]=[13.5 14.6 13.8] dr=0.12 t=2.5ps kin=7.09 pot=5.66 Rg=5.682 dt=21.7fs dx=12.88pm \n",
      "INFO:root:block    8 pos[1]=[13.4 14.5 13.8] dr=0.12 t=2.7ps kin=7.11 pot=5.62 Rg=5.690 dt=21.7fs dx=12.90pm \n",
      "INFO:root:block    9 pos[1]=[13.3 14.4 13.7] dr=0.12 t=2.9ps kin=7.29 pot=5.41 Rg=5.699 dt=21.7fs dx=13.06pm \n",
      "INFO:root:block   10 pos[1]=[13.3 14.4 13.6] dr=0.12 t=3.1ps kin=7.00 pot=5.67 Rg=5.708 dt=21.7fs dx=12.79pm \n",
      "INFO:root:block   11 pos[1]=[13.2 14.3 13.6] dr=0.12 t=3.3ps kin=7.48 pot=5.14 Rg=5.719 dt=21.7fs dx=13.23pm \n",
      "INFO:root:block   12 pos[1]=[13.1 14.3 13.5] dr=0.13 t=3.5ps kin=7.00 pot=5.60 Rg=5.731 dt=21.7fs dx=12.80pm \n",
      "INFO:root:block   13 pos[1]=[13.0 14.2 13.5] dr=0.12 t=3.7ps kin=7.46 pot=5.11 Rg=5.742 dt=21.7fs dx=13.21pm \n",
      "INFO:root:block   14 pos[1]=[12.9 14.1 13.5] dr=0.12 t=4.0ps kin=6.93 pot=5.61 Rg=5.755 dt=21.7fs dx=12.74pm \n",
      "INFO:root:block   15 pos[1]=[12.8 14.1 13.5] dr=0.12 t=4.2ps kin=7.30 pot=5.21 Rg=5.769 dt=21.7fs dx=13.07pm \n",
      "INFO:root:block   16 pos[1]=[12.7 14.0 13.5] dr=0.12 t=4.4ps kin=7.10 pot=5.40 Rg=5.783 dt=21.7fs dx=12.88pm \n",
      "INFO:root:block   17 pos[1]=[12.6 13.9 13.4] dr=0.12 t=4.6ps kin=7.11 pot=5.37 Rg=5.798 dt=21.7fs dx=12.90pm \n",
      "INFO:root:block   18 pos[1]=[12.5 13.8 13.5] dr=0.12 t=4.8ps kin=7.23 pot=5.22 Rg=5.814 dt=21.7fs dx=13.01pm \n"
     ]
    }
   ],
   "source": [
    "%rm  test/*\n",
    "data = polychrom.starting_conformations.grow_cubic(1000,30)\n",
    "\n",
    "\"\"\"\n",
    "Here we created a hdf5Reporter attached to a foler test, and we are saving 5 blocks per file \n",
    "(you should probalby use 50 here or 100. 5 is just for a showcase)\n",
    "\"\"\"\n",
    "reporter = HDF5Reporter(folder=\"test\", max_data_length=5)\n",
    "\n",
    "\n",
    "\"\"\"\n",
    "Passing a reporter to the simulation object - many reporters are possible, and more will be added in a future\n",
    "\"\"\"\n",
    "sim = Simulation(N=1000, error_tol=0.001, collision_rate=0.01, integrator =\"variableLangevin\", platform=\"CPU\", \n",
    "                reporters=[reporter])\n",
    "sim.set_data(data)\n",
    "sim.add_force(polychrom.forcekits.polymer_chains(sim))\n",
    "sim._apply_forces()\n",
    "sim.add_force(polychrom.forces.spherical_confinement(sim, density=0.1))\n",
    "\n",
    "\n",
    "for i in range(19):        \n",
    "    \"\"\"\n",
    "    Here we pass two extra records: a string and an array-like object.\n",
    "    First becomes an attr, and second becomes an HDF5 dataset\n",
    "    \"\"\"\n",
    "    sim.do_block(10, save_extras={\"eggs\": \"I don't eat green eggs and ham!!!\", \"spam\":[1,2,3]})\n",
    "\n",
    "\"\"\"\n",
    "Here we are not forgetting to dump the last set of blocks that the reporter has. \n",
    "We have to do it at the end of every simulation. \n",
    "\n",
    "I tried adding it to the destructor to make it automatic,\n",
    "but some weird interactions with garbage collection made it not very useable. \n",
    "\"\"\"\n",
    "reporter.dump_data()\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### This is a list of files created in the trajectory folder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total 820\r\n",
      "drwxrwxr-x 2 magus magus   4096 Nov  3 19:42 .\r\n",
      "drwxrwxr-x 5 magus magus   4096 Nov  3 19:37 ..\r\n",
      "-rw-rw-r-- 1 magus magus 184664 Nov  3 19:42 applied_forces_0.h5\r\n",
      "-rw-rw-r-- 1 magus magus 153097 Nov  3 19:42 blocks_0-4.h5\r\n",
      "-rw-rw-r-- 1 magus magus 153310 Nov  3 19:42 blocks_10-14.h5\r\n",
      "-rw-rw-r-- 1 magus magus 124131 Nov  3 19:42 blocks_15-18.h5\r\n",
      "-rw-rw-r-- 1 magus magus 153267 Nov  3 19:42 blocks_5-9.h5\r\n",
      "-rw-rw-r-- 1 magus magus  13829 Nov  3 19:42 forcekit_polymer_chains_0.h5\r\n",
      "-rw-rw-r-- 1 magus magus   6144 Nov  3 19:42 initArgs_0.h5\r\n",
      "-rw-rw-r-- 1 magus magus  21427 Nov  3 19:42 starting_conformation_0.h5\r\n"
     ]
    }
   ],
   "source": [
    "!ls -la test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['test/blocks_0-4.h5::0',\n",
       " 'test/blocks_0-4.h5::1',\n",
       " 'test/blocks_0-4.h5::2',\n",
       " 'test/blocks_0-4.h5::3',\n",
       " 'test/blocks_0-4.h5::4',\n",
       " 'test/blocks_5-9.h5::5',\n",
       " 'test/blocks_5-9.h5::6',\n",
       " 'test/blocks_5-9.h5::7',\n",
       " 'test/blocks_5-9.h5::8',\n",
       " 'test/blocks_5-9.h5::9',\n",
       " 'test/blocks_10-14.h5::10',\n",
       " 'test/blocks_10-14.h5::11',\n",
       " 'test/blocks_10-14.h5::12',\n",
       " 'test/blocks_10-14.h5::13',\n",
       " 'test/blocks_10-14.h5::14',\n",
       " 'test/blocks_15-18.h5::15',\n",
       " 'test/blocks_15-18.h5::16',\n",
       " 'test/blocks_15-18.h5::17',\n",
       " 'test/blocks_15-18.h5::18']"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "files = list_URIs(\"test\")\n",
    "files   #  these are the URIs for individual blocks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'pos': array([[13.85938259, 14.29982365, 13.97754854],\n",
       "        [12.91956974, 14.15092887, 13.89732582],\n",
       "        [11.91674213, 14.14851085, 14.23578607],\n",
       "        ...,\n",
       "        [15.06419512, 15.14672962, 12.94432912],\n",
       "        [15.03398223, 14.04029841, 13.0160602 ],\n",
       "        [14.04277101, 13.97448235, 13.0520061 ]]),\n",
       " 'spam': array([1, 2, 3]),\n",
       " 'block': 0,\n",
       " 'eggs': \"I don't eat green eggs and ham!!!\",\n",
       " 'kineticEnergy': 8.352826107894396,\n",
       " 'potentialEnergy': 3.8424990462960795,\n",
       " 'time': 0.8790092152610536}"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\"\"\"\n",
    "Loading the entire blosk by URI, with position and other information\n",
    "for that, use polychrom.hdf5_format.load_URI\n",
    "\n",
    "Note how our custom-added eggs and spam appear below. \n",
    "\n",
    "\"\"\"\n",
    "load_URI(files[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[13.85938259, 14.29982365, 13.97754854],\n",
       "       [12.91956974, 14.15092887, 13.89732582],\n",
       "       [11.91674213, 14.14851085, 14.23578607],\n",
       "       ...,\n",
       "       [15.06419512, 15.14672962, 12.94432912],\n",
       "       [15.03398223, 14.04029841, 13.0160602 ],\n",
       "       [14.04277101, 13.97448235, 13.0520061 ]])"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\"\"\"\n",
    "It is backwards compatible with polymerutils.load as well, and it gives you just the XYZ\n",
    "\"\"\"\n",
    "polychrom.polymerutils.load(files[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[13.77670386, 14.44293733, 13.96202581],\n",
       "       [12.7067856 , 14.20429917, 13.90140179],\n",
       "       [11.75147301, 14.17780749, 14.27290361],\n",
       "       ...,\n",
       "       [15.05743924, 15.05442324, 12.92902711],\n",
       "       [15.05150974, 14.19658634, 13.04518545],\n",
       "       [14.08012849, 13.96889462, 13.08442326]])"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\"\"\"\n",
    "There is also a universal function \"fetch_block\"\n",
    "It can fetch both old-style filenames and new-style URIs just by block ID\n",
    "\"\"\"\n",
    "\n",
    "polychrom.polymerutils.fetch_block(\"test\",2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'pos': array([[13.77670386, 14.44293733, 13.96202581],\n",
       "        [12.7067856 , 14.20429917, 13.90140179],\n",
       "        [11.75147301, 14.17780749, 14.27290361],\n",
       "        ...,\n",
       "        [15.05743924, 15.05442324, 12.92902711],\n",
       "        [15.05150974, 14.19658634, 13.04518545],\n",
       "        [14.08012849, 13.96889462, 13.08442326]]),\n",
       " 'spam': array([1, 2, 3]),\n",
       " 'block': 2,\n",
       " 'eggs': \"I don't eat green eggs and ham!!!\",\n",
       " 'kineticEnergy': 7.7540847730028135,\n",
       " 'potentialEnergy': 5.083867047820113,\n",
       " 'time': 1.3487397978843496}"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\"\"\"\n",
    "By default it fetches XYZ only, but can do full output\n",
    "(of course in the old-style filenames there is no full output so default is False)\n",
    "\"\"\"\n",
    "\n",
    "polychrom.polymerutils.fetch_block(\"test\",2, full_output=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'GPU': '0',\n",
       " 'N': 1000,\n",
       " 'PBCbox': False,\n",
       " 'collision_rate': 0.01,\n",
       " 'error_tol': 0.001,\n",
       " 'integrator': 'variableLangevin',\n",
       " 'length_scale': 1.0,\n",
       " 'mass': 100,\n",
       " 'max_Ek': 10,\n",
       " 'platform': 'CPU',\n",
       " 'precision': 'mixed',\n",
       " 'temperature': 300,\n",
       " 'verbose': False}"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\"\"\"\n",
    "Finally, loading the saved file with initial conformations. \n",
    "\"\"\"\n",
    "load_hdf5_file(\"test/initArgs_0.h5\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "simple things are saved as attrs\n",
      "('GPU', '0')\n",
      "('N', 1000)\n",
      "('PBCbox', False)\n",
      "('collision_rate', 0.01)\n",
      "('error_tol', 0.001)\n",
      "('integrator', 'variableLangevin')\n",
      "('length_scale', 1.0)\n",
      "('mass', 100)\n",
      "('max_Ek', 10)\n",
      "('platform', 'CPU')\n",
      "('precision', 'mixed')\n",
      "('temperature', 300)\n",
      "('verbose', False)\n",
      "\n",
      " groups of the data files are: \n",
      "[('15', <HDF5 group \"/15\" (2 members)>), ('16', <HDF5 group \"/16\" (2 members)>), ('17', <HDF5 group \"/17\" (2 members)>), ('18', <HDF5 group \"/18\" (2 members)>)]\n",
      "\n",
      " looking at block 15 datasets\n",
      "[('pos', <HDF5 dataset \"pos\": shape (1000, 3), type \"<f8\">), ('spam', <HDF5 dataset \"spam\": shape (3,), type \"<i8\">)]\n",
      "\n",
      " looking at block 15 attrs\n",
      "[('block', 15), ('eggs', \"I don't eat green eggs and ham!!!\"), ('kineticEnergy', 7.301050292099044), ('potentialEnergy', 5.21200792658476), ('time', 4.182637056604648)]\n",
      "Note that blocks in simulation and in a reporter are syncronized for a simple simulation when you're saving every block starting right away\n"
     ]
    }
   ],
   "source": [
    "\"\"\"\n",
    "And how it actually looks in HDF5\n",
    "\"\"\"\n",
    "import h5py \n",
    "\n",
    "print(\"simple things are saved as attrs\")\n",
    "\n",
    "for i in h5py.File(\"test/initArgs_0.h5\").attrs.items():\n",
    "    print(i)\n",
    "    \n",
    "    \n",
    "    \n",
    "myfile = h5py.File(\"test/blocks_15-18.h5\",'r') \n",
    "\n",
    "print(\"\\n groups of the data files are: \")\n",
    "print(list(myfile.items()))\n",
    "\n",
    "\n",
    "print(\"\\n looking at block 15 datasets\")\n",
    "print(list(myfile[\"15\"].items()))\n",
    "\n",
    "\n",
    "print(\"\\n looking at block 15 attrs\")\n",
    "print(list(myfile[\"15\"].attrs.items()))\n",
    "\n",
    "print(\"Note that blocks in simulation and in a reporter are syncronized for a simple simulation \"\n",
    "      \"when you're saving every block starting right away\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'angles': array([[  0,   1,   2],\n",
       "        [  1,   2,   3],\n",
       "        [  2,   3,   4],\n",
       "        ...,\n",
       "        [995, 996, 997],\n",
       "        [996, 997, 998],\n",
       "        [997, 998, 999]]), 'bonds': array([[  0,   1],\n",
       "        [  1,   2],\n",
       "        [  2,   3],\n",
       "        ...,\n",
       "        [996, 997],\n",
       "        [997, 998],\n",
       "        [998, 999]]), 'chains': array([[   0, 1000,    0]])}"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "load_hdf5_file(\"test/forcekit_polymer_chains_0.h5\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'bacon': array([1, 2, 3, 4, 5]), 'a': 'eggs', 'b': 'spam'}"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#  Now we are just saving an array to an HDF5 file with a save_hdf5_file method \n",
    "save_hdf5_file(\"testfile.h5\",{\"a\":\"eggs\", \"b\":\"spam\", \"bacon\":[1,2,3,4,5]}, mode=\"w\")\n",
    "load_hdf5_file(\"testfile.h5\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## and this is how you would continue a simulation "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(18, {'pos': array([[12.51086061, 13.83818873, 13.45350043],\n",
       "         [12.78139492, 14.50096674, 14.28880353],\n",
       "         [11.79955253, 14.67199473, 14.5766987 ],\n",
       "         ...,\n",
       "         [15.36093788, 14.99209487, 12.99663787],\n",
       "         [15.36078007, 14.3093237 , 13.89419497],\n",
       "         [14.49182306, 14.03461682, 13.49497998]]),\n",
       "  'spam': array([1, 2, 3]),\n",
       "  'block': 18,\n",
       "  'eggs': \"I don't eat green eggs and ham!!!\",\n",
       "  'kineticEnergy': 7.233734407429165,\n",
       "  'potentialEnergy': 5.216835906387824,\n",
       "  'time': 4.832271791303491})"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\n",
    "rep = HDF5Reporter(folder=\"test\",  overwrite=False, check_exists=False)\n",
    "ind, data = rep.continue_trajectory()\n",
    "ind, data  # look at what is returned"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now you would run something like that, and block numbering  will be consistent\n",
    "\n",
    "`\n",
    "sim = Simulation(..., reporters=[rep]) \n",
    "sim.set_data(data[\"pos\"])`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}