06-ipyparallel-tutorial-summary.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "toc": "true"
   },
   "source": [
    "# Table of Contents\n",
    " <p><div class=\"lev1\"><a href=\"#summary\"><span class=\"toc-item-num\">1&nbsp;&nbsp;</span>summary</a></div><div class=\"lev2\"><a href=\"#Direct-view\"><span class=\"toc-item-num\">1.1&nbsp;&nbsp;</span>Direct view</a></div><div class=\"lev2\"><a href=\"#LoadBalancedView\"><span class=\"toc-item-num\">1.2&nbsp;&nbsp;</span>LoadBalancedView</a></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# summary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-04-14T13:26:29.854925",
     "start_time": "2017-04-14T15:26:29.512468+02:00"
    },
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from __future__ import print_function, division # legacy support\n",
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-04-14T13:26:30.493330",
     "start_time": "2017-04-14T15:26:30.354037+02:00"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Direct view"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-04-05T22:09:54.545784",
     "start_time": "2017-04-06T00:09:54.538633+02:00"
    }
   },
   "source": [
    "Direct view is a simple way to perform parallel calculations in ipython, however this has some drawbacks:\n",
    "\n",
    "Advantages:\n",
    "- Easy\n",
    "\n",
    "Disadvantages:\n",
    "- While performing the parallel calculations, the notebook kernel is busy. This means that you can change whatever you want in the notebook, but you have to wait until the parallel calculation has finished before you can evaluate other cells. (So for example during the parallel calculation you want to make a figure, however the evaluation of the figure code will only be done after finishing the parallel calculation).\n",
    "- For the direct view you have to 'scatter' your scenarios before you can start the simulations. This can be a drawback in some cases, for example: You have 10 cores and 1000 scenarios you want to calculate. So when you use the scatter command, you will give 100 scenarios to every core. Seems reasonable, right? However, imagine that some scenarios are really fast and others are really slow... this can have as an impact that some cores will be finished, while the others still have to do a lot of work.\n",
    "- Less options\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-04-14T13:26:31.321141",
     "start_time": "2017-04-14T15:26:31.297189+02:00"
    },
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0, 1, 2, 3]\n"
     ]
    }
   ],
   "source": [
    "# Import client for parallel calculation\n",
    "from ipyparallel import Client\n",
    "# Make client using the cores started with --profile='nbserver'\n",
    "c = Client(profile='nbserver')\n",
    "# Make directView of all clients, used to parse data/objects to cores\n",
    "dview = c[:]\n",
    "# Print all clients\n",
    "print(c.ids)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-04-14T13:26:31.779604",
     "start_time": "2017-04-14T15:26:31.715519+02:00"
    },
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%%px --local\n",
    "# Execute on all cores but also for serial calculations, has to be loaded on the FIRST line\n",
    "# This code should only be runned when using self defined classes \n",
    "# Load dill, More general pickling class\n",
    "import dill\n",
    "# fallback to pickle instead of cPickle, so that dill can take over\n",
    "import pickle\n",
    "from ipykernel import serialize\n",
    "serialize.pickle = pickle"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-04-14T13:26:31.977658",
     "start_time": "2017-04-14T15:26:31.936368+02:00"
    },
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "importing math on engine(s)\n"
     ]
    }
   ],
   "source": [
    "# Import on all cores\n",
    "with c[:].sync_imports():\n",
    "    import math"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "There are two options to load your functions and other variables on the different cores\n",
    "1. Use \"%%px --local\" to load everything which is defined in the cells both in the local and the core environments\n",
    "2. Use \"dview['data'] = data\" to parse local variables to all the different cores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-04-14T13:26:32.446973",
     "start_time": "2017-04-14T15:26:32.399067+02:00"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%%px --local\n",
    "def get_sum(pararray = None):\n",
    "    return sum(pararray)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-04-14T13:26:32.659398",
     "start_time": "2017-04-14T15:26:32.653542+02:00"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "samples = 5000\n",
    "scenarios = np.random.random([samples,3])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-04-14T13:26:32.934413",
     "start_time": "2017-04-14T15:26:32.930407+02:00"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Example for using 4 parallel cores\n",
    "def mul(a, b):\n",
    "    return a * b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-04-14T13:26:33.245061",
     "start_time": "2017-04-14T15:26:33.233482+02:00"
    },
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "res = dview.map(mul, [5, 6, 7, 8], [8, 9, 10, 11])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-04-14T13:26:33.556912",
     "start_time": "2017-04-14T15:26:33.550254+02:00"
    },
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[40, 54, 70, 88]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "res.result()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-04-14T13:26:33.887546",
     "start_time": "2017-04-14T15:26:33.870771+02:00"
    },
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "res = dview.apply(mul, 5, 6)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-04-14T13:26:34.217681",
     "start_time": "2017-04-14T15:26:34.209903+02:00"
    },
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[30, 30, 30, 30]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "res.get()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In a DirectView interface, we can either use Blocking (synchronous) execution, in which all results must finish computing \n",
    "before any results are recorded, or non-blocking (asynchronous) execution, where we receive results as they finish.\n",
    "\n",
    "- Get results directly without using the .result option\n",
    "    <pre><code>\n",
    "    out = dview.map_sync(get_sum, scenarios)\n",
    "    </code></pre>\n",
    "- Create objects for which you have to use the .result option\n",
    "    <pre><code>\n",
    "    out = dview.map_async(get_sum, scenarios)\n",
    "    </code></pre>\n",
    "\n",
    "However only the number of runned scenarios will equal the number of processors,\n",
    "therefore we have to \"scatter\" the scenarios over the different cores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-04-14T13:26:34.883740",
     "start_time": "2017-04-14T15:26:34.851670+02:00"
    },
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AsyncResult: scatter>"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dview.scatter('scenarios',scenarios)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-04-14T13:26:35.213977",
     "start_time": "2017-04-14T15:26:35.169238+02:00"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%px out = [get_sum(scen) for scen in scenarios]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-04-14T13:26:35.536332",
     "start_time": "2017-04-14T15:26:35.489056+02:00"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "out = dview.gather('out')\n",
    "out = out.get()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-04-14T13:26:35.800963",
     "start_time": "2017-04-14T15:26:35.792556+02:00"
    },
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "df_out = pd.DataFrame(np.hstack([scenarios, np.atleast_2d(out).T]), \n",
    "             columns=['par1','par2','par3','sum'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-04-14T13:26:36.124669",
     "start_time": "2017-04-14T15:26:36.110954+02:00"
    },
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>par1</th>\n",
       "      <th>par2</th>\n",
       "      <th>par3</th>\n",
       "      <th>sum</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.905388</td>\n",
       "      <td>0.433759</td>\n",
       "      <td>0.017719</td>\n",
       "      <td>1.356865</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.883372</td>\n",
       "      <td>0.157088</td>\n",
       "      <td>0.026599</td>\n",
       "      <td>1.067060</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.807046</td>\n",
       "      <td>0.814463</td>\n",
       "      <td>0.527293</td>\n",
       "      <td>2.148802</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.408789</td>\n",
       "      <td>0.909581</td>\n",
       "      <td>0.504522</td>\n",
       "      <td>1.822892</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.772068</td>\n",
       "      <td>0.009461</td>\n",
       "      <td>0.342701</td>\n",
       "      <td>1.124229</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       par1      par2      par3       sum\n",
       "0  0.905388  0.433759  0.017719  1.356865\n",
       "1  0.883372  0.157088  0.026599  1.067060\n",
       "2  0.807046  0.814463  0.527293  2.148802\n",
       "3  0.408789  0.909581  0.504522  1.822892\n",
       "4  0.772068  0.009461  0.342701  1.124229"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_out.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## LoadBalancedView"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "LoadBalancedView is a more advanced way of performing parallel calculations\n",
    "\n",
    "Advantages:\n",
    "- Parallel calculations are not directly connected to your notebook, this means that during the parallel calculations your notebook is as productive as usual. For very fast functions the difference will be non-existing (or even the other way around), but when load increases you will see the advantage of the LoadBalancedView.\n",
    "- The addressed cores are used in the most efficient way, because the assignment of the work to the different cores is done dynamically during the calculation. Thereby avoiding unbalanced workload of the different cores.\n",
    "- More options\n",
    "- All the stuff which is printed during each simulation is not shown, keeping your notebook tidy and clean ^^\n",
    "\n",
    "Disadvantages:\n",
    "- No printing during calculation, errors are not popping up. So if something goes wrong with simulations, this will not directly show you an error. The advantage is that the next calculation is started automatically, so no loss of time. The disadvantage is that when all simulations are failing, this only pops up when you ask for the results."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-04-05T22:39:16.064724",
     "start_time": "2017-04-06T00:39:16.042898+02:00"
    }
   },
   "source": [
    "In a loadBalancedView interface, we can either use Blocking (synchronous) execution, in which all results must finish computing \n",
    "before any results are recorded, or non-blocking (asynchronous) execution, where we receive results as they finish.\n",
    "\n",
    "- Get results directly without using the .result option\n",
    "    <pre><code>\n",
    "    out = lview.map_sync(get_sum, scenarios)\n",
    "    </code></pre>\n",
    "- Create objects for which you have to use the .result option\n",
    "    <pre><code>\n",
    "    out = lview.map_async(get_sum, scenarios)\n",
    "    </code></pre>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-04-14T13:26:39.734035",
     "start_time": "2017-04-14T15:26:39.727765+02:00"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "lview = c.load_balanced_view()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "chunksize is of high importance in this case, because the calculations take very little time. For low size of chunks (e.g. the number of elements in the scenarios that are calculated at once), the overhead is high due to communication between scheduler and engines"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-04-14T13:26:42.833792",
     "start_time": "2017-04-14T15:26:40.434132+02:00"
    },
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " 500/500 tasks finished after    2 s\n",
      "done\n",
      "the total calculation time is: 2.36536\n"
     ]
    }
   ],
   "source": [
    "out = lview.map_async(get_sum, scenarios, chunksize=10)\n",
    "out.wait_interactive()\n",
    "print('the total calculation time is: %g'%out.wall_time)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-04-14T13:26:43.898448",
     "start_time": "2017-04-14T15:26:43.582197+02:00"
    },
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  50/50 tasks finished after    0 s\n",
      "done\n",
      "the total calculation time is: 0.305344\n"
     ]
    }
   ],
   "source": [
    "out = lview.map_async(get_sum, scenarios, chunksize=100)\n",
    "out.wait_interactive()\n",
    "print('the total calculation time is: %g'%out.wall_time)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-04-14T13:26:44.506158",
     "start_time": "2017-04-14T15:26:44.500315+02:00"
    },
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "df_out = pd.DataFrame(np.hstack([scenarios, np.atleast_2d(out.result()).T]), \n",
    "             columns=['par1','par2','par3','sum'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-04-14T13:26:46.333160",
     "start_time": "2017-04-14T15:26:46.318311+02:00"
    },
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>par1</th>\n",
       "      <th>par2</th>\n",
       "      <th>par3</th>\n",
       "      <th>sum</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.905388</td>\n",
       "      <td>0.433759</td>\n",
       "      <td>0.017719</td>\n",
       "      <td>1.356865</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.883372</td>\n",
       "      <td>0.157088</td>\n",
       "      <td>0.026599</td>\n",
       "      <td>1.067060</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.807046</td>\n",
       "      <td>0.814463</td>\n",
       "      <td>0.527293</td>\n",
       "      <td>2.148802</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.408789</td>\n",
       "      <td>0.909581</td>\n",
       "      <td>0.504522</td>\n",
       "      <td>1.822892</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.772068</td>\n",
       "      <td>0.009461</td>\n",
       "      <td>0.342701</td>\n",
       "      <td>1.124229</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       par1      par2      par3       sum\n",
       "0  0.905388  0.433759  0.017719  1.356865\n",
       "1  0.883372  0.157088  0.026599  1.067060\n",
       "2  0.807046  0.814463  0.527293  2.148802\n",
       "3  0.408789  0.909581  0.504522  1.822892\n",
       "4  0.772068  0.009461  0.342701  1.124229"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_out.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "hide_input": false,
  "kernelspec": {
   "display_name": "Python [conda env:ipyparallelseminar]",
   "language": "python",
   "name": "conda-env-ipyparallelseminar-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  },
  "toc": {
   "toc_cell": true,
   "toc_number_sections": true,
   "toc_threshold": 6,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}