Concurrency.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Concurrency in Python"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### GIL\n",
    "In Python's main interpreter implementation, the one we all use (CPython), there is a __GIL__ or Global Interpreter Lock that ensure only one thread may execute Python bytecode at a time. This is because CPython is not thread safe.\n",
    "\n",
    "As such, we really need to look at multiprocessing when we're trying to use multiple cores effectively in pure Python."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Data structures and pickle\n",
    "To communicate between processes, both concurrent.futures (the main implementation of multiprocessing and multiprocessing in the Standard Library) and other implementation rely on pickle, which allows saving Python objects into files.\n",
    "\n",
    "However, not all objects can be _'pickled'_. Namely, you can't pickle a lot of things including:\n",
    "- Lambdas\n",
    "- Default dictionaries (defaultdict)\n",
    "- Custom objects\n",
    "\n",
    "Please refer to [Pickle's documentation](https://docs.python.org/3/library/pickle.html#what-can-be-pickled-and-unpickled) for more information on what can and can't be pickled."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Single-threaded"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "run_control": {
     "marked": false
    }
   },
   "outputs": [],
   "source": [
    "# Setup\n",
    "import random\n",
    "COUNT = pow(10, 6)\n",
    "random_values = {random.random(): [] for _ in range(100)}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Execution\n",
    "for random_value, inner_values in random_values.items():\n",
    "    for i in range(COUNT):\n",
    "        inner_values.append(random_value * random.random())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import gc\n",
    "del random_values\n",
    "gc.collect()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Multiprocess"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "### Standard Library\n",
    "Won't work on Windows or mac. Only Linux."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "# Setup\n",
    "random_values_parallel = {random.random() for _ in range(100)}\n",
    "\n",
    "def inner_loop(random_value):\n",
    "    inner_values = []\n",
    "    for i in range(COUNT):\n",
    "        inner_values.append(random_value * random.random())\n",
    "    return random_value,  inner_values\n",
    "\n",
    "from concurrent.futures import ProcessPoolExecutor\n",
    "pool = ProcessPoolExecutor()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "random_values_parallel_filled = dict(pool.map(inner_loop, random_values_parallel))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "del random_values_parallel_filled\n",
    "gc.collect()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Loky"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#!pip3 install loky"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Setup\n",
    "from loky import get_reusable_executor\n",
    "pool = get_reusable_executor()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "random_values_parallel_filled = dict(pool.map(inner_loop, random_values_parallel))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "del random_values_parallel_filled\n",
    "gc.collect()"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}