speed_of_broadcasting.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the release of the NumPy paper I've had people ask me more questions about efficiency and speed within pandas, and whether broadcasting is worth the added complexity. I find it important to clarify that I don't think broadcasting is complex - it's much simplier than writing loops yourself! - it's that we just don't typically think in this way when programming, and thus many of our programming languages and toolkits don't take advantage of the benefits of [array programming](https://en.wikipedia.org/wiki/Array_programming).\n",
    "\n",
    "Let's look at a couple of examples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "# some reusable setup code\n",
    "def setup_dataframe(size=1000):\n",
    "    df=pd.DataFrame({\"data\":np.random.normal(size=size), \"mean\":np.empty(size)})\n",
    "    mean=np.mean(df[\"data\"])\n",
    "    return df, mean"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "del(df)\n",
    "del(m)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First up is building a new column in a `DataFrame` through a traditional `for` loop. Note that for each time we want to write a value into the dataframe we have to look up the existing value and subtract the mean."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "169 ms ± 2.95 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)\n"
     ]
    }
   ],
   "source": [
    "%%timeit df, mean = setup_dataframe()\n",
    "for i in range(len(df)):\n",
    "    df.iloc[i,1]=df[\"data\"].iloc[i]-mean"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next up is iteratively building the new column through a list comprehension, then assigning it in one statement to the `DataFrame`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "374 µs ± 5.32 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)\n"
     ]
    }
   ],
   "source": [
    "%%timeit df, mean = setup_dataframe()\n",
    "df[\"mean\"]=[x-mean for x in df[\"data\"]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that's pretty fast - 382 microseconds on my machine, much much better than a straight forward iteration through the `DataFrame` filling in values!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's worth for a moment considering python built in functions. In particular, the `map` function is built into the library and doesn't talk about whether it will work in parallel or not, just that it will take any iterable and apply a given function to it. Unfortunatly, that it will take any iterable is really an indication of how it will work. But let's do a bit of setup outside of the timed function to give it a fighting chance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# define our function to run\n",
    "def subtract_mean(x,mean):\n",
    "    return x-mean\n",
    "\n",
    "# some additional values we will need to be setup\n",
    "def setup_additional_dataframe(size=1000):\n",
    "    df=pd.DataFrame({\"data\":np.random.normal(size=size)})\n",
    "    mean=np.mean(df[\"data\"])\n",
    "    list_of_means=[mean for i in range(len(df))]\n",
    "    arguments=list(zip(df[\"data\"], list_of_means))\n",
    "    return (df, mean, list_of_means, arguments)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "564 µs ± 4.29 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)\n"
     ]
    }
   ],
   "source": [
    "%%timeit df, mean, list_of_means, arguments = setup_additional_dataframe()\n",
    "df[\"mean\"]=pd.DataFrame(map(subtract_mean, df[\"data\"], list_of_means))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Aha! But that's pretty close to our list comprehension. And surely, we could use python's parallization libraries to improve this, right? Even if `map()` isn't parallel, the `threading` and `multiprocessing` libraries within python must be and they have a function that looks just like `map()`! So let's setup some parameters for that exploration (it looks much the same as the previous)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "16"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import multiprocessing\n",
    "pool = multiprocessing.Pool()\n",
    "multiprocessing.cpu_count()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, before we fire this up, please keep in mind there is overhead each time we are going to create a new process on the system. There are plenty of parameters which effect this, but one is how many processes are going to work in parallel (and by not providing a parameter to `Pool()` we are saying use as many as are available) much data we want to batch into call. Let's try a few different batch sizes (which we do through the `chunksize` parameter)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "39.3 ms ± 1.74 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)\n"
     ]
    }
   ],
   "source": [
    "%%timeit df, mean, list_of_means, arguments = setup_additional_dataframe()\n",
    "df[\"mean\"]=pd.DataFrame(pool.starmap(subtract_mean, arguments, chunksize=1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4.82 ms ± 96.5 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)\n"
     ]
    }
   ],
   "source": [
    "%%timeit df, mean, list_of_means, arguments = setup_additional_dataframe()\n",
    "df[\"mean\"]=pd.DataFrame(pool.starmap(subtract_mean, arguments, chunksize=10))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.95 ms ± 208 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)\n"
     ]
    }
   ],
   "source": [
    "%%timeit df, mean, list_of_means, arguments = setup_additional_dataframe()\n",
    "df[\"mean\"]=pd.DataFrame(pool.starmap(subtract_mean, arguments, chunksize=25))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.62 ms ± 50 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)\n"
     ]
    }
   ],
   "source": [
    "%%timeit df, mean, list_of_means, arguments = setup_additional_dataframe()\n",
    "df[\"mean\"]=pd.DataFrame(pool.starmap(subtract_mean, arguments, chunksize=250))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.15 ms ± 47.4 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)\n"
     ]
    }
   ],
   "source": [
    "%%timeit df, mean, list_of_means, arguments = setup_additional_dataframe()\n",
    "df[\"mean\"]=pd.DataFrame(pool.starmap(subtract_mean, arguments, chunksize=500))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I found this interesting, I wonder what happens if we just crank the number of items in our array up, do we eventually get closed to our current fastest implementation, the list comprehension?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "286 ms ± 3.84 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n"
     ]
    }
   ],
   "source": [
    "%%timeit df, mean, list_of_means, arguments = setup_additional_dataframe(size=1000000)\n",
    "df[\"mean\"]=[x-mean for x in df[\"data\"]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "655 ms ± 12.3 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n"
     ]
    }
   ],
   "source": [
    "%%timeit df, mean, list_of_means, arguments = setup_additional_dataframe(size=1000000)\n",
    "df[\"mean\"]=pd.DataFrame(pool.starmap(subtract_mean, arguments, chunksize=10000))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, we've seen a breadth of different approaches. Let's now take a look at broadcasting though"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.66 ms ± 46.6 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)\n"
     ]
    }
   ],
   "source": [
    "%%timeit df, mean, list_of_means, arguments = setup_additional_dataframe(size=1000000)\n",
    "df[\"mean\"]=df[\"data\"]-mean"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And to compare with our initial test cases..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "215 µs ± 5.66 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)\n"
     ]
    }
   ],
   "source": [
    "%%timeit df, mean = setup_dataframe(size=1000)\n",
    "df[\"mean\"]=df[\"data\"]-mean"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, on my system right now this looks strong for broadcasting! Just over 200 microseconds for 1,000 items, and 2 and a half milliseconds for 1,000,000 values. That's much faster than our previous leader, the list comprehension with 380 microseconds, and 281 millisconds repsectively!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}