memory_profiler.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Profile memory usage\n",
    "We notice that our data files are large and pandas dataframes approach the limit of what can be handled on a 16GiB RAM computer when reading in a whole year's worth of trip data.\n",
    "This notebook simply profiles memory usage and improvements possible by\n",
    "* Using smaller `numeric` types\n",
    "* Using `categorical` type instead of `object` (strings)\n",
    "* Using `datetime`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "from tabulate import tabulate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Initial Memory Usage\n",
    "How big is this 4GiB (disk) data when we load it in as a pandas dataframe? There will be size inflation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "df = pd.read_csv(\"data/NY_2019.csv\")\n",
    "df.drop(\"Unnamed: 0\", axis=1, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# Here's what the data looks like\n",
    "# Surprising...Why does `startstationid` have a decimal point?\n",
    "df.sample(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# initial usage, no optimization. see column dtypes\n",
    "df.info(memory_usage=\"deep\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# Total Memory Usage GiB (Note that this is larger than the space the data occupies on disk: ~4GiB)\n",
    "start_memory = df.memory_usage(index=False, deep=True).sum() / (2 ** 30)\n",
    "print(f\"{round(start_memory, 3)} GiB\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Use smaller numeric types\n",
    "Let's take a look at how big a number we can store in each integer. Then, we can decide what type of int to use for each numeric column"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# Max integer values (we have no negative numbers in our data, so not checking the lower bounds)\n",
    "int_min_max = [\n",
    "    [\"int64\", np.iinfo(np.int64).min, np.iinfo(np.int64).max],\n",
    "    [\"int32\", np.iinfo(np.int32).min, np.iinfo(np.int32).max],\n",
    "    [\"int16\", np.iinfo(np.int16).min, np.iinfo(np.int16).max],\n",
    "    [\"int8\", np.iinfo(np.int8).min, np.iinfo(np.int8).max],\n",
    "]\n",
    "print(\n",
    "    tabulate(\n",
    "        int_min_max,\n",
    "        headers=[\"type\", \"min value\", \"max value\"],\n",
    "        showindex=True,\n",
    "        tablefmt=\"github\",\n",
    "        numalign=\"right\",\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# What is the max for each numeric column?\n",
    "\n",
    "citibike_min_max = [\n",
    "    [\"tripduration\", df.tripduration.min(), df.tripduration.max()],\n",
    "    [\"startstationid\", df.startstationid.min(), df.startstationid.max()],\n",
    "    [\"endstationid\", df.endstationid.min(), df.endstationid.max()],\n",
    "    [\"bikeid\", df.bikeid.min(), df.bikeid.max()],\n",
    "    [\"birthyear\", df.birthyear.min(), df.birthyear.max()],\n",
    "]\n",
    "print(\n",
    "    tabulate(\n",
    "        citibike_min_max,\n",
    "        headers=[\"column\", \"min value\", \"max value\"],\n",
    "        showindex=True,\n",
    "        tablefmt=\"github\",\n",
    "        numalign=\"right\",\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### What do we find out?\n",
    "Someone took a 44 day bike trip...\n",
    "More pertinently, we have no negative values so we can use **unsigned ints** when downcasting to save even more space\n",
    "\n",
    "#### Downcasting the data\n",
    "Now that we've looked at the numeric values, we can tell pandas to use the downcast to the appropriate numeric types for each column"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# Drop NAs before downcasting\n",
    "df.dropna(axis=0, inplace=True)\n",
    "\n",
    "# # Use smaller numeric types\n",
    "# df['tripduration'] = df['tripduration'].astype('int32')\n",
    "# df['startstationid'] = df['startstationid'].astype('int16')\n",
    "# df['endstationid'] = df['endstationid'].astype('int16')\n",
    "# df['bikeid'] = df['bikeid'].astype('int32')\n",
    "# df['birthyear'] = df['birthyear'].astype('int16')\n",
    "# df['gender'] = df['gender'].astype('int8')\n",
    "\n",
    "# actually, let's downcast automatically instead of manually...\n",
    "# NOTE: for floats, we lose precision, but that isn't important because we are not doing arithmetic operations that would require high precision\n",
    "# E.g., float32 gives 6 digits of precision as opposed to 15 for float64\n",
    "for column in df:\n",
    "    if df[column].dtype == \"float64\":\n",
    "        df[column] = pd.to_numeric(df[column], downcast=\"float\")\n",
    "    if df[column].dtype == \"int64\":\n",
    "        df[column] = pd.to_numeric(df[column], downcast=\"unsigned\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# profile memory again\n",
    "downcasted_memory = df.memory_usage(index=False, deep=True).sum() / (2 ** 30)\n",
    "print(f\"{round(downcasted_memory, 3)} GiB\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Use categorical type\n",
    "The `usertype` column is categorical. A Citi Bike user can be either a `Subscriber` or a `Customer`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "df[\"usertype\"] = df[\"usertype\"].astype(\"category\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# profile memory again\n",
    "categorical_memory = df.memory_usage(index=False, deep=True).sum() / (2 ** 30)\n",
    "print(f\"{round(categorical_memory, 3)} GiB\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### DateTime\n",
    "The `starttime` and `stoptime` columns for a trip being as strings. \n",
    "When we do our time series data analysis, we'd like them to be `datetime`s\n",
    "Will this reduce the size?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "df[\"starttime\"] = pd.to_datetime(df[\"starttime\"])\n",
    "df[\"stoptime\"] = pd.to_datetime(df[\"stoptime\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# profile memory again\n",
    "datetime_memory = df.memory_usage(index=False, deep=True).sum() / (2 ** 30)\n",
    "print(f\"{round(datetime_memory, 3)} GiB\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Outcome\n",
    "Wow! Using DateTime helps a lot and we get significant gains in memory reduction from using a categorical type. Smaller numeric types give a smaller percentage reduction, but still useful"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "print(\n",
    "    f\"Reduced dataframe size by {round(100*(start_memory - datetime_memory)/start_memory, 2)}%\"\n",
    ")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}