notebooks/pint-pandas.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Pandas support\n",
    "\n",
    "<div class=\"alert alert-warning\">\n",
    "\n",
    "**Warning:** pandas support is currently experimental, don't expect everything to work.\n",
    "\n",
    "</div>\n",
    "\n",
    "It is convenient to use the Pandas package when dealing with numerical data, so Pint provides PintArray. A PintArray is a Pandas Extension Array, which allows Pandas to recognise the Quantity and store it in Pandas DataFrames and Series."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Installation\n",
    "\n",
    "\n",
    "Pandas support is provided by the `pint-pandas` package. To install it use either:\n",
    "```\n",
    "python -m pip install pint-pandas\n",
    "```\n",
    "Or:\n",
    "```\n",
    "conda install -c conda-forge pint-pandas\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Basic example"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This example will show the simplist way to use pandas with pint and the underlying objects. It's slightly fiddly as you are not reading from a file. A more normal use case is given in Reading a csv.\n",
    "\n",
    "First some imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import pint\n",
    "import pint_pandas\n",
    "\n",
    "pint_pandas.show_versions()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we create a DataFrame with PintArrays as columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.DataFrame(\n",
    "    {\n",
    "        \"torque\": pd.Series([1.0, 2.0, 2.0, 3.0], dtype=\"pint[lbf ft]\"),\n",
    "        \"angular_velocity\": pd.Series([1.0, 2.0, 2.0, 3.0], dtype=\"pint[rpm]\"),\n",
    "    }\n",
    ")\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Operations with columns are units aware so behave as we would intuitively expect."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"power\"] = df[\"torque\"] * df[\"angular_velocity\"]\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see the columns' units in the dtypes attribute"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each column can be accessed as a Pandas Series"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.power"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Which contains a PintArray"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.power.values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The PintArray contains a Quantity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.power.values.quantity"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pandas Series accessors are provided for most Quantity properties and methods, which will convert the result to a Series where possible."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.power.pint.units"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.power.pint.to(\"kW\").values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reading from csv\n",
    "\n",
    "Reading from files is the far more standard way to use pandas. To facilitate this, DataFrame accessors are provided to make it easy to get to PintArrays. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import pint\n",
    "import pint_pandas\n",
    "import io"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's the contents of the csv file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_data = \"\"\"ShaftSpeedIndex,rpm,1200,1200,1200,1600,1600,1600,2300,2300,2300\n",
    "pump,,A,B,C,A,B,C,A,B,C\n",
    "TestDate,No Unit,01/01,01/01,01/01,01/01,01/01,01/01,01/02,01/02,01/02\n",
    "ShaftSpeed,rpm,1200,1200,1200,1600,1600,1600,2300,2300,2300\n",
    "FlowRate,m^3 h^-1,8.72,9.28,9.31,11.61,12.78,13.51,18.32,17.90,19.23\n",
    "DifferentialPressure,kPa,162.03,144.16,136.47,286.86,241.41,204.21,533.17,526.74,440.76\n",
    "ShaftPower,kW,1.32,1.23,1.18,3.09,2.78,2.50,8.59,8.51,7.61\n",
    "Efficiency,dimensionless,30.60,31.16,30.70,30.72,31.83,31.81,32.52,31.67,32.05\"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's read that into a DataFrame.\n",
    "Here io.StringIO is used in place of reading a file from disk, whereas a csv file path would typically be used and is shown commented."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv(io.StringIO(test_data), header=[0, 1], index_col=[0, 1]).T\n",
    "# df = pd.read_csv(\"/path/to/test_data.csv\", header=[0, 1])\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then use the DataFrame's pint accessor's quantify method to convert the columns from `np.ndarray`s to PintArrays, with units from the bottom column level.\n",
    "\n",
    "Using 'No Unit' as the unit will prevent quantify converting a column to a PintArray. This can be changed by changing pint_pandas.pint_array.NO_UNIT."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_ = df.pint.quantify(level=-1)\n",
    "df_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's confirm the units have been parsed correctly"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here the h in m^3 h^-1 has been parsed as the planck constant. Let's change the unit to hours."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_[\"FlowRate\"] = pint_pandas.PintArray(\n",
    "    df_[\"FlowRate\"].values.quantity.m, dtype=\"pint[m^3/hr]\"\n",
    ")\n",
    "df_.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As previously, operations between DataFrame columns are unit aware"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_.ShaftPower / df_.ShaftSpeed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_[\"ShaftTorque\"] = df_.ShaftPower / df_.ShaftSpeed\n",
    "df_[\"FluidPower\"] = df_[\"FlowRate\"] * df_[\"DifferentialPressure\"]\n",
    "df_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The DataFrame's `pint.dequantify` method then allows us to retrieve the units information as a header row once again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_.pint.dequantify()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This allows for some rather powerful abilities. For example, to change a column's units"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_[\"FluidPower\"] = df_[\"FluidPower\"].pint.to(\"kW\")\n",
    "df_[\"FlowRate\"] = df_[\"FlowRate\"].pint.to(\"L/s\")\n",
    "df_[\"ShaftTorque\"] = df_[\"ShaftTorque\"].pint.to(\"N m\")\n",
    "df_.pint.dequantify()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The units are harder to read than they need be, so lets change pints default format for displaying units."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pint_pandas.PintType.ureg.formatter.default_format = \"P~\"\n",
    "df_.pint.dequantify()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "or the entire table's units"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_.pint.to_base_units().pint.dequantify()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Plotting\n",
    "Pint's matplotlib support allows columns with the same dimensionality to be plotted."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pint_pandas.PintType.ureg.setup_matplotlib()\n",
    "ax = df_[[\"ShaftPower\", \"FluidPower\"]].unstack(\"pump\").plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ax.yaxis.units"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that indexes cannot store PintArrays, so don't contain unit information"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(ax.xaxis.units)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Advanced example\n",
    "This example shows alternative ways to use pint with pandas and other features.\n",
    "\n",
    "Start with the same imports."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import pint\n",
    "import pint_pandas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll be use a shorthand for PintArray"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "PA_ = pint_pandas.PintArray"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And set up a unit registry and quantity shorthand."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ureg = pint.UnitRegistry()\n",
    "Q_ = ureg.Quantity"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Operations between PintArrays of different unit registry will not work. We can change the unit registry that will be used in creating new PintArrays to prevent this issue."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pint_pandas.PintType.ureg = ureg"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These are the possible ways to create a PintArray.\n",
    "\n",
    "Note that pint[unit] must be used for the Series constuctor, whereas the PintArray constructor allows the unit string or object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.DataFrame(\n",
    "    {\n",
    "        \"length\": pd.Series([1.0, 2.0], dtype=\"pint[m]\"),\n",
    "        \"width\": PA_([2.0, 3.0], dtype=\"pint[m]\"),\n",
    "        \"distance\": PA_([2.0, 3.0], dtype=\"m\"),\n",
    "        \"height\": PA_([2.0, 3.0], dtype=ureg.m),\n",
    "        \"depth\": PA_.from_1darray_quantity(Q_([2, 3], ureg.m)),\n",
    "        \"displacement\": PA_(Q_([2.0, 3.0], ureg.m)),\n",
    "    }\n",
    ")\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.length.values.units"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}