examples-dask.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# d6tstack with Dask\n",
    "\n",
    "Dask is a great library for out-of-core computing. But if input files are not properly organized it quickly breaks. For example:\n",
    "\n",
    "1) if columns are different between files, dask won't even read the data! It doesn't tell you what you need to do to fix it.\n",
    "\n",
    "2) if column order is rearranged between files it will read data, but into the wrong columns and you won't notice it\n",
    "\n",
    "Dask can't handle those scenarios. With d6tstack you can easily fix the situation with just a few lines of code!\n",
    "\n",
    "For more instructions, examples and documentation see https://github.com/d6t/d6tstack"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Base Case: Columns are same between all files\n",
    "As a base case, we have input files which have consistent input columns and thus can be easily read in dask."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/anaconda3/lib/python3.6/importlib/_bootstrap.py:219: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88\n",
      "  return f(*args, **kwds)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>date</th>\n",
       "      <th>sales</th>\n",
       "      <th>cost</th>\n",
       "      <th>profit</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2011-02-01</td>\n",
       "      <td>200</td>\n",
       "      <td>-90</td>\n",
       "      <td>110</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2011-02-02</td>\n",
       "      <td>200</td>\n",
       "      <td>-90</td>\n",
       "      <td>110</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2011-02-03</td>\n",
       "      <td>200</td>\n",
       "      <td>-90</td>\n",
       "      <td>110</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2011-02-04</td>\n",
       "      <td>200</td>\n",
       "      <td>-90</td>\n",
       "      <td>110</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2011-02-05</td>\n",
       "      <td>200</td>\n",
       "      <td>-90</td>\n",
       "      <td>110</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>2011-02-06</td>\n",
       "      <td>200</td>\n",
       "      <td>-90</td>\n",
       "      <td>110</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>2011-02-07</td>\n",
       "      <td>200</td>\n",
       "      <td>-90</td>\n",
       "      <td>110</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>2011-02-08</td>\n",
       "      <td>200</td>\n",
       "      <td>-90</td>\n",
       "      <td>110</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>2011-02-09</td>\n",
       "      <td>200</td>\n",
       "      <td>-90</td>\n",
       "      <td>110</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>2011-02-10</td>\n",
       "      <td>200</td>\n",
       "      <td>-90</td>\n",
       "      <td>110</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2011-01-01</td>\n",
       "      <td>100</td>\n",
       "      <td>-80</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2011-01-02</td>\n",
       "      <td>100</td>\n",
       "      <td>-80</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2011-01-03</td>\n",
       "      <td>100</td>\n",
       "      <td>-80</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2011-01-04</td>\n",
       "      <td>100</td>\n",
       "      <td>-80</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2011-01-05</td>\n",
       "      <td>100</td>\n",
       "      <td>-80</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>2011-01-06</td>\n",
       "      <td>100</td>\n",
       "      <td>-80</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>2011-01-07</td>\n",
       "      <td>100</td>\n",
       "      <td>-80</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>2011-01-08</td>\n",
       "      <td>100</td>\n",
       "      <td>-80</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>2011-01-09</td>\n",
       "      <td>100</td>\n",
       "      <td>-80</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>2011-01-10</td>\n",
       "      <td>100</td>\n",
       "      <td>-80</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2011-03-01</td>\n",
       "      <td>300</td>\n",
       "      <td>-100</td>\n",
       "      <td>200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2011-03-02</td>\n",
       "      <td>300</td>\n",
       "      <td>-100</td>\n",
       "      <td>200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2011-03-03</td>\n",
       "      <td>300</td>\n",
       "      <td>-100</td>\n",
       "      <td>200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2011-03-04</td>\n",
       "      <td>300</td>\n",
       "      <td>-100</td>\n",
       "      <td>200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2011-03-05</td>\n",
       "      <td>300</td>\n",
       "      <td>-100</td>\n",
       "      <td>200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>2011-03-06</td>\n",
       "      <td>300</td>\n",
       "      <td>-100</td>\n",
       "      <td>200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>2011-03-07</td>\n",
       "      <td>300</td>\n",
       "      <td>-100</td>\n",
       "      <td>200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>2011-03-08</td>\n",
       "      <td>300</td>\n",
       "      <td>-100</td>\n",
       "      <td>200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>2011-03-09</td>\n",
       "      <td>300</td>\n",
       "      <td>-100</td>\n",
       "      <td>200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>2011-03-10</td>\n",
       "      <td>300</td>\n",
       "      <td>-100</td>\n",
       "      <td>200</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         date  sales  cost  profit\n",
       "0  2011-02-01    200   -90     110\n",
       "1  2011-02-02    200   -90     110\n",
       "2  2011-02-03    200   -90     110\n",
       "3  2011-02-04    200   -90     110\n",
       "4  2011-02-05    200   -90     110\n",
       "5  2011-02-06    200   -90     110\n",
       "6  2011-02-07    200   -90     110\n",
       "7  2011-02-08    200   -90     110\n",
       "8  2011-02-09    200   -90     110\n",
       "9  2011-02-10    200   -90     110\n",
       "0  2011-01-01    100   -80      20\n",
       "1  2011-01-02    100   -80      20\n",
       "2  2011-01-03    100   -80      20\n",
       "3  2011-01-04    100   -80      20\n",
       "4  2011-01-05    100   -80      20\n",
       "5  2011-01-06    100   -80      20\n",
       "6  2011-01-07    100   -80      20\n",
       "7  2011-01-08    100   -80      20\n",
       "8  2011-01-09    100   -80      20\n",
       "9  2011-01-10    100   -80      20\n",
       "0  2011-03-01    300  -100     200\n",
       "1  2011-03-02    300  -100     200\n",
       "2  2011-03-03    300  -100     200\n",
       "3  2011-03-04    300  -100     200\n",
       "4  2011-03-05    300  -100     200\n",
       "5  2011-03-06    300  -100     200\n",
       "6  2011-03-07    300  -100     200\n",
       "7  2011-03-08    300  -100     200\n",
       "8  2011-03-09    300  -100     200\n",
       "9  2011-03-10    300  -100     200"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import dask.dataframe as dd\n",
    "\n",
    "# consistent format\n",
    "ddf = dd.read_csv('test-data/input/test-data-input-csv-clean-*.csv')\n",
    "ddf.compute()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Problem Case 1: Columns are different between files\n",
    "That worked well. But what happens if your input files have inconsistent columns across files? Say for example one file has a new column that the other files don't have."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "ename": "ValueError",
     "evalue": "Length mismatch: Expected axis has 5 elements, new values have 4 elements",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-23-1bbb9709e59f>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m()\u001b[0m\n\u001b[0;32m      1\u001b[0m \u001b[1;31m# consistent format\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      2\u001b[0m \u001b[0mddf\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mdd\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mread_csv\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m'test-data/input/test-data-input-csv-colmismatch-*.csv'\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 3\u001b[1;33m \u001b[0mddf\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mcompute\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[1;32mC:\\Anaconda3\\lib\\site-packages\\dask\\base.py\u001b[0m in \u001b[0;36mcompute\u001b[1;34m(self, **kwargs)\u001b[0m\n\u001b[0;32m    153\u001b[0m         \u001b[0mdask\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mbase\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mcompute\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    154\u001b[0m         \"\"\"\n\u001b[1;32m--> 155\u001b[1;33m         \u001b[1;33m(\u001b[0m\u001b[0mresult\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mcompute\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mtraverse\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;32mFalse\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    156\u001b[0m         \u001b[1;32mreturn\u001b[0m \u001b[0mresult\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    157\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mC:\\Anaconda3\\lib\\site-packages\\dask\\base.py\u001b[0m in \u001b[0;36mcompute\u001b[1;34m(*args, **kwargs)\u001b[0m\n\u001b[0;32m    402\u001b[0m     postcomputes = [a.__dask_postcompute__() if is_dask_collection(a)\n\u001b[0;32m    403\u001b[0m                     else (None, a) for a in args]\n\u001b[1;32m--> 404\u001b[1;33m     \u001b[0mresults\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mget\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mdsk\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mkeys\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    405\u001b[0m     \u001b[0mresults_iter\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0miter\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mresults\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    406\u001b[0m     return tuple(a if f is None else f(next(results_iter), *a)\n",
      "\u001b[1;32mC:\\Anaconda3\\lib\\site-packages\\dask\\threaded.py\u001b[0m in \u001b[0;36mget\u001b[1;34m(dsk, result, cache, num_workers, **kwargs)\u001b[0m\n\u001b[0;32m     73\u001b[0m     results = get_async(pool.apply_async, len(pool._pool), dsk, result,\n\u001b[0;32m     74\u001b[0m                         \u001b[0mcache\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mcache\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mget_id\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0m_thread_get_id\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 75\u001b[1;33m                         pack_exception=pack_exception, **kwargs)\n\u001b[0m\u001b[0;32m     76\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     77\u001b[0m     \u001b[1;31m# Cleanup pools associated to dead threads\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mC:\\Anaconda3\\lib\\site-packages\\dask\\local.py\u001b[0m in \u001b[0;36mget_async\u001b[1;34m(apply_async, num_workers, dsk, result, cache, get_id, rerun_exceptions_locally, pack_exception, raise_exception, callbacks, dumps, loads, **kwargs)\u001b[0m\n\u001b[0;32m    519\u001b[0m                         \u001b[0m_execute_task\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mtask\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mdata\u001b[0m\u001b[1;33m)\u001b[0m  \u001b[1;31m# Re-execute locally\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    520\u001b[0m                     \u001b[1;32melse\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 521\u001b[1;33m                         \u001b[0mraise_exception\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mexc\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mtb\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    522\u001b[0m                 \u001b[0mres\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mworker_id\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mloads\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mres_info\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    523\u001b[0m                 \u001b[0mstate\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;34m'cache'\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m[\u001b[0m\u001b[0mkey\u001b[0m\u001b[1;33m]\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mres\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mC:\\Anaconda3\\lib\\site-packages\\dask\\compatibility.py\u001b[0m in \u001b[0;36mreraise\u001b[1;34m(exc, tb)\u001b[0m\n\u001b[0;32m     65\u001b[0m         \u001b[1;32mif\u001b[0m \u001b[0mexc\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m__traceback__\u001b[0m \u001b[1;32mis\u001b[0m \u001b[1;32mnot\u001b[0m \u001b[0mtb\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     66\u001b[0m             \u001b[1;32mraise\u001b[0m \u001b[0mexc\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mwith_traceback\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mtb\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 67\u001b[1;33m         \u001b[1;32mraise\u001b[0m \u001b[0mexc\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     68\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     69\u001b[0m \u001b[1;32melse\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mC:\\Anaconda3\\lib\\site-packages\\dask\\local.py\u001b[0m in \u001b[0;36mexecute_task\u001b[1;34m(key, task_info, dumps, loads, get_id, pack_exception)\u001b[0m\n\u001b[0;32m    288\u001b[0m     \u001b[1;32mtry\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    289\u001b[0m         \u001b[0mtask\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mdata\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mloads\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mtask_info\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 290\u001b[1;33m         \u001b[0mresult\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0m_execute_task\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mtask\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mdata\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    291\u001b[0m         \u001b[0mid\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mget_id\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    292\u001b[0m         \u001b[0mresult\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mdumps\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mresult\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mid\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mC:\\Anaconda3\\lib\\site-packages\\dask\\local.py\u001b[0m in \u001b[0;36m_execute_task\u001b[1;34m(arg, cache, dsk)\u001b[0m\n\u001b[0;32m    269\u001b[0m         \u001b[0mfunc\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0margs\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0marg\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;36m0\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0marg\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;36m1\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    270\u001b[0m         \u001b[0margs2\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;33m[\u001b[0m\u001b[0m_execute_task\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0ma\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mcache\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;32mfor\u001b[0m \u001b[0ma\u001b[0m \u001b[1;32min\u001b[0m \u001b[0margs\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 271\u001b[1;33m         \u001b[1;32mreturn\u001b[0m \u001b[0mfunc\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m*\u001b[0m\u001b[0margs2\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    272\u001b[0m     \u001b[1;32melif\u001b[0m \u001b[1;32mnot\u001b[0m \u001b[0mishashable\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0marg\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    273\u001b[0m         \u001b[1;32mreturn\u001b[0m \u001b[0marg\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mC:\\Anaconda3\\lib\\site-packages\\dask\\compatibility.py\u001b[0m in \u001b[0;36mapply\u001b[1;34m(func, args, kwargs)\u001b[0m\n\u001b[0;32m     46\u001b[0m     \u001b[1;32mdef\u001b[0m \u001b[0mapply\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mfunc\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0margs\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mkwargs\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;32mNone\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     47\u001b[0m         \u001b[1;32mif\u001b[0m \u001b[0mkwargs\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 48\u001b[1;33m             \u001b[1;32mreturn\u001b[0m \u001b[0mfunc\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m*\u001b[0m\u001b[0margs\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     49\u001b[0m         \u001b[1;32melse\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     50\u001b[0m             \u001b[1;32mreturn\u001b[0m \u001b[0mfunc\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m*\u001b[0m\u001b[0margs\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mC:\\Anaconda3\\lib\\site-packages\\dask\\dataframe\\io\\csv.py\u001b[0m in \u001b[0;36mpandas_read_text\u001b[1;34m(reader, b, header, kwargs, dtypes, columns, write_header, enforce)\u001b[0m\n\u001b[0;32m     69\u001b[0m         \u001b[1;32mraise\u001b[0m \u001b[0mValueError\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m\"Columns do not match\"\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mdf\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mcolumns\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mcolumns\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     70\u001b[0m     \u001b[1;32melif\u001b[0m \u001b[0mcolumns\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 71\u001b[1;33m         \u001b[0mdf\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mcolumns\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mcolumns\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     72\u001b[0m     \u001b[1;32mreturn\u001b[0m \u001b[0mdf\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     73\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mC:\\Anaconda3\\lib\\site-packages\\pandas\\core\\generic.py\u001b[0m in \u001b[0;36m__setattr__\u001b[1;34m(self, name, value)\u001b[0m\n\u001b[0;32m   3625\u001b[0m         \u001b[1;32mtry\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   3626\u001b[0m             \u001b[0mobject\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m__getattribute__\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mname\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m-> 3627\u001b[1;33m             \u001b[1;32mreturn\u001b[0m \u001b[0mobject\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m__setattr__\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mname\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mvalue\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m   3628\u001b[0m         \u001b[1;32mexcept\u001b[0m \u001b[0mAttributeError\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   3629\u001b[0m             \u001b[1;32mpass\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mpandas/_libs/properties.pyx\u001b[0m in \u001b[0;36mpandas._libs.properties.AxisProperty.__set__\u001b[1;34m()\u001b[0m\n",
      "\u001b[1;32mC:\\Anaconda3\\lib\\site-packages\\pandas\\core\\generic.py\u001b[0m in \u001b[0;36m_set_axis\u001b[1;34m(self, axis, labels)\u001b[0m\n\u001b[0;32m    557\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    558\u001b[0m     \u001b[1;32mdef\u001b[0m \u001b[0m_set_axis\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0maxis\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mlabels\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 559\u001b[1;33m         \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_data\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mset_axis\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0maxis\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mlabels\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    560\u001b[0m         \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_clear_item_cache\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    561\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mC:\\Anaconda3\\lib\\site-packages\\pandas\\core\\internals.py\u001b[0m in \u001b[0;36mset_axis\u001b[1;34m(self, axis, new_labels)\u001b[0m\n\u001b[0;32m   3072\u001b[0m             raise ValueError('Length mismatch: Expected axis has %d elements, '\n\u001b[0;32m   3073\u001b[0m                              \u001b[1;34m'new values have %d elements'\u001b[0m \u001b[1;33m%\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m-> 3074\u001b[1;33m                              (old_len, new_len))\n\u001b[0m\u001b[0;32m   3075\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   3076\u001b[0m         \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0maxes\u001b[0m\u001b[1;33m[\u001b[0m\u001b[0maxis\u001b[0m\u001b[1;33m]\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mnew_labels\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mValueError\u001b[0m: Length mismatch: Expected axis has 5 elements, new values have 4 elements"
     ]
    }
   ],
   "source": [
    "# consistent format\n",
    "ddf = dd.read_csv('test-data/input/test-data-input-csv-colmismatch-*.csv')\n",
    "ddf.compute()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fixing the problem with d6stack\n",
    "Urgh! There's no way to use these files in dask. You don't even know what's going on. What file caused the problem? Why did it cause a problem? All you know is one file has more columns than the first file.\n",
    "\n",
    "You can either manually process those files or use d6tstack to easily check for such a situation and fix it with a few lines of code - no manual processing required. Let's take a look!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sniffing columns ok\n",
      "all equal False\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>date</th>\n",
       "      <th>sales</th>\n",
       "      <th>cost</th>\n",
       "      <th>profit</th>\n",
       "      <th>profit2</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>file_path</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>test-data/input/test-data-input-csv-colmismatch-feb.csv</th>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>test-data/input/test-data-input-csv-colmismatch-jan.csv</th>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>test-data/input/test-data-input-csv-colmismatch-mar.csv</th>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                    date  sales  cost  profit  profit2\n",
       "file_path                                                                             \n",
       "test-data/input/test-data-input-csv-colmismatch...  True   True  True    True    False\n",
       "test-data/input/test-data-input-csv-colmismatch...  True   True  True    True    False\n",
       "test-data/input/test-data-input-csv-colmismatch...  True   True  True    True     True"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import glob\n",
    "import d6tstack.combine_csv\n",
    "\n",
    "cfg_fnames = list(glob.glob('test-data/input/test-data-input-csv-colmismatch-*.csv'))\n",
    "c = d6tstack.combine_csv.CombinerCSV(cfg_fnames)\n",
    "\n",
    "# check columns\n",
    "print('all equal',c.is_all_equal())\n",
    "print('')\n",
    "c.is_column_present()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before using dask you can quickly use d6stack to check if all colums are consistent with `d6tstack.combine_csv.CombinerCSV.is_all_equal()`. If they are not consistent you can easily see which files are causing problems with `d6tstack.combine_csv.CombinerCSV.is_col_present()`, in this case there is a new column \"profit2\" in \"test-data-input-csv-colmismatch-mar.csv\".\n",
    "\n",
    "**Let's use d6stack to fix the situation.** We will use out-of-core processing with `d6tstack.combine_csv.CombinerCSVAdvanced.combine_save()` to save data from all files into one combined file with constistent columns. Any missing data is filled with NaN (to keep only common columns use `cfg_col_sel=c.col_preview['columns_common']`) Just 2 lines of code! "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sniffing columns ok\n",
      "writing test-data/output/d6tstack-test-data-input-csv-colmismatch-feb.csv ok\n",
      "writing test-data/output/d6tstack-test-data-input-csv-colmismatch-jan.csv ok\n",
      "writing test-data/output/d6tstack-test-data-input-csv-colmismatch-mar.csv ok\n"
     ]
    }
   ],
   "source": [
    "# out-of-core combining\n",
    "fnames = d6tstack.combine_csv.CombinerCSV(cfg_fnames).to_csv_align(output_dir='test-data/output/')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "NB: Instead of `to_csv_align()` you can also run `to_csv_combine()` which creates a single combined file.\n",
    "\n",
    "Now you can read this in dask and do whatever you wanted to do in the first place."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>date</th>\n",
       "      <th>sales</th>\n",
       "      <th>cost</th>\n",
       "      <th>profit</th>\n",
       "      <th>profit2</th>\n",
       "      <th>filepath</th>\n",
       "      <th>filename</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2011-02-01</td>\n",
       "      <td>200</td>\n",
       "      <td>-90</td>\n",
       "      <td>110</td>\n",
       "      <td>NaN</td>\n",
       "      <td>test-data/input/test-data-input-csv-colmismatc...</td>\n",
       "      <td>test-data-input-csv-colmismatch-feb.csv</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2011-02-02</td>\n",
       "      <td>200</td>\n",
       "      <td>-90</td>\n",
       "      <td>110</td>\n",
       "      <td>NaN</td>\n",
       "      <td>test-data/input/test-data-input-csv-colmismatc...</td>\n",
       "      <td>test-data-input-csv-colmismatch-feb.csv</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2011-02-03</td>\n",
       "      <td>200</td>\n",
       "      <td>-90</td>\n",
       "      <td>110</td>\n",
       "      <td>NaN</td>\n",
       "      <td>test-data/input/test-data-input-csv-colmismatc...</td>\n",
       "      <td>test-data-input-csv-colmismatch-feb.csv</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2011-02-04</td>\n",
       "      <td>200</td>\n",
       "      <td>-90</td>\n",
       "      <td>110</td>\n",
       "      <td>NaN</td>\n",
       "      <td>test-data/input/test-data-input-csv-colmismatc...</td>\n",
       "      <td>test-data-input-csv-colmismatch-feb.csv</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2011-02-05</td>\n",
       "      <td>200</td>\n",
       "      <td>-90</td>\n",
       "      <td>110</td>\n",
       "      <td>NaN</td>\n",
       "      <td>test-data/input/test-data-input-csv-colmismatc...</td>\n",
       "      <td>test-data-input-csv-colmismatch-feb.csv</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>2011-02-06</td>\n",
       "      <td>200</td>\n",
       "      <td>-90</td>\n",
       "      <td>110</td>\n",
       "      <td>NaN</td>\n",
       "      <td>test-data/input/test-data-input-csv-colmismatc...</td>\n",
       "      <td>test-data-input-csv-colmismatch-feb.csv</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>2011-02-07</td>\n",
       "      <td>200</td>\n",
       "      <td>-90</td>\n",
       "      <td>110</td>\n",
       "      <td>NaN</td>\n",
       "      <td>test-data/input/test-data-input-csv-colmismatc...</td>\n",
       "      <td>test-data-input-csv-colmismatch-feb.csv</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>2011-02-08</td>\n",
       "      <td>200</td>\n",
       "      <td>-90</td>\n",
       "      <td>110</td>\n",
       "      <td>NaN</td>\n",
       "      <td>test-data/input/test-data-input-csv-colmismatc...</td>\n",
       "      <td>test-data-input-csv-colmismatch-feb.csv</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>2011-02-09</td>\n",
       "      <td>200</td>\n",
       "      <td>-90</td>\n",
       "      <td>110</td>\n",
       "      <td>NaN</td>\n",
       "      <td>test-data/input/test-data-input-csv-colmismatc...</td>\n",
       "      <td>test-data-input-csv-colmismatch-feb.csv</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>2011-02-10</td>\n",
       "      <td>200</td>\n",
       "      <td>-90</td>\n",
       "      <td>110</td>\n",
       "      <td>NaN</td>\n",
       "      <td>test-data/input/test-data-input-csv-colmismatc...</td>\n",
       "      <td>test-data-input-csv-colmismatch-feb.csv</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2011-01-01</td>\n",
       "      <td>100</td>\n",
       "      <td>-80</td>\n",
       "      <td>20</td>\n",
       "      <td>NaN</td>\n",
       "      <td>test-data/input/test-data-input-csv-colmismatc...</td>\n",
       "      <td>test-data-input-csv-colmismatch-jan.csv</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2011-01-02</td>\n",
       "      <td>100</td>\n",
       "      <td>-80</td>\n",
       "      <td>20</td>\n",
       "      <td>NaN</td>\n",
       "      <td>test-data/input/test-data-input-csv-colmismatc...</td>\n",
       "      <td>test-data-input-csv-colmismatch-jan.csv</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2011-01-03</td>\n",
       "      <td>100</td>\n",
       "      <td>-80</td>\n",
       "      <td>20</td>\n",
       "      <td>NaN</td>\n",
       "      <td>test-data/input/test-data-input-csv-colmismatc...</td>\n",
       "      <td>test-data-input-csv-colmismatch-jan.csv</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2011-01-04</td>\n",
       "      <td>100</td>\n",
       "      <td>-80</td>\n",
       "      <td>20</td>\n",
       "      <td>NaN</td>\n",
       "      <td>test-data/input/test-data-input-csv-colmismatc...</td>\n",
       "      <td>test-data-input-csv-colmismatch-jan.csv</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2011-01-05</td>\n",
       "      <td>100</td>\n",
       "      <td>-80</td>\n",
       "      <td>20</td>\n",
       "      <td>NaN</td>\n",
       "      <td>test-data/input/test-data-input-csv-colmismatc...</td>\n",
       "      <td>test-data-input-csv-colmismatch-jan.csv</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>2011-01-06</td>\n",
       "      <td>100</td>\n",
       "      <td>-80</td>\n",
       "      <td>20</td>\n",
       "      <td>NaN</td>\n",
       "      <td>test-data/input/test-data-input-csv-colmismatc...</td>\n",
       "      <td>test-data-input-csv-colmismatch-jan.csv</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>2011-01-07</td>\n",
       "      <td>100</td>\n",
       "      <td>-80</td>\n",
       "      <td>20</td>\n",
       "      <td>NaN</td>\n",
       "      <td>test-data/input/test-data-input-csv-colmismatc...</td>\n",
       "      <td>test-data-input-csv-colmismatch-jan.csv</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>2011-01-08</td>\n",
       "      <td>100</td>\n",
       "      <td>-80</td>\n",
       "      <td>20</td>\n",
       "      <td>NaN</td>\n",
       "      <td>test-data/input/test-data-input-csv-colmismatc...</td>\n",
       "      <td>test-data-input-csv-colmismatch-jan.csv</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>2011-01-09</td>\n",
       "      <td>100</td>\n",
       "      <td>-80</td>\n",
       "      <td>20</td>\n",
       "      <td>NaN</td>\n",
       "      <td>test-data/input/test-data-input-csv-colmismatc...</td>\n",
       "      <td>test-data-input-csv-colmismatch-jan.csv</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>2011-01-10</td>\n",
       "      <td>100</td>\n",
       "      <td>-80</td>\n",
       "      <td>20</td>\n",
       "      <td>NaN</td>\n",
       "      <td>test-data/input/test-data-input-csv-colmismatc...</td>\n",
       "      <td>test-data-input-csv-colmismatch-jan.csv</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2011-03-01</td>\n",
       "      <td>300</td>\n",
       "      <td>-100</td>\n",
       "      <td>200</td>\n",
       "      <td>400.0</td>\n",
       "      <td>test-data/input/test-data-input-csv-colmismatc...</td>\n",
       "      <td>test-data-input-csv-colmismatch-mar.csv</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2011-03-02</td>\n",
       "      <td>300</td>\n",
       "      <td>-100</td>\n",
       "      <td>200</td>\n",
       "      <td>400.0</td>\n",
       "      <td>test-data/input/test-data-input-csv-colmismatc...</td>\n",
       "      <td>test-data-input-csv-colmismatch-mar.csv</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2011-03-03</td>\n",
       "      <td>300</td>\n",
       "      <td>-100</td>\n",
       "      <td>200</td>\n",
       "      <td>400.0</td>\n",
       "      <td>test-data/input/test-data-input-csv-colmismatc...</td>\n",
       "      <td>test-data-input-csv-colmismatch-mar.csv</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2011-03-04</td>\n",
       "      <td>300</td>\n",
       "      <td>-100</td>\n",
       "      <td>200</td>\n",
       "      <td>400.0</td>\n",
       "      <td>test-data/input/test-data-input-csv-colmismatc...</td>\n",
       "      <td>test-data-input-csv-colmismatch-mar.csv</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2011-03-05</td>\n",
       "      <td>300</td>\n",
       "      <td>-100</td>\n",
       "      <td>200</td>\n",
       "      <td>400.0</td>\n",
       "      <td>test-data/input/test-data-input-csv-colmismatc...</td>\n",
       "      <td>test-data-input-csv-colmismatch-mar.csv</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>2011-03-06</td>\n",
       "      <td>300</td>\n",
       "      <td>-100</td>\n",
       "      <td>200</td>\n",
       "      <td>400.0</td>\n",
       "      <td>test-data/input/test-data-input-csv-colmismatc...</td>\n",
       "      <td>test-data-input-csv-colmismatch-mar.csv</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>2011-03-07</td>\n",
       "      <td>300</td>\n",
       "      <td>-100</td>\n",
       "      <td>200</td>\n",
       "      <td>400.0</td>\n",
       "      <td>test-data/input/test-data-input-csv-colmismatc...</td>\n",
       "      <td>test-data-input-csv-colmismatch-mar.csv</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>2011-03-08</td>\n",
       "      <td>300</td>\n",
       "      <td>-100</td>\n",
       "      <td>200</td>\n",
       "      <td>400.0</td>\n",
       "      <td>test-data/input/test-data-input-csv-colmismatc...</td>\n",
       "      <td>test-data-input-csv-colmismatch-mar.csv</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>2011-03-09</td>\n",
       "      <td>300</td>\n",
       "      <td>-100</td>\n",
       "      <td>200</td>\n",
       "      <td>400.0</td>\n",
       "      <td>test-data/input/test-data-input-csv-colmismatc...</td>\n",
       "      <td>test-data-input-csv-colmismatch-mar.csv</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>2011-03-10</td>\n",
       "      <td>300</td>\n",
       "      <td>-100</td>\n",
       "      <td>200</td>\n",
       "      <td>400.0</td>\n",
       "      <td>test-data/input/test-data-input-csv-colmismatc...</td>\n",
       "      <td>test-data-input-csv-colmismatch-mar.csv</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         date  sales  cost  profit  profit2                                           filepath                                 filename\n",
       "0  2011-02-01    200   -90     110      NaN  test-data/input/test-data-input-csv-colmismatc...  test-data-input-csv-colmismatch-feb.csv\n",
       "1  2011-02-02    200   -90     110      NaN  test-data/input/test-data-input-csv-colmismatc...  test-data-input-csv-colmismatch-feb.csv\n",
       "2  2011-02-03    200   -90     110      NaN  test-data/input/test-data-input-csv-colmismatc...  test-data-input-csv-colmismatch-feb.csv\n",
       "3  2011-02-04    200   -90     110      NaN  test-data/input/test-data-input-csv-colmismatc...  test-data-input-csv-colmismatch-feb.csv\n",
       "4  2011-02-05    200   -90     110      NaN  test-data/input/test-data-input-csv-colmismatc...  test-data-input-csv-colmismatch-feb.csv\n",
       "5  2011-02-06    200   -90     110      NaN  test-data/input/test-data-input-csv-colmismatc...  test-data-input-csv-colmismatch-feb.csv\n",
       "6  2011-02-07    200   -90     110      NaN  test-data/input/test-data-input-csv-colmismatc...  test-data-input-csv-colmismatch-feb.csv\n",
       "7  2011-02-08    200   -90     110      NaN  test-data/input/test-data-input-csv-colmismatc...  test-data-input-csv-colmismatch-feb.csv\n",
       "8  2011-02-09    200   -90     110      NaN  test-data/input/test-data-input-csv-colmismatc...  test-data-input-csv-colmismatch-feb.csv\n",
       "9  2011-02-10    200   -90     110      NaN  test-data/input/test-data-input-csv-colmismatc...  test-data-input-csv-colmismatch-feb.csv\n",
       "0  2011-01-01    100   -80      20      NaN  test-data/input/test-data-input-csv-colmismatc...  test-data-input-csv-colmismatch-jan.csv\n",
       "1  2011-01-02    100   -80      20      NaN  test-data/input/test-data-input-csv-colmismatc...  test-data-input-csv-colmismatch-jan.csv\n",
       "2  2011-01-03    100   -80      20      NaN  test-data/input/test-data-input-csv-colmismatc...  test-data-input-csv-colmismatch-jan.csv\n",
       "3  2011-01-04    100   -80      20      NaN  test-data/input/test-data-input-csv-colmismatc...  test-data-input-csv-colmismatch-jan.csv\n",
       "4  2011-01-05    100   -80      20      NaN  test-data/input/test-data-input-csv-colmismatc...  test-data-input-csv-colmismatch-jan.csv\n",
       "5  2011-01-06    100   -80      20      NaN  test-data/input/test-data-input-csv-colmismatc...  test-data-input-csv-colmismatch-jan.csv\n",
       "6  2011-01-07    100   -80      20      NaN  test-data/input/test-data-input-csv-colmismatc...  test-data-input-csv-colmismatch-jan.csv\n",
       "7  2011-01-08    100   -80      20      NaN  test-data/input/test-data-input-csv-colmismatc...  test-data-input-csv-colmismatch-jan.csv\n",
       "8  2011-01-09    100   -80      20      NaN  test-data/input/test-data-input-csv-colmismatc...  test-data-input-csv-colmismatch-jan.csv\n",
       "9  2011-01-10    100   -80      20      NaN  test-data/input/test-data-input-csv-colmismatc...  test-data-input-csv-colmismatch-jan.csv\n",
       "0  2011-03-01    300  -100     200    400.0  test-data/input/test-data-input-csv-colmismatc...  test-data-input-csv-colmismatch-mar.csv\n",
       "1  2011-03-02    300  -100     200    400.0  test-data/input/test-data-input-csv-colmismatc...  test-data-input-csv-colmismatch-mar.csv\n",
       "2  2011-03-03    300  -100     200    400.0  test-data/input/test-data-input-csv-colmismatc...  test-data-input-csv-colmismatch-mar.csv\n",
       "3  2011-03-04    300  -100     200    400.0  test-data/input/test-data-input-csv-colmismatc...  test-data-input-csv-colmismatch-mar.csv\n",
       "4  2011-03-05    300  -100     200    400.0  test-data/input/test-data-input-csv-colmismatc...  test-data-input-csv-colmismatch-mar.csv\n",
       "5  2011-03-06    300  -100     200    400.0  test-data/input/test-data-input-csv-colmismatc...  test-data-input-csv-colmismatch-mar.csv\n",
       "6  2011-03-07    300  -100     200    400.0  test-data/input/test-data-input-csv-colmismatc...  test-data-input-csv-colmismatch-mar.csv\n",
       "7  2011-03-08    300  -100     200    400.0  test-data/input/test-data-input-csv-colmismatc...  test-data-input-csv-colmismatch-mar.csv\n",
       "8  2011-03-09    300  -100     200    400.0  test-data/input/test-data-input-csv-colmismatc...  test-data-input-csv-colmismatch-mar.csv\n",
       "9  2011-03-10    300  -100     200    400.0  test-data/input/test-data-input-csv-colmismatc...  test-data-input-csv-colmismatch-mar.csv"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# consistent format\n",
    "ddf = dd.read_csv('test-data/output/d6tstack-test-data-input-csv-colmismatch-*.csv')\n",
    "ddf.compute()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Problem Case 2: Columns are reordered between files\n",
    "This is a sneaky case. The columns are the same but the order is different! Dask will read everything just fine without a warning but your data is totally messed up!\n",
    "\n",
    "In the example below, the \"profit\" column contains data from the \"cost\" column!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>date</th>\n",
       "      <th>sales</th>\n",
       "      <th>cost</th>\n",
       "      <th>profit</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2011-02-01</td>\n",
       "      <td>200</td>\n",
       "      <td>-90</td>\n",
       "      <td>110</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2011-02-02</td>\n",
       "      <td>200</td>\n",
       "      <td>-90</td>\n",
       "      <td>110</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2011-02-03</td>\n",
       "      <td>200</td>\n",
       "      <td>-90</td>\n",
       "      <td>110</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2011-02-04</td>\n",
       "      <td>200</td>\n",
       "      <td>-90</td>\n",
       "      <td>110</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2011-02-05</td>\n",
       "      <td>200</td>\n",
       "      <td>-90</td>\n",
       "      <td>110</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>2011-02-06</td>\n",
       "      <td>200</td>\n",
       "      <td>-90</td>\n",
       "      <td>110</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>2011-02-07</td>\n",
       "      <td>200</td>\n",
       "      <td>-90</td>\n",
       "      <td>110</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>2011-02-08</td>\n",
       "      <td>200</td>\n",
       "      <td>-90</td>\n",
       "      <td>110</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>2011-02-09</td>\n",
       "      <td>200</td>\n",
       "      <td>-90</td>\n",
       "      <td>110</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>2011-02-10</td>\n",
       "      <td>200</td>\n",
       "      <td>-90</td>\n",
       "      <td>110</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2011-01-01</td>\n",
       "      <td>100</td>\n",
       "      <td>-80</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2011-01-02</td>\n",
       "      <td>100</td>\n",
       "      <td>-80</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2011-01-03</td>\n",
       "      <td>100</td>\n",
       "      <td>-80</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2011-01-04</td>\n",
       "      <td>100</td>\n",
       "      <td>-80</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2011-01-05</td>\n",
       "      <td>100</td>\n",
       "      <td>-80</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>2011-01-06</td>\n",
       "      <td>100</td>\n",
       "      <td>-80</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>2011-01-07</td>\n",
       "      <td>100</td>\n",
       "      <td>-80</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>2011-01-08</td>\n",
       "      <td>100</td>\n",
       "      <td>-80</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>2011-01-09</td>\n",
       "      <td>100</td>\n",
       "      <td>-80</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>2011-01-10</td>\n",
       "      <td>100</td>\n",
       "      <td>-80</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2011-03-01</td>\n",
       "      <td>300</td>\n",
       "      <td>200</td>\n",
       "      <td>-100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2011-03-02</td>\n",
       "      <td>300</td>\n",
       "      <td>200</td>\n",
       "      <td>-100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2011-03-03</td>\n",
       "      <td>300</td>\n",
       "      <td>200</td>\n",
       "      <td>-100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2011-03-04</td>\n",
       "      <td>300</td>\n",
       "      <td>200</td>\n",
       "      <td>-100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2011-03-05</td>\n",
       "      <td>300</td>\n",
       "      <td>200</td>\n",
       "      <td>-100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>2011-03-06</td>\n",
       "      <td>300</td>\n",
       "      <td>200</td>\n",
       "      <td>-100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>2011-03-07</td>\n",
       "      <td>300</td>\n",
       "      <td>200</td>\n",
       "      <td>-100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>2011-03-08</td>\n",
       "      <td>300</td>\n",
       "      <td>200</td>\n",
       "      <td>-100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>2011-03-09</td>\n",
       "      <td>300</td>\n",
       "      <td>200</td>\n",
       "      <td>-100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>2011-03-10</td>\n",
       "      <td>300</td>\n",
       "      <td>200</td>\n",
       "      <td>-100</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         date  sales  cost  profit\n",
       "0  2011-02-01    200   -90     110\n",
       "1  2011-02-02    200   -90     110\n",
       "2  2011-02-03    200   -90     110\n",
       "3  2011-02-04    200   -90     110\n",
       "4  2011-02-05    200   -90     110\n",
       "5  2011-02-06    200   -90     110\n",
       "6  2011-02-07    200   -90     110\n",
       "7  2011-02-08    200   -90     110\n",
       "8  2011-02-09    200   -90     110\n",
       "9  2011-02-10    200   -90     110\n",
       "0  2011-01-01    100   -80      20\n",
       "1  2011-01-02    100   -80      20\n",
       "2  2011-01-03    100   -80      20\n",
       "3  2011-01-04    100   -80      20\n",
       "4  2011-01-05    100   -80      20\n",
       "5  2011-01-06    100   -80      20\n",
       "6  2011-01-07    100   -80      20\n",
       "7  2011-01-08    100   -80      20\n",
       "8  2011-01-09    100   -80      20\n",
       "9  2011-01-10    100   -80      20\n",
       "0  2011-03-01    300   200    -100\n",
       "1  2011-03-02    300   200    -100\n",
       "2  2011-03-03    300   200    -100\n",
       "3  2011-03-04    300   200    -100\n",
       "4  2011-03-05    300   200    -100\n",
       "5  2011-03-06    300   200    -100\n",
       "6  2011-03-07    300   200    -100\n",
       "7  2011-03-08    300   200    -100\n",
       "8  2011-03-09    300   200    -100\n",
       "9  2011-03-10    300   200    -100"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# consistent format\n",
    "ddf = dd.read_csv('test-data/input/test-data-input-csv-reorder-*.csv')\n",
    "ddf.compute()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sniffing columns ok\n",
      "all columns equal? False\n",
      "\n",
      "in what order do columns appear in the files?\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>date</th>\n",
       "      <th>sales</th>\n",
       "      <th>cost</th>\n",
       "      <th>profit</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   date  sales  cost  profit\n",
       "0     0      1     2       3\n",
       "1     0      1     2       3\n",
       "2     0      1     3       2"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cfg_fnames = list(glob.glob('test-data/input/test-data-input-csv-reorder-*.csv'))\n",
    "c = d6tstack.combine_csv.CombinerCSV(cfg_fnames)\n",
    "\n",
    "# check columns\n",
    "col_sniff = c.sniff_columns()\n",
    "print('all columns equal?' , c.is_all_equal())\n",
    "print('')\n",
    "print('in what order do columns appear in the files?')\n",
    "print('')\n",
    "col_sniff['df_columns_order'].reset_index(drop=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, just a useful check before loading data into dask you can see that the columns don't line up. It's very fast to run because it only reads the headers, there's NO reason for you NOT to do it from a QA perspective.\n",
    "\n",
    "Same as above, the fix is the same few lines of code with d6stack."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sniffing columns ok\n",
      "writing test-data/output/d6tstack-test-data-input-csv-reorder-feb.csv ok\n",
      "writing test-data/output/d6tstack-test-data-input-csv-reorder-jan.csv ok\n",
      "writing test-data/output/d6tstack-test-data-input-csv-reorder-mar.csv ok\n"
     ]
    }
   ],
   "source": [
    "# out-of-core combining\n",
    "fnames = d6tstack.combine_csv.CombinerCSV(cfg_fnames).to_csv_align(output_dir='test-data/output/')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>date</th>\n",
       "      <th>sales</th>\n",
       "      <th>cost</th>\n",
       "      <th>profit</th>\n",
       "      <th>filepath</th>\n",
       "      <th>filename</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2011-02-01</td>\n",
       "      <td>200</td>\n",
       "      <td>-90</td>\n",
       "      <td>110</td>\n",
       "      <td>test-data/input/test-data-input-csv-reorder-fe...</td>\n",
       "      <td>test-data-input-csv-reorder-feb.csv</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2011-02-02</td>\n",
       "      <td>200</td>\n",
       "      <td>-90</td>\n",
       "      <td>110</td>\n",
       "      <td>test-data/input/test-data-input-csv-reorder-fe...</td>\n",
       "      <td>test-data-input-csv-reorder-feb.csv</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2011-02-03</td>\n",
       "      <td>200</td>\n",
       "      <td>-90</td>\n",
       "      <td>110</td>\n",
       "      <td>test-data/input/test-data-input-csv-reorder-fe...</td>\n",
       "      <td>test-data-input-csv-reorder-feb.csv</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2011-02-04</td>\n",
       "      <td>200</td>\n",
       "      <td>-90</td>\n",
       "      <td>110</td>\n",
       "      <td>test-data/input/test-data-input-csv-reorder-fe...</td>\n",
       "      <td>test-data-input-csv-reorder-feb.csv</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2011-02-05</td>\n",
       "      <td>200</td>\n",
       "      <td>-90</td>\n",
       "      <td>110</td>\n",
       "      <td>test-data/input/test-data-input-csv-reorder-fe...</td>\n",
       "      <td>test-data-input-csv-reorder-feb.csv</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>2011-02-06</td>\n",
       "      <td>200</td>\n",
       "      <td>-90</td>\n",
       "      <td>110</td>\n",
       "      <td>test-data/input/test-data-input-csv-reorder-fe...</td>\n",
       "      <td>test-data-input-csv-reorder-feb.csv</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>2011-02-07</td>\n",
       "      <td>200</td>\n",
       "      <td>-90</td>\n",
       "      <td>110</td>\n",
       "      <td>test-data/input/test-data-input-csv-reorder-fe...</td>\n",
       "      <td>test-data-input-csv-reorder-feb.csv</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>2011-02-08</td>\n",
       "      <td>200</td>\n",
       "      <td>-90</td>\n",
       "      <td>110</td>\n",
       "      <td>test-data/input/test-data-input-csv-reorder-fe...</td>\n",
       "      <td>test-data-input-csv-reorder-feb.csv</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>2011-02-09</td>\n",
       "      <td>200</td>\n",
       "      <td>-90</td>\n",
       "      <td>110</td>\n",
       "      <td>test-data/input/test-data-input-csv-reorder-fe...</td>\n",
       "      <td>test-data-input-csv-reorder-feb.csv</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>2011-02-10</td>\n",
       "      <td>200</td>\n",
       "      <td>-90</td>\n",
       "      <td>110</td>\n",
       "      <td>test-data/input/test-data-input-csv-reorder-fe...</td>\n",
       "      <td>test-data-input-csv-reorder-feb.csv</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2011-01-01</td>\n",
       "      <td>100</td>\n",
       "      <td>-80</td>\n",
       "      <td>20</td>\n",
       "      <td>test-data/input/test-data-input-csv-reorder-ja...</td>\n",
       "      <td>test-data-input-csv-reorder-jan.csv</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2011-01-02</td>\n",
       "      <td>100</td>\n",
       "      <td>-80</td>\n",
       "      <td>20</td>\n",
       "      <td>test-data/input/test-data-input-csv-reorder-ja...</td>\n",
       "      <td>test-data-input-csv-reorder-jan.csv</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2011-01-03</td>\n",
       "      <td>100</td>\n",
       "      <td>-80</td>\n",
       "      <td>20</td>\n",
       "      <td>test-data/input/test-data-input-csv-reorder-ja...</td>\n",
       "      <td>test-data-input-csv-reorder-jan.csv</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2011-01-04</td>\n",
       "      <td>100</td>\n",
       "      <td>-80</td>\n",
       "      <td>20</td>\n",
       "      <td>test-data/input/test-data-input-csv-reorder-ja...</td>\n",
       "      <td>test-data-input-csv-reorder-jan.csv</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2011-01-05</td>\n",
       "      <td>100</td>\n",
       "      <td>-80</td>\n",
       "      <td>20</td>\n",
       "      <td>test-data/input/test-data-input-csv-reorder-ja...</td>\n",
       "      <td>test-data-input-csv-reorder-jan.csv</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>2011-01-06</td>\n",
       "      <td>100</td>\n",
       "      <td>-80</td>\n",
       "      <td>20</td>\n",
       "      <td>test-data/input/test-data-input-csv-reorder-ja...</td>\n",
       "      <td>test-data-input-csv-reorder-jan.csv</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>2011-01-07</td>\n",
       "      <td>100</td>\n",
       "      <td>-80</td>\n",
       "      <td>20</td>\n",
       "      <td>test-data/input/test-data-input-csv-reorder-ja...</td>\n",
       "      <td>test-data-input-csv-reorder-jan.csv</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>2011-01-08</td>\n",
       "      <td>100</td>\n",
       "      <td>-80</td>\n",
       "      <td>20</td>\n",
       "      <td>test-data/input/test-data-input-csv-reorder-ja...</td>\n",
       "      <td>test-data-input-csv-reorder-jan.csv</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>2011-01-09</td>\n",
       "      <td>100</td>\n",
       "      <td>-80</td>\n",
       "      <td>20</td>\n",
       "      <td>test-data/input/test-data-input-csv-reorder-ja...</td>\n",
       "      <td>test-data-input-csv-reorder-jan.csv</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>2011-01-10</td>\n",
       "      <td>100</td>\n",
       "      <td>-80</td>\n",
       "      <td>20</td>\n",
       "      <td>test-data/input/test-data-input-csv-reorder-ja...</td>\n",
       "      <td>test-data-input-csv-reorder-jan.csv</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2011-03-01</td>\n",
       "      <td>300</td>\n",
       "      <td>-100</td>\n",
       "      <td>200</td>\n",
       "      <td>test-data/input/test-data-input-csv-reorder-ma...</td>\n",
       "      <td>test-data-input-csv-reorder-mar.csv</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2011-03-02</td>\n",
       "      <td>300</td>\n",
       "      <td>-100</td>\n",
       "      <td>200</td>\n",
       "      <td>test-data/input/test-data-input-csv-reorder-ma...</td>\n",
       "      <td>test-data-input-csv-reorder-mar.csv</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2011-03-03</td>\n",
       "      <td>300</td>\n",
       "      <td>-100</td>\n",
       "      <td>200</td>\n",
       "      <td>test-data/input/test-data-input-csv-reorder-ma...</td>\n",
       "      <td>test-data-input-csv-reorder-mar.csv</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2011-03-04</td>\n",
       "      <td>300</td>\n",
       "      <td>-100</td>\n",
       "      <td>200</td>\n",
       "      <td>test-data/input/test-data-input-csv-reorder-ma...</td>\n",
       "      <td>test-data-input-csv-reorder-mar.csv</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2011-03-05</td>\n",
       "      <td>300</td>\n",
       "      <td>-100</td>\n",
       "      <td>200</td>\n",
       "      <td>test-data/input/test-data-input-csv-reorder-ma...</td>\n",
       "      <td>test-data-input-csv-reorder-mar.csv</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>2011-03-06</td>\n",
       "      <td>300</td>\n",
       "      <td>-100</td>\n",
       "      <td>200</td>\n",
       "      <td>test-data/input/test-data-input-csv-reorder-ma...</td>\n",
       "      <td>test-data-input-csv-reorder-mar.csv</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>2011-03-07</td>\n",
       "      <td>300</td>\n",
       "      <td>-100</td>\n",
       "      <td>200</td>\n",
       "      <td>test-data/input/test-data-input-csv-reorder-ma...</td>\n",
       "      <td>test-data-input-csv-reorder-mar.csv</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>2011-03-08</td>\n",
       "      <td>300</td>\n",
       "      <td>-100</td>\n",
       "      <td>200</td>\n",
       "      <td>test-data/input/test-data-input-csv-reorder-ma...</td>\n",
       "      <td>test-data-input-csv-reorder-mar.csv</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>2011-03-09</td>\n",
       "      <td>300</td>\n",
       "      <td>-100</td>\n",
       "      <td>200</td>\n",
       "      <td>test-data/input/test-data-input-csv-reorder-ma...</td>\n",
       "      <td>test-data-input-csv-reorder-mar.csv</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>2011-03-10</td>\n",
       "      <td>300</td>\n",
       "      <td>-100</td>\n",
       "      <td>200</td>\n",
       "      <td>test-data/input/test-data-input-csv-reorder-ma...</td>\n",
       "      <td>test-data-input-csv-reorder-mar.csv</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         date  sales  cost  profit                                           filepath                             filename\n",
       "0  2011-02-01    200   -90     110  test-data/input/test-data-input-csv-reorder-fe...  test-data-input-csv-reorder-feb.csv\n",
       "1  2011-02-02    200   -90     110  test-data/input/test-data-input-csv-reorder-fe...  test-data-input-csv-reorder-feb.csv\n",
       "2  2011-02-03    200   -90     110  test-data/input/test-data-input-csv-reorder-fe...  test-data-input-csv-reorder-feb.csv\n",
       "3  2011-02-04    200   -90     110  test-data/input/test-data-input-csv-reorder-fe...  test-data-input-csv-reorder-feb.csv\n",
       "4  2011-02-05    200   -90     110  test-data/input/test-data-input-csv-reorder-fe...  test-data-input-csv-reorder-feb.csv\n",
       "5  2011-02-06    200   -90     110  test-data/input/test-data-input-csv-reorder-fe...  test-data-input-csv-reorder-feb.csv\n",
       "6  2011-02-07    200   -90     110  test-data/input/test-data-input-csv-reorder-fe...  test-data-input-csv-reorder-feb.csv\n",
       "7  2011-02-08    200   -90     110  test-data/input/test-data-input-csv-reorder-fe...  test-data-input-csv-reorder-feb.csv\n",
       "8  2011-02-09    200   -90     110  test-data/input/test-data-input-csv-reorder-fe...  test-data-input-csv-reorder-feb.csv\n",
       "9  2011-02-10    200   -90     110  test-data/input/test-data-input-csv-reorder-fe...  test-data-input-csv-reorder-feb.csv\n",
       "0  2011-01-01    100   -80      20  test-data/input/test-data-input-csv-reorder-ja...  test-data-input-csv-reorder-jan.csv\n",
       "1  2011-01-02    100   -80      20  test-data/input/test-data-input-csv-reorder-ja...  test-data-input-csv-reorder-jan.csv\n",
       "2  2011-01-03    100   -80      20  test-data/input/test-data-input-csv-reorder-ja...  test-data-input-csv-reorder-jan.csv\n",
       "3  2011-01-04    100   -80      20  test-data/input/test-data-input-csv-reorder-ja...  test-data-input-csv-reorder-jan.csv\n",
       "4  2011-01-05    100   -80      20  test-data/input/test-data-input-csv-reorder-ja...  test-data-input-csv-reorder-jan.csv\n",
       "5  2011-01-06    100   -80      20  test-data/input/test-data-input-csv-reorder-ja...  test-data-input-csv-reorder-jan.csv\n",
       "6  2011-01-07    100   -80      20  test-data/input/test-data-input-csv-reorder-ja...  test-data-input-csv-reorder-jan.csv\n",
       "7  2011-01-08    100   -80      20  test-data/input/test-data-input-csv-reorder-ja...  test-data-input-csv-reorder-jan.csv\n",
       "8  2011-01-09    100   -80      20  test-data/input/test-data-input-csv-reorder-ja...  test-data-input-csv-reorder-jan.csv\n",
       "9  2011-01-10    100   -80      20  test-data/input/test-data-input-csv-reorder-ja...  test-data-input-csv-reorder-jan.csv\n",
       "0  2011-03-01    300  -100     200  test-data/input/test-data-input-csv-reorder-ma...  test-data-input-csv-reorder-mar.csv\n",
       "1  2011-03-02    300  -100     200  test-data/input/test-data-input-csv-reorder-ma...  test-data-input-csv-reorder-mar.csv\n",
       "2  2011-03-03    300  -100     200  test-data/input/test-data-input-csv-reorder-ma...  test-data-input-csv-reorder-mar.csv\n",
       "3  2011-03-04    300  -100     200  test-data/input/test-data-input-csv-reorder-ma...  test-data-input-csv-reorder-mar.csv\n",
       "4  2011-03-05    300  -100     200  test-data/input/test-data-input-csv-reorder-ma...  test-data-input-csv-reorder-mar.csv\n",
       "5  2011-03-06    300  -100     200  test-data/input/test-data-input-csv-reorder-ma...  test-data-input-csv-reorder-mar.csv\n",
       "6  2011-03-07    300  -100     200  test-data/input/test-data-input-csv-reorder-ma...  test-data-input-csv-reorder-mar.csv\n",
       "7  2011-03-08    300  -100     200  test-data/input/test-data-input-csv-reorder-ma...  test-data-input-csv-reorder-mar.csv\n",
       "8  2011-03-09    300  -100     200  test-data/input/test-data-input-csv-reorder-ma...  test-data-input-csv-reorder-mar.csv\n",
       "9  2011-03-10    300  -100     200  test-data/input/test-data-input-csv-reorder-ma...  test-data-input-csv-reorder-mar.csv"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# consistent format\n",
    "ddf = dd.read_csv('test-data/output/d6tstack-test-data-input-csv-reorder-*.csv')\n",
    "ddf.compute()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}