example/ex_store_data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# What we are going to do\n",
    "In the tutorial, I will show you how to use ezHDF to save two csv files (data1.csv and data2.csv in this folder) into HDF format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/shutingpi/anaconda3/lib/python3.6/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.\n",
      "  from ._conv import register_converters as _register_converters\n"
     ]
    }
   ],
   "source": [
    "import sys; sys.path.insert(0,'/Users/shutingpi/Dropbox/ezHDF')\n",
    "from ezHDF.ezHDF import ezHDF\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# use pandas to create a chunk reader of the cvs files\n",
    "assuming we are going to read large files (much, much, larger than your RAM), it is impossile to load the data at once. What you can do it to read a chunk of data each time. It can be easily achieved using pandas with the parameter \"chunksize\". "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create chunk reader object\n",
    "wkdir = '/Users/shutingpi/Dropbox/ezHDF/Example'\n",
    "reader1 = pd.read_csv(wkdir+'/data1.csv', sep = ',', engine = 'c', \n",
    "                    error_bad_lines=False, warn_bad_lines=False, index_col = False,\n",
    "                    chunksize= 100)\n",
    "\n",
    "reader2 = pd.read_csv(wkdir+'/data2.csv', sep = ',', engine = 'c', \n",
    "                    error_bad_lines=False, warn_bad_lines=False, index_col = False,\n",
    "                    chunksize= 100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# create a exHDF.hdf_store object\n",
    "ok, now let's define a hdf_store object. It is the major object to handle your data. mode can be 'w', 'r', 'a', as [h5py file object](http://docs.h5py.org/en/latest/high/file.html) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create hdf_store object\n",
    "store = ezHDF(wkdir = wkdir, hdf_name = 'my_hdf.h5', mode = 'w')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# set parameters for the dataset\n",
    "To use ezHDF, we have to manually input the column name and data type of each column. \n",
    "\n",
    "When assigning the data type, use 's' for string, 'i' for integer and 'f' for float. You may also want to input the working directory so that ezHDF can work in the correct folder\n",
    "\n",
    "**Note that column names don't necessary equal to what shown in your csv file. It can be arbiraty names. However the data type must be consistent with your raw data**. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# define some parameters\n",
    "col_name1 = ['str0','int1','float2','float3','str4','str5','str6']\n",
    "dtype1 = ['s','i','f','f','s','s','s']\n",
    "col_name2 = ['str0','int1','str2','str3','float4','float5','str6']\n",
    "dtype2 = ['s','i','s','s','f','f','s']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# create new data sets in the object to store your data\n",
    "when create a new data set, you will also need to provide a parameter called \"container_size\". When create a dataset in ezHDF, it will comes with a \"size\", i.e. the number of rows. In the beginning, all the rows are empty now. If you append new data smaller than the container_size, ezHDF won't need to request more size from the disk. If you append new data to a dataset that makes the container_size insufficient, ezHDF will **\"automatically\" (yes, you don't have to do it manually)** request more space from the disk to increase the container_size which will make your code slower. Therefore, it is suggest to set a initial container_size slightly larger than your data. \n",
    "\n",
    "Since we have 10K rows in data1.csv and 20K rows in data2.csv, we should set an initial container_size equal or slight largr than that."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "store.new_dataset(ds_name = 'data1', container_size = 12000, column_names = col_name1, column_dtype = dtype1)\n",
    "store.new_dataset(ds_name = 'data2', container_size = 22000, column_names = col_name2, column_dtype = dtype2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# get dataset list\n",
    "you can get all existing datasets using get_dataset_list() "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['data1', 'data2']"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "store.get_dataset_list()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# check info about the object\n",
    "You can get basic information of the hdf_store object. Here n_rows means how many rows of data have been stored. n_container means the size of the container. Since we have put any data in the dataset, n_rows =0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "--- ezHDF hdf_store info ---\n",
      "\n",
      "dataset name: data1\n",
      "column names:\n",
      "   ['str0', 'int1', 'float2', 'float3', 'str4', 'str5', 'str6']\n",
      "column dtype:[ s,i,f,f,s,s,s ]\n",
      "n_rows: 0\n",
      "n_container: 12000\n",
      "\n",
      "dataset name: data2\n",
      "column names:\n",
      "   ['str0', 'int1', 'str2', 'str3', 'float4', 'float5', 'str6']\n",
      "column dtype:[ s,i,s,s,f,f,s ]\n",
      "n_rows: 0\n",
      "n_container: 22000\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# show object information\n",
    "store.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# resize the container\n",
    "you can also reassign the size of the container if you are not satifity current container size. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dataset (data1) now resized to (15000) rows\n",
      "dataset (data2) now resized to (25000) rows\n"
     ]
    }
   ],
   "source": [
    "# resize container size\n",
    "store.resize('data1', 15000)\n",
    "store.resize('data2', 25000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# check information agian\n",
    "you can find the container sizes are 15K and 25K respectively."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "--- ezHDF hdf_store info ---\n",
      "\n",
      "dataset name: data1\n",
      "column names:\n",
      "   ['str0', 'int1', 'float2', 'float3', 'str4', 'str5', 'str6']\n",
      "column dtype:[ s,i,f,f,s,s,s ]\n",
      "n_rows: 0\n",
      "n_container: 15000\n",
      "\n",
      "dataset name: data2\n",
      "column names:\n",
      "   ['str0', 'int1', 'str2', 'str3', 'float4', 'float5', 'str6']\n",
      "column dtype:[ s,i,s,s,f,f,s ]\n",
      "n_rows: 0\n",
      "n_container: 25000\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# show object information\n",
    "store.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# put data to each dataset\n",
    "you can use hdf_store.append_to_dataset to put your data in the dataset. Note that, if you pandas data frame uses row index as a single column, you have to drop it before append to dataset.   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['data1', 'data2']"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# get all dataset list\n",
    "store.get_dataset_list()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# put data1 into dataset data1\n",
    "for chunk in reader1:\n",
    "    chunk = chunk.drop(chunk.columns[0], axis = 1)\n",
    "    store.append(ds_name = 'data1', data = chunk)\n",
    "\n",
    "# put data2 into dataset data2\n",
    "for chunk in reader2:\n",
    "    chunk = chunk.drop(chunk.columns[0], axis = 1)\n",
    "    store.append(ds_name = 'data2', data = chunk)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# check information again\n",
    "check info again, you will find n_row has been changed for each dataset because we alrady put some data (10K and 20L) in the datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "--- ezHDF hdf_store info ---\n",
      "\n",
      "dataset name: data1\n",
      "column names:\n",
      "   ['str0', 'int1', 'float2', 'float3', 'str4', 'str5', 'str6']\n",
      "column dtype:[ s,i,f,f,s,s,s ]\n",
      "n_rows: 10000\n",
      "n_container: 15000\n",
      "\n",
      "dataset name: data2\n",
      "column names:\n",
      "   ['str0', 'int1', 'str2', 'str3', 'float4', 'float5', 'str6']\n",
      "column dtype:[ s,i,s,s,f,f,s ]\n",
      "n_rows: 20000\n",
      "n_container: 25000\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# check information of the HD file again, n_container > n_rows\n",
    "store.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# resize container\n",
    "once you have put all data in the dataset, it will be a waste if n_container > n_rows. You can either use hdf_store.resize() to resize the container manually or you can use hdf_store.auto_resize() to let ezHDF resize container size automatically. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dataset (data1) now resized to (10000) rows\n",
      "dataset (data2) now resized to (20000) rows\n"
     ]
    }
   ],
   "source": [
    "# let resize n_container = n_rows, to save storage size\n",
    "store.auto_resize(ds_name = 'data1')\n",
    "store.auto_resize(ds_name = 'data2')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "--- ezHDF hdf_store info ---\n",
      "\n",
      "dataset name: data1\n",
      "column names:\n",
      "   ['str0', 'int1', 'float2', 'float3', 'str4', 'str5', 'str6']\n",
      "column dtype:[ s,i,f,f,s,s,s ]\n",
      "n_rows: 10000\n",
      "n_container: 10000\n",
      "\n",
      "dataset name: data2\n",
      "column names:\n",
      "   ['str0', 'int1', 'str2', 'str3', 'float4', 'float5', 'str6']\n",
      "column dtype:[ s,i,s,s,f,f,s ]\n",
      "n_rows: 20000\n",
      "n_container: 20000\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# check info again, now container size is equal to n_rows\n",
    "store.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# close the file\n",
    "Remember to close the file if you no longer want to use it. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "store.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Ok, now you know everything about data storing"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}