00_DataProcessing.ipynb

{
    "cells": [
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "# Data Processing\nIn this notebook we convert the ImageNet data to the appropriate format so that we can use it for training.\n\nThe dataset has many versions, the one commonly used for image classification is ILSVRC 2012. Go to the [download page](http://www.image-net.org/download-images) (you may need to register an account), and find the page for ILSVRC2012. You will need to download two files ILSVRC2012_img_train.tar and ILSVRC2012_img_val.tar"
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": "from pathlib import Path"
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": "DATA=Path(\"/data\")"
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": "!mkdir -p {DATA/\"train\"}\n!tar -C {DATA/\"train\"} -xf {DATA/\"ILSVRC2012_img_train.tar\"}"
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": "import tarfile\nfrom tqdm import tqdm_notebook\nimport os"
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": "filenames = list((DATA/\"train\").glob(\"*.tar\"))\npbar = tqdm_notebook(total=len(filenames))\nfor class_tar in filenames:\n    pbar.set_description('Extracting '+class_tar.name+ '...')\n    class_dir = os.path.splitext(class_tar)[0]\n    os.mkdir(class_dir)\n    with tarfile.open(class_tar) as f:\n        f.extractall(class_dir)\n    os.remove(class_tar)\n    pbar.update(1)"
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": "!rm -r {DATA/\"validation\"}"
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": "!mkdir -p {DATA/\"validation\"}\n!tar -C {DATA/\"validation\"} -xf {DATA/\"ILSVRC2012_img_val.tar\"}"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "The validation data comes without labels so wee ned to run a script to asign the images to the appropriate classes."
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": "validation_path = DATA/\"validation\"\nvalidation_preparation_script = Path(os.getcwd())/\"valprep.sh\""
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": "!bash -c \"cd {validation_path} && {validation_preparation_script}\""
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "Finally we package the processed directories so that we can upload them quicker."
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": "!cd {DATA} && tar -czvf train.tar.gz train"
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": "!cd {DATA} && tar -czvf validation.tar.gz validation"
        }
    ],
    "metadata": {
        "jupytext": {
            "text_representation": {
                "extension": ".py",
                "format_name": "light",
                "format_version": "1.3",
                "jupytext_version": "0.8.6"
            }
        },
        "kernelspec": {
            "display_name": "Python 3",
            "language": "python",
            "name": "python3"
        }
    },
    "nbformat": 4,
    "nbformat_minor": 2
}