experiments/04_PlanetKaggle_GPU.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Experiment 04: Amazon Planet (GPU version)\n",
    "\n",
    "This experiment uses the data from the Kaggle competition [Planet: Understanding the Amazon from Space](https://www.kaggle.com/c/planet-understanding-the-amazon-from-space/leaderboard). Here we use a pretrained ResNet50 model to generate the features from the dataset.\n",
    "\n",
    "The details of the machine we used and the version of the libraries can be found in [experiment 01](01_airline.ipynb)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "System version: 3.5.2 |Anaconda custom (64-bit)| (default, Jul  2 2016, 17:53:06) \n",
      "[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)]\n",
      "XGBoost version: 0.6\n",
      "LightGBM version: 0.2\n"
     ]
    }
   ],
   "source": [
    "import sys, os\n",
    "from collections import defaultdict\n",
    "import numpy as np\n",
    "import pkg_resources\n",
    "from libs.loaders import load_planet_kaggle\n",
    "from libs.planet_kaggle import threshold_prediction\n",
    "from libs.timer import Timer\n",
    "import lightgbm as lgb\n",
    "import xgboost as xgb\n",
    "from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score\n",
    "from tqdm import tqdm\n",
    "import tensorflow as tf\n",
    "from keras.backend.tensorflow_backend import set_session, get_session\n",
    "\n",
    "print(\"System version: {}\".format(sys.version))\n",
    "print(\"XGBoost version: {}\".format(pkg_resources.get_distribution('xgboost').version))\n",
    "print(\"LightGBM version: {}\".format(pkg_resources.get_distribution('lightgbm').version))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "env: MOUNT_POINT=/datadrive\n"
     ]
    }
   ],
   "source": [
    "%env MOUNT_POINT=/datadrive"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "#Configure TF to use only one GPU, by default TF allocates memory in all GPUs\n",
    "config = tf.ConfigProto(device_count = {'GPU': 1})\n",
    "#Configure TF to limit the amount of GPU memory, by default TF takes all of them. \n",
    "config.gpu_options.per_process_gpu_memory_fraction = 0.3\n",
    "set_session(tf.Session(config=config))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "The images are loaded and featurised using a pretrained ResNet50 model available from Keras"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "X_train, y_train, X_test, y_test = load_planet_kaggle()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(35000, 2048)\n",
      "(35000, 17)\n",
      "(5479, 2048)\n",
      "(5479, 17)\n"
     ]
    }
   ],
   "source": [
    "print(X_train.shape)\n",
    "print(y_train.shape)\n",
    "print(X_test.shape)\n",
    "print(y_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## XGBoost "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "We will use a one-v-rest. So each classifier will be responsible for determining whether the assigned tag applies to the image"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "def train_and_validate_xgboost(params, train_features, train_labels, validation_features, num_boost_round):\n",
    "    n_classes = train_labels.shape[1]\n",
    "    y_val_pred = np.zeros((validation_features.shape[0], n_classes))\n",
    "    time_results = defaultdict(list)\n",
    "    for class_i in tqdm(range(n_classes)):\n",
    "        dtrain = xgb.DMatrix(data=train_features, label=train_labels[:, class_i])\n",
    "        dtest = xgb.DMatrix(data=validation_features)\n",
    "        with Timer() as t:\n",
    "            model = xgb.train(params, dtrain, num_boost_round=num_boost_round)\n",
    "        time_results['train_time'].append(t.interval)\n",
    "        \n",
    "        with Timer() as t:\n",
    "            y_val_pred[:, class_i] = model.predict(dtest)\n",
    "        time_results['test_time'].append(t.interval)\n",
    "        \n",
    "    return y_val_pred, time_results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "def train_and_validate_lightgbm(params, train_features, train_labels, validation_features, num_boost_round):\n",
    "    n_classes = train_labels.shape[1]\n",
    "    y_val_pred = np.zeros((validation_features.shape[0], n_classes))\n",
    "    time_results = defaultdict(list)\n",
    "    for class_i in tqdm(range(n_classes)):\n",
    "        lgb_train = lgb.Dataset(train_features, train_labels[:, class_i], free_raw_data=False)\n",
    "        with Timer() as t:\n",
    "            model = lgb.train(params, lgb_train, num_boost_round = num_boost_round)\n",
    "        time_results['train_time'].append(t.interval)\n",
    "        \n",
    "        with Timer() as t:\n",
    "            y_val_pred[:, class_i] = model.predict(validation_features)\n",
    "        time_results['test_time'].append(t.interval)\n",
    "        \n",
    "    return y_val_pred, time_results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "metrics_dict = {\n",
    "    'Accuracy': accuracy_score,\n",
    "    'Precision': lambda y_true, y_pred: precision_score(y_true, y_pred, average='samples'),\n",
    "    'Recall': lambda y_true, y_pred: recall_score(y_true, y_pred, average='samples'),\n",
    "    'F1': lambda y_true, y_pred: f1_score(y_true, y_pred, average='samples'),\n",
    "}\n",
    "\n",
    "def classification_metrics(metrics, y_true, y_pred):\n",
    "    return {metric_name:metric(y_true, y_pred) for metric_name, metric in metrics.items()}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "results_dict = dict()\n",
    "num_rounds = 50"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Now we are going to define the different models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "xgb_params = {'max_depth':2, #'max_depth':6 \n",
    "              'objective':'binary:logistic', \n",
    "              'min_child_weight':1, \n",
    "              'learning_rate':0.1, \n",
    "              'scale_pos_weight':2, \n",
    "              'gamma':0.1, \n",
    "              'reg_lamda':1, \n",
    "              'subsample':1,\n",
    "              'tree_method':'exact', \n",
    "              'updater':'grow_gpu',\n",
    "             }"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "*NOTE: We got an out of memory error with xgb. Please see the comments at the end of the notebook.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "y_pred, timing_results = train_and_validate_xgboost(xgb_params, X_train, y_train, X_test, num_boost_round=num_rounds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "results_dict['xgb']={\n",
    "    'train_time': np.sum(timing_results['train_time']),\n",
    "    'test_time': np.sum(timing_results['test_time']),\n",
    "    'performance': classification_metrics(metrics_dict, \n",
    "                                          y_test, \n",
    "                                          threshold_prediction(y_pred, threshold=0.1)) \n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "\n",
    "\n",
    "Now let's try with XGBoost histogram.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "xgb_hist_params = {'max_depth':0, \n",
    "                  'max_leaves':2**6, \n",
    "                  'objective':'binary:logistic', \n",
    "                  'min_child_weight':1, \n",
    "                  'learning_rate':0.1, \n",
    "                  'scale_pos_weight':2, \n",
    "                  'gamma':0.1, \n",
    "                  'reg_lamda':1, \n",
    "                  'subsample':1,\n",
    "                  'tree_method':'hist', \n",
    "                  'grow_policy':'lossguide',\n",
    "                  'updater':'grow_gpu_hist',\n",
    "                  'max_bins': 63\n",
    "                 }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "  0%|          | 0/17 [00:00<?, ?it/s]\u001b[A\n",
      "100%|██████████| 17/17 [35:08<00:00, 115.04s/it]\n"
     ]
    }
   ],
   "source": [
    "y_pred, timing_results = train_and_validate_xgboost(xgb_hist_params, X_train, y_train, X_test, num_boost_round=num_rounds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "results_dict['xgb_hist']={\n",
    "    'train_time': np.sum(timing_results['train_time']),\n",
    "    'test_time': np.sum(timing_results['test_time']),\n",
    "    'performance': classification_metrics(metrics_dict, \n",
    "                                          y_test, \n",
    "                                          threshold_prediction(y_pred, threshold=0.1)) \n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## LightGBM \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "lgb_params = {'num_leaves': 2**6,\n",
    "             'learning_rate': 0.1,\n",
    "             'scale_pos_weight': 2,\n",
    "             'min_split_gain': 0.1,\n",
    "             'min_child_weight': 1,\n",
    "             'reg_lambda': 1,\n",
    "             'subsample': 1,\n",
    "             'objective':'binary',\n",
    "             'device': 'gpu',\n",
    "             'task': 'train',\n",
    "             'max_bin': 63\n",
    "             }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 17/17 [05:25<00:00, 15.56s/it]\n"
     ]
    }
   ],
   "source": [
    "y_pred, timing_results = train_and_validate_lightgbm(lgb_params, X_train, y_train, X_test, num_boost_round=num_rounds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "results_dict['lgbm']={\n",
    "    'train_time': np.sum(timing_results['train_time']),\n",
    "    'test_time': np.sum(timing_results['test_time']),\n",
    "    'performance': classification_metrics(metrics_dict, \n",
    "                                          y_test, \n",
    "                                          threshold_prediction(y_pred, threshold=0.1)) \n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Finally, we show the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "    \"lgbm\": {\n",
      "        \"performance\": {\n",
      "            \"Accuracy\": 0.3719656871691915,\n",
      "            \"F1\": 0.8219281534713183,\n",
      "            \"Precision\": 0.7435648956911961,\n",
      "            \"Recall\": 0.9733034790846435\n",
      "        },\n",
      "        \"test_time\": 0.2703422480117297,\n",
      "        \"train_time\": 317.68381078500533\n",
      "    },\n",
      "    \"xgb_hist\": {\n",
      "        \"performance\": {\n",
      "            \"Accuracy\": 0.37871874429640445,\n",
      "            \"F1\": 0.8220252909027159,\n",
      "            \"Precision\": 0.7447899193746976,\n",
      "            \"Recall\": 0.9720717197264013\n",
      "        },\n",
      "        \"test_time\": 0.24103524297242984,\n",
      "        \"train_time\": 2028.4320792920043\n",
      "    }\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "# Results\n",
    "print(json.dumps(results_dict, indent=4, sort_keys=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "In this dataset we have a big feature size, 2048. When using the standard version of XGBoost, xgb, we get an out of memory using a NVIDIA M60 GPU, even if we reduce the max depth of the tree to 2. A solution to this issue would be to reduce the feature size. One option could be using PCA and another could be to use a different featurizer, instead of ResNet whose last hidden layer has 2048 units, we could use VGG, [also provided by Keras](https://github.com/fchollet/keras/blob/master/keras/applications/vgg16.py), whose last hidden layer has 512 units. \n",
    "\n",
    "As it can be seen, LightGBM is faster than XGBoost, but in this case the speed is lower than in the CPU version. The GPU implementation cannot always speed up the training, since it has some additional cost of memory copy between CPU and GPU. So when the data size is small and the number of features is large, the GPU version will be slower."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.5",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}