keypoint_description.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Keypoint(Patch) Description\n",
    "<a id='top_cell'></a>  \n",
    "This project will be all about defining and training a convolutional neural network to perform keypoint  description. \n",
    "PyTorch tutorials are available at here: [pytorch tutorials](https://github.com/yunjey/pytorch-tutorial)\n",
    "\n",
    "Today we will go through:\n",
    "### 1. [Load and visualize the data](#load_data_cell)\n",
    "### 2. [Build an example deep network](#build_network_cell)\n",
    "### 3. [Train the deep network](#train_network_cell)\n",
    "### 4. [Generate deep features](#generate_deep_features_cell)\n",
    "\n",
    "***\n",
    "\n",
    "We will use below dataset in this project:\n",
    "###  [The Photo Tourism dataset ](http://phototour.cs.washington.edu/patches/default.htm)\n",
    "\n",
    "It is also available in PyTorch torchvision datasets: [pytorch version](https://pytorch.org/docs/stable/_modules/torchvision/datasets/phototour.html#PhotoTour)\n",
    "\n",
    "This dataset consists of 1024 x 1024 bitmap (.bmp) images, each containing a 16 x 16 array of image patches. Here are some examples:\n",
    "\n",
    "<table><tr><td><img src='images/patches0001.bmp' width=68% ></td><td><img src='images/patches1482.bmp' width=68%></td></tr></table>    \n",
    "For details of how the scale and orientation is established, please see the paper:  \n",
    "<p class=\"style8\"><font size=\"2\">S. Winder and M. Brown. <strong>Learning Local Image \n",
    "\t\t\t\tDescriptors</strong>. To appear <i>International Conference on \n",
    "\t\t\t\tComputer Vision and Pattern Recognition (CVPR2007)</i> (</font><a href=\"http://research.microsoft.com/~swinder/papers/winder_brown_cvpr07.pdf\"><span class=\"style9\">pdf \n",
    "\t\t\t\t300Kb</span></a><font size=\"2\">)</font></p>\n",
    "\n",
    "\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from __future__ import division, print_function\n",
    "import glob\n",
    "import os\n",
    "import cv2\n",
    "import PIL\n",
    "import random\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import matplotlib.image as mpimg\n",
    "import torch\n",
    "import torch.nn.init\n",
    "import torch.nn as nn\n",
    "import torch.optim as optim\n",
    "import torch.backends.cudnn as cudnn\n",
    "import torch.nn.functional as F\n",
    "import torchvision.datasets as dset\n",
    "import torchvision.transforms as transforms\n",
    "from tqdm import tqdm\n",
    "from torch.autograd import Variable\n",
    "from copy import deepcopy, copy\n",
    "from config_profile import args\n",
    "from Utils import cv2_scale36, cv2_scale, np_reshape, np_reshape64\n",
    "from Utils import L2Norm, cv2_scale, np_reshape\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Check GPU availability, using nvidia-smi "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Since there are two GPUs on each pelican server, you can either select it as 0 or 1\n",
    "os.environ[\"CUDA_VISIBLE_DEVICES\"] = \"1\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "---\n",
    "<a id='load_data_cell'></a>\n",
    "## Load and visualize the data\n",
    "\n",
    "In this section, we will \n",
    "#### 1. [Define a PyTorch dataset](#pytorch_dataset_cell)\n",
    "#### 2. [Define a PyTorch dataloader](#pytorch_dataloader_cell)\n",
    "#### 3. [Load data](#load_dataset_cell)\n",
    "#### 4. [Visualizaiton of the Training and Testing Data](#visualize_dataset_cell)\n",
    "\n",
    "[BackToTop](#top_cell)\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='pytorch_dataset_cell'></a>\n",
    "### Define PyTorch dataset\n",
    "\n",
    "[BackToSection](#load_data_cell)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class TripletPhotoTour(dset.PhotoTour):\n",
    "    \"\"\"\n",
    "    From the PhotoTour Dataset it generates triplet samples\n",
    "    note: a triplet is composed by a pair of matching images and one of\n",
    "    different class.\n",
    "    \"\"\"\n",
    "    def __init__(self, train=True, transform=None, batch_size = None,load_random_triplets = False,  *arg, **kw):\n",
    "        super(TripletPhotoTour, self).__init__(*arg, **kw)\n",
    "        self.transform = transform\n",
    "        self.out_triplets = load_random_triplets\n",
    "        self.train = train\n",
    "        self.n_triplets = args.n_triplets\n",
    "        \n",
    "        \n",
    "        self.batch_size = batch_size\n",
    "        self.triplets = self.generate_triplets(self.labels, self.n_triplets)\n",
    "            \n",
    "    @staticmethod\n",
    "    def generate_triplets(labels, num_triplets):\n",
    "        def create_indices(_labels):\n",
    "            inds = dict()\n",
    "            for idx, ind in enumerate(_labels):\n",
    "                if ind not in inds:\n",
    "                    inds[ind] = []\n",
    "                inds[ind].append(idx)\n",
    "            return inds\n",
    "\n",
    "        triplets = []\n",
    "        indices = create_indices(labels.numpy())\n",
    "        unique_labels = np.unique(labels.numpy())\n",
    "        n_classes = unique_labels.shape[0]\n",
    "        # add only unique indices in batch\n",
    "        already_idxs = set()\n",
    "\n",
    "        for x in tqdm(range(num_triplets)):\n",
    "            if len(already_idxs) >= args.batch_size:\n",
    "                already_idxs = set()\n",
    "            c1 = np.random.randint(0, n_classes)\n",
    "            while c1 in already_idxs:\n",
    "                c1 = np.random.randint(0, n_classes)\n",
    "            already_idxs.add(c1)\n",
    "            c2 = np.random.randint(0, n_classes)\n",
    "            while c1 == c2:\n",
    "                c2 = np.random.randint(0, n_classes)\n",
    "            if len(indices[c1]) == 2:  # hack to speed up process\n",
    "                n1, n2 = 0, 1\n",
    "            else:\n",
    "                n1 = np.random.randint(0, len(indices[c1]))\n",
    "                n2 = np.random.randint(0, len(indices[c1]))\n",
    "                while n1 == n2:\n",
    "                    n2 = np.random.randint(0, len(indices[c1]))\n",
    "            n3 = np.random.randint(0, len(indices[c2]))\n",
    "            triplets.append([indices[c1][n1], indices[c1][n2], indices[c2][n3]])\n",
    "        return torch.LongTensor(np.array(triplets))\n",
    "\n",
    "    def __getitem__(self, index):\n",
    "        def transform_img(img):\n",
    "            if self.transform is not None:\n",
    "                img = self.transform(img.numpy())\n",
    "            return img\n",
    "\n",
    "        t = self.triplets[index]\n",
    "        a, p, n = self.data[t[0]], self.data[t[1]], self.data[t[2]]\n",
    "\n",
    "        img_a = transform_img(a)\n",
    "        img_p = transform_img(p)\n",
    "        img_n = None\n",
    "        if self.out_triplets:\n",
    "            img_n = transform_img(n)\n",
    "        # transform images if required\n",
    "        if args.fliprot:\n",
    "            do_flip = random.random() > 0.5\n",
    "            do_rot = random.random() > 0.5\n",
    "            if do_rot:\n",
    "                img_a = img_a.permute(0,2,1)\n",
    "                img_p = img_p.permute(0,2,1)\n",
    "                if self.out_triplets:\n",
    "                    img_n = img_n.permute(0,2,1)\n",
    "            if do_flip:\n",
    "                img_a = torch.from_numpy(deepcopy(img_a.numpy()[:,:,::-1]))\n",
    "                img_p = torch.from_numpy(deepcopy(img_p.numpy()[:,:,::-1]))\n",
    "                if self.out_triplets:\n",
    "                    img_n = torch.from_numpy(deepcopy(img_n.numpy()[:,:,::-1]))\n",
    "        return (img_a, img_p, img_n)\n",
    "        \n",
    "\n",
    "    def __len__(self):\n",
    "        return self.triplets.size(0)\n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='pytorch_dataloader_cell'></a>\n",
    "### Define the dataloader\n",
    "[BackToSection](#load_data_cell)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_loaders(dataset_names, load_random_triplets = False, verbose=False):\n",
    "    \"\"\"\n",
    "    For training, we use dataset 'liberty';\n",
    "    For testing, we use dataset 'notredame' and 'yosemite'\n",
    "    \n",
    "    \"\"\"\n",
    "    test_dataset_names = copy(dataset_names)\n",
    "    test_dataset_names.remove(args.training_set)\n",
    "\n",
    "    kwargs = {'num_workers': args.num_workers, 'pin_memory': args.pin_memory} if args.cuda else {}\n",
    "\n",
    "    np_reshape64 = lambda x: np.reshape(x, (64, 64, 1))\n",
    "    transform_test = transforms.Compose([\n",
    "            transforms.Lambda(np_reshape64),\n",
    "            transforms.ToPILImage(),\n",
    "            transforms.Resize(32),\n",
    "            transforms.ToTensor()])\n",
    "    transform_train = transforms.Compose([\n",
    "            transforms.Lambda(np_reshape64),\n",
    "            transforms.ToPILImage(),\n",
    "            transforms.RandomRotation(5,PIL.Image.BILINEAR),\n",
    "            transforms.RandomResizedCrop(32, scale = (0.9,1.0),ratio = (0.9,1.1)),\n",
    "            transforms.Resize(32),\n",
    "            transforms.ToTensor()])\n",
    "    transform = transforms.Compose([\n",
    "            transforms.Lambda(cv2_scale),\n",
    "            transforms.Lambda(np_reshape),\n",
    "            transforms.ToTensor(),\n",
    "            transforms.Normalize((args.mean_image,), (args.std_image,))])\n",
    "    if not args.augmentation:\n",
    "        transform_train = transform\n",
    "        transform_test = transform\n",
    "    train_loader = torch.utils.data.DataLoader(\n",
    "            TripletPhotoTour(train=True,\n",
    "                             load_random_triplets = load_random_triplets,\n",
    "                             batch_size=args.batch_size,\n",
    "                             root=args.dataroot,\n",
    "                             name=args.training_set,\n",
    "                             download=True,\n",
    "                             transform=transform_train),\n",
    "                             batch_size=args.batch_size,\n",
    "                             shuffle=False, **kwargs)\n",
    "\n",
    "    test_loaders = [{'name': name,\n",
    "                     'dataloader': torch.utils.data.DataLoader(\n",
    "             TripletPhotoTour(train=False,\n",
    "                     batch_size=args.test_batch_size,\n",
    "                     load_random_triplets = load_random_triplets,\n",
    "                     root=args.dataroot,\n",
    "                     name=name,\n",
    "                     download=True,\n",
    "                     transform=transform_test),\n",
    "                        batch_size=args.test_batch_size,\n",
    "                        shuffle=False, **kwargs)}\n",
    "                    for name in test_dataset_names]\n",
    "\n",
    "    return train_loader, test_loaders[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "<a id='load_dataset_cell'></a>\n",
    "\n",
    "### Load Data\n",
    "\n",
    "Load the Photo Tourism dataset by PyTorch. Below line (function 'create_loader') will help you to download the dataset to your directory. The data dir and other configuration setings are specified in config_profile.py.\n",
    "\n",
    "[BackToSection](#load_data_cell)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "dataset_names = ['liberty', 'notredame']\n",
    "\n",
    "args.n_triplets = 100000    # for illustration, here we only use 5000 triples; in your experiment, set it as 100000\n",
    "args.epochs = 60             # in your experiment, set it as 60; For CNN1, it will take ~ 1hr20mins if n_triplets = 100000\n",
    "\n",
    "train_loader, validation_loader = create_loaders(dataset_names, load_random_triplets = args.load_random_triplets)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "<a id='visualize_dataset_cell'></a>\n",
    "##  Visualizaiton of the Training and Testing Data\n",
    "Below are some examples of patches in this dataset.\n",
    "\n",
    "[BackToSection](#load_data_cell)\n",
    "\n",
    "#### Data in Training\n",
    "In the training phase, the input data is a batch of patch pairs: X = {(patch_a, patch_p)}, which represents the anchor patch and the positive patch, respectively. \n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_examples(sample_batched, n_samples=3, labels=['A', 'P', 'N']):\n",
    "    \n",
    "    cols = ['Sample {}'.format(col) for col in range(0, n_samples)]\n",
    "    rows = ['Patch {}'.format(row) for row in labels]\n",
    "    nrow = len(rows)\n",
    "    \n",
    "    fig, axes = plt.subplots(nrows=len(rows), ncols=n_samples, figsize=(12, 8))\n",
    "    for ax, col in zip(axes[0], cols):\n",
    "        ax.set_title(col)\n",
    "\n",
    "    for ax, row in zip(axes[:,0], rows):\n",
    "        ax.set_ylabel(row, rotation=90, size='large')\n",
    "\n",
    "#     for idx, img_tensor in enumerate(sample_batched):\n",
    "    for idx in range(nrow):\n",
    "        img_tensor = sample_batched[idx]\n",
    "        for jdx in range(n_samples):\n",
    "            img = img_tensor[jdx, 0]\n",
    "            axes[idx][jdx].imshow(img, cmap='gray')\n",
    "\n",
    "    fig.tight_layout()\n",
    "    plt.show()\n",
    "\n",
    "for i_batch, sample_batched in enumerate(train_loader):\n",
    "    print(\"In training and validation, each data entry generates {} elements: anchor, positive, and negative.\".format(len(sample_batched)))\n",
    "    print(\"Each of them have the size of: {}\".format(sample_batched[0].shape))\n",
    "    print(\"Below we show in each column one triplet: top row shows patch a; mid row shows patch p; and bot row shows patch n.\")\n",
    "          \n",
    "    if i_batch == 0:\n",
    "        plot_examples(sample_batched, 3)\n",
    "        break\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "---\n",
    "<a id='build_network_cell'></a>\n",
    "## Build an exmaple deep network  \n",
    "\n",
    "In this section, we will:\n",
    "#### 1. [Build the deep network: DesNet](#build_desNet_cell)\n",
    "#### 2. [Setup optimization](#set_opt_cell)\n",
    "\n",
    "[BackToTop](#top_cell)\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='build_desNet_cell'></a>\n",
    "### Build the deep network: DesNet\n",
    "The DesNet is a simple CNN network, which only contains two CNN blocks.\n",
    "\n",
    "\n",
    "[BackToSection](#build_network_cell)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load network from the python file. You need to submit these .py files to TA\n",
    "#from CNN1 import DesNet       # uncomment this line if you are using DesNet from CNN1.py\n",
    "from CNN3 import DesNet      # uncomment this line if you are using DesNet from CNN2.py\n",
    "#from CNN3 import DesNet      # uncomment this line if you are using DesNet from CNN3.py\n",
    "\n",
    "test_model = DesNet()\n",
    "# check model architecture\n",
    "\n",
    "print(model)\n",
    "\n",
    "if args.cuda:\n",
    "    model.cuda()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='set_opt_cell'></a>\n",
    "### Define optimization\n",
    "We will use SGD, but you can change it to ADAM by modifying arg.lr in config_profile.py\n",
    "\n",
    "[BackToSection](#build_network_cell)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# define optimizer\n",
    "def create_optimizer(model, new_lr):\n",
    "    # setup optimizer\n",
    "    if args.optimizer == 'sgd':\n",
    "        optimizer = optim.SGD(model.parameters(), lr=new_lr,\n",
    "                              momentum=0.9, dampening=0.9,\n",
    "                              weight_decay=args.wd)\n",
    "    else:\n",
    "        raise Exception('Not supported optimizer: {0}'.format(args.optimizer))\n",
    "    return optimizer\n",
    "optimizer1 = create_optimizer(model.features, args.lr)\n",
    "print(optimizer1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "---\n",
    "<a id='train_network_cell'></a>\n",
    "## train the deep network  \n",
    "In this section, we will:\n",
    "#### 1. [Define a training module](#define_train_module_cell)\n",
    "#### 2. [Define a testing module](#define_test_module_cell)\n",
    "#### 3. [Train and test on the validation data](#training_module_cell)\n",
    "\n",
    "[BackToTop](#top_cell)\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='define_train_module_cell'></a>\n",
    "### Define a training module\n",
    "\n",
    "[BackToSection](#train_network_cell)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train(train_loader, model, optimizer, epoch, logger, load_triplets  = False):\n",
    "    # switch to train mode\n",
    "    model.train()\n",
    "    pbar = tqdm(enumerate(train_loader))\n",
    "    for batch_idx, data in pbar:\n",
    "        data_a, data_p, data_n = data\n",
    "        \n",
    "\n",
    "        if args.cuda:\n",
    "            data_a, data_p, data_n  = data_a.cuda(), data_p.cuda(), data_n.cuda()\n",
    "            out_a = model(data_a)\n",
    "            out_p = model(data_p)\n",
    "            out_n = model(data_n)\n",
    "        \n",
    "        loss = loss_DesNet(out_a, out_p, out_n, anchor_swap = False, margin = 1.0, loss_type = \"triplet_margin\")\n",
    "                \n",
    "        if args.decor:\n",
    "            loss += CorrelationPenaltyLoss()(out_a)\n",
    "            \n",
    "        if args.gor:\n",
    "            loss += args.alpha*global_orthogonal_regularization(out_a, out_n)\n",
    "            \n",
    "        optimizer.zero_grad()\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "        adjust_learning_rate(optimizer)\n",
    "        if batch_idx % args.log_interval == 0:\n",
    "            pbar.set_description(\n",
    "                'Train Epoch: {} [{}/{} ({:.0f}%)]\\tLoss: {:.6f}'.format(\n",
    "                    epoch, batch_idx * len(data_a), len(train_loader.dataset),\n",
    "                           100. * batch_idx / len(train_loader),\n",
    "                    loss.item()))\n",
    "\n",
    "    if (args.enable_logging):\n",
    "        logger.log_value('loss', loss.item()).step()\n",
    "\n",
    "    try:\n",
    "        os.stat('{}{}'.format(args.model_dir,suffix))\n",
    "    except:\n",
    "        os.makedirs('{}{}'.format(args.model_dir,suffix))\n",
    "\n",
    "    torch.save({'epoch': epoch + 1, 'state_dict': model.state_dict()},\n",
    "               '{}{}/checkpoint_{}.pth'.format(args.model_dir,suffix,epoch))\n",
    "    \n",
    "    \n",
    "def adjust_learning_rate(optimizer):\n",
    "    \"\"\"Updates the learning rate given the learning rate decay.\n",
    "    The routine has been implemented according to the original Lua SGD optimizer\n",
    "    \"\"\"\n",
    "    for group in optimizer.param_groups:\n",
    "        if 'step' not in group:\n",
    "            group['step'] = 0.\n",
    "        else:\n",
    "            group['step'] += 1.\n",
    "        group['lr'] = args.lr * (\n",
    "        1.0 - float(group['step']) * float(args.batch_size) / (args.n_triplets * float(args.epochs)))\n",
    "    return"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='define_test_module_cell'></a>\n",
    "### Define a testing module\n",
    "[BackToSection](#train_network_cell)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def test(test_loader, model, epoch, logger, logger_test_name):\n",
    "    # switch to evaluate mode\n",
    "    model.eval()\n",
    "\n",
    "    losses = 0\n",
    "\n",
    "    pbar = tqdm(enumerate(test_loader))\n",
    "    \n",
    "    for batch_idx, data in pbar:\n",
    "        data_a, data_p, data_n = data\n",
    "        \n",
    "\n",
    "        if args.cuda:\n",
    "            data_a, data_p, data_n  = data_a.cuda(), data_p.cuda(), data_n.cuda()\n",
    "            out_a = model(data_a)\n",
    "            out_p = model(data_p)\n",
    "            out_n = model(data_n)\n",
    "        \n",
    "        loss = loss_DesNet(out_a, out_p, out_n, anchor_swap = False, margin = 1.0, loss_type = \"triplet_margin\")\n",
    "        losses = losses + loss.cpu().numpy()\n",
    "    ave_loss = losses/len(test_loader)\n",
    "    print('\\33[91mLoss on validation: {:.8f}\\n\\33[0m'.format(ave_loss))\n",
    "\n",
    "    if (args.enable_logging):\n",
    "        logger.log_value(logger_test_name+' vloss', ave_loss)\n",
    "    return\n",
    "\n",
    "\n",
    "def ErrorRateAt95Recall(labels, scores):\n",
    "    distances = 1.0 / (scores + 1e-8)\n",
    "    recall_point = 0.95\n",
    "    labels = labels[np.argsort(distances)]\n",
    "    # Sliding threshold: get first index where recall >= recall_point. \n",
    "    # This is the index where the number of elements with label==1 below the threshold reaches a fraction of \n",
    "    # 'recall_point' of the total number of elements with label==1. \n",
    "    # (np.argmax returns the first occurrence of a '1' in a bool array). \n",
    "    threshold_index = np.argmax(np.cumsum(labels) >= recall_point * np.sum(labels)) \n",
    "\n",
    "    FP = np.sum(labels[:threshold_index] == 0) # Below threshold (i.e., labelled positive), but should be negative\n",
    "    TN = np.sum(labels[threshold_index:] == 0) # Above threshold (i.e., labelled negative), and should be negative\n",
    "    return float(FP) / float(FP + TN)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='training_module_cell'></a>\n",
    "### Training \n",
    "\n",
    "[BackToSection](#train_network_cell)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "start = args.start_epoch\n",
    "end = start + args.epochs\n",
    "logger, file_logger = None, None\n",
    "triplet_flag = args.load_random_triplets\n",
    "from Losses import loss_DesNet\n",
    "TEST_ON_W1BS = True\n",
    "LOG_DIR = args.log_dir\n",
    "if(args.enable_logging):\n",
    "    from Loggers import Logger, FileLogger\n",
    "    logger = Logger(LOG_DIR)\n",
    "    \n",
    "suffix = '{}_{}_{}_as_fliprot'.format(args.experiment_name, args.training_set, args.batch_reduce)\n",
    "\n",
    "res_fpr_liberty = torch.zeros(end-start,1)\n",
    "res_fpr_notredame = torch.zeros(end-start, 1)\n",
    "res_fpr_yosemite = torch.zeros(end-start, 1)\n",
    "\n",
    "for epoch in range(start, end):\n",
    "\n",
    "    # iterate over test loaders and test results\n",
    "    train(train_loader, model, optimizer1, epoch, logger, triplet_flag)\n",
    "    with torch.no_grad():\n",
    "        test(validation_loader['dataloader'], model, epoch, logger, validation_loader['name'])\n",
    "\n",
    "    #randomize train loader batches\n",
    "    train_loader, _ = create_loaders(dataset_names, load_random_triplets=triplet_flag)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "[3:25 PM]\n",
    "### Select the best model, and save it as CNN*.pth; * can be 1, 2, or 3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.python.summary.summary_iterator import summary_iterator\n",
    "train_losses = []\n",
    "val_losses = []\n",
    "logfile = 'logs/events.out.tfevents.1619643416.pelican02.eecs.oregonstate.edu'\n",
    "\n",
    "for summary in summary_iterator(logfile):\n",
    "    for value in summary.summary.value:\n",
    "        if value.tag == 'loss':\n",
    "            train_losses.append(value.simple_value)\n",
    "        elif value.tag == 'notredame_vloss':\n",
    "            val_losses.append(value.simple_value)\n",
    "print('min train loss {0} and index {1}'.format(min(train_losses), train_losses.index(min(train_losses))))\n",
    "print('train loss: {0}'.format(train_losses))\n",
    "print('min val loss {0} and index {1}'.format(min(val_losses), val_losses.index(min(val_losses))))\n",
    "print('value loss: {0}'.format(val_losses))\n",
    "\n",
    "#print(train_losses)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "fig, ax = plt.subplots()\n",
    "x =list(range(0,60))\n",
    "val = val_losses\n",
    "train = train_losses\n",
    "ax.plot(x, val, label='Validation loss')\n",
    "ax.plot(x, train, label='Training Loss')\n",
    "#ax.plot(x, x**3, label='cubic')  # ... and some more.\n",
    "ax.set_xlabel('Epoch')  # Add an x-label to the axes.\n",
    "ax.set_ylabel('Triplet Loss')  # Add a y-label to the axes.\n",
    "ax.set_title(\"lr = 20, batch-size = 1024\")  # Add a title to the axes.\n",
    "ax.legend()  # Add a legend."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "---\n",
    "<a id='generate_deep_features_cell'></a>\n",
    "## Generate deep features\n",
    "In this section, we will use your trained network to generate deep features for each patch:\n",
    "#### 1. [load weights](#load_weights_module_cell)\n",
    "#### 2. [load patches](#load_raw_patch_files_module_cell)\n",
    "#### 3. [get deep features](#get_deep_features_module_cell)\n",
    "\n",
    "\n",
    "\n",
    "[BackToTop](#top_cell)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='load_weights_module_cell'></a>\n",
    "### Load network weights\n",
    "[BackToSection](#generate_deep_features_cell)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "trained_weight_path = \"models/liberty_train/_liberty_min_as_fliprot/checkpoint_54.pth\" # suppose you select  checkpoint_4.pth as the best model for this architecture\n",
    "test_model = DesNet()\n",
    "if args.cuda:\n",
    "    test_model.cuda()\n",
    "trained_weight = torch.load(trained_weight_path)['state_dict']\n",
    "test_model.load_state_dict(trained_weight)\n",
    "test_model.eval()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='load_raw_patch_files_module_cell'></a>\n",
    "### Load raw patch files\n",
    "Assume that the raw patch file is stored as patches.pth\n",
    "\n",
    "[BackToSection](#generate_deep_features_cell)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "patches_dir = \"SIFT_patches.pth\"            # these patches are from keypoint detection results\n",
    "patches = torch.load(patches_dir)\n",
    "print(patches.shape)                  # in your case, the shape should be [10, 200, 1, 32, 32]\n",
    "num_imgs, num_pts, _, _, _ = patches.shape\n",
    "patches = patches.view(-1, 1, 32, 32).cuda()\n",
    "print(patches.shape)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='get_deep_features_module_cell'></a>\n",
    "### Get deep features\n",
    "[BackToSection](#generate_deep_features_cell)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "features = test_model(patches)\n",
    "print(features.shape)\n",
    "features = features.view(num_imgs, num_pts, 256).cpu().data\n",
    "print(features.shape)                  # in your case, the shape should be [10, 200, 128]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# save to file, with the name of *_features_CNN*.pth\n",
    "features_dir = \"SIFT_features_CNN3_3.pth\"\n",
    "torch.save(features, features_dir)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}