docs/source/manual/tutorial/tutorial1_rgb_network.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "9258e220",
   "metadata": {},
   "source": [
    "Basic ToyData Pipeline Tutorial\n",
    "===============================\n",
    "\n",
    "This demonstrates an end-to-end pipeline on RGB toydata\n",
    "\n",
    "The tutorial generates its own training data so it can be run with minimal\n",
    "effort to test that all core components of the system are working.\n",
    "\n",
    "This walks through the entire process of fit -> predict -> evaluate to\n",
    "train a fusion model on RGB data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dcb027f0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# This tutorial will generate its own training data. Change these paths to\n",
    "# wherever you would like the data to go (or use the defaults).  In general\n",
    "# experiments will have a \"data\" DVC directory where the raw data lives, and an\n",
    "# \"experiment\" DVC directory where you will train your model and store the\n",
    "# results of prediction and evaluation.\n",
    "\n",
    "# In this example we are not using any DVC directories, but we will use DVC in\n",
    "# the variable names to be consistent with future tutorials.\n",
    "\n",
    "DVC_DATA_DPATH=$HOME/data/dvc-repos/toy_data_dvc\n",
    "DVC_EXPT_DPATH=$HOME/data/dvc-repos/toy_expt_dvc\n",
    "\n",
    "mkdir -p \"$DVC_DATA_DPATH\"\n",
    "mkdir -p \"$DVC_EXPT_DPATH\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "44729955",
   "metadata": {},
   "source": [
    "Generate Toy Data\n",
    "-----------------\n",
    "\n",
    "Now that we know where the data and our intermediate files will go, lets\n",
    "generate the data we will use to train and evaluate with.\n",
    "\n",
    "The kwcoco package comes with a commandline utility called `kwcoco toydata` to\n",
    "accomplish this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "afe59613",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the names of the kwcoco files to generate\n",
    "TRAIN_FPATH=$DVC_DATA_DPATH/vidshapes_rgb_train/data.kwcoco.json\n",
    "VALI_FPATH=$DVC_DATA_DPATH/vidshapes_rgb_vali/data.kwcoco.json\n",
    "TEST_FPATH=$DVC_DATA_DPATH/vidshapes_rgb_test/data.kwcoco.json\n",
    "\n",
    "# Generate toy datasets using the \"kwcoco toydata\" tool\n",
    "kwcoco toydata vidshapes1-frames5-amazon --bundle_dpath \"$DVC_DATA_DPATH\"/vidshapes_rgb_train\n",
    "kwcoco toydata vidshapes4-frames5-amazon --bundle_dpath \"$DVC_DATA_DPATH\"/vidshapes_rgb_vali\n",
    "kwcoco toydata vidshapes2-frames6-amazon --bundle_dpath \"$DVC_DATA_DPATH\"/vidshapes_rgb_test"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "34d07692",
   "metadata": {},
   "source": [
    "Inspect Generated Kwcoco Files\n",
    "------------------------------\n",
    "\n",
    "Now that we have generated the kwcoco files, lets get used to the 'geowatch'\n",
    "and 'kwcoco' command tooling to insepct the content of the files.\n",
    "\n",
    "Printing statistics is a good first step. The kwcoco stats are for basic\n",
    "image-level statistics, whereas the geowatch stats will give information\n",
    "relevant to the geowatch project, i.e. about videos, sensors, and channels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1696e998",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# First try the kwcoco stats (can pass multiple files)\n",
    "kwcoco stats \"$TRAIN_FPATH\" \"$VALI_FPATH\" \"$TEST_FPATH\"\n",
    "\n",
    "# Next try the geowatch stats\n",
    "geowatch stats \"$TRAIN_FPATH\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8f02906d",
   "metadata": {},
   "source": [
    "Another important CLI tool is 'geowatch visualize' which can be used to\n",
    "visually inspect the contents of a kwcoco file. It does this by simply dumping\n",
    "image files to disk.  This is most useful when the underlying dataset has data\n",
    "outside of the visual range, but it will work on 'regular' rgb data too!\n",
    "\n",
    "Running visualize by default will write images for all channels in the exiting\n",
    "'kwcoco bundle' (i.e. the directory that contains the kwcoco json file) with a\n",
    "hash corresponding to the state of the kwcoco file. It will also output all the\n",
    "channels by default. Use 'geowatch visualize --help' for a list of additional\n",
    "options.\n",
    "\n",
    "Some useful options are:\n",
    "\n",
    "    * '--channels' to view only specific channels\n",
    "    * '--animate' create animated gifs from the sequence\n",
    "    * '--viz_dpath' specify a custom output directory"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e686e721",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Try visualizing the path to the training kwcoco file\n",
    "geowatch visualize \"$TRAIN_FPATH\" --viz_dpath=\"$DVC_EXPT_DPATH/_viz_toyrgb\" --animate=True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8893788c",
   "metadata": {},
   "outputs": [],
   "source": [
    "display < $DVC_EXPT_DPATH/_viz_toyrgb/toy_video_1/toy_video_1_anns_r_g_b.gif"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bab4b9fc",
   "metadata": {},
   "source": [
    "Training, Prediction, and Evaluation\n",
    "------------------------------------\n",
    "\n",
    "Now that we are more comfortable with kwcoco files, lets get into the simplest\n",
    "and most direct way of training a fusion model. This is done by simply calling\n",
    "'geowatch.tasks.fusion' as the main module. We will specify:\n",
    "\n",
    "Data arguments:\n",
    "\n",
    "* paths to the training and validation kwcoco files\n",
    "* what channels we want to early / late fuse (given by a kwcoco sensorchan spec)\n",
    "* information about the input chip size and temporal window\n",
    "\n",
    "Model arguments:\n",
    "\n",
    "* the underlying architecture\n",
    "\n",
    "Other arguments:\n",
    "\n",
    "* learning rate schedulers\n",
    "* optimizers\n",
    "* training strategies\n",
    "\n",
    "In this tutorial we will use 'gpu' as our lightning accelerator.\n",
    "Please read the lightning docs for other available trainer settings:\n",
    "https://pytorch-lightning.readthedocs.io/en/stable/common/trainer.html#devices\n",
    "\n",
    "We will also specify a work directory that will be similar to directories used\n",
    "when real geowatch models are trained."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e0c5f6e0",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "WORKDIR=$DVC_EXPT_DPATH/training/$HOSTNAME/$USER\n",
    "EXPERIMENT_NAME=ToyRGB_Demo_V001\n",
    "DATASET_CODE=ToyRGB\n",
    "DEFAULT_ROOT_DIR=$WORKDIR/$DATASET_CODE/runs/$EXPERIMENT_NAME\n",
    "MAX_STEPS=512\n",
    "TARGET_LR=3e-4\n",
    "WEIGHT_DECAY=$(python -c \"print($TARGET_LR * 1e-2)\")\n",
    "python -m geowatch.tasks.fusion fit --config \"\n",
    "data:\n",
    "    num_workers          : 4\n",
    "    train_dataset        : $TRAIN_FPATH\n",
    "    vali_dataset         : $VALI_FPATH\n",
    "    channels             : 'r|g|b'\n",
    "    time_steps           : 5\n",
    "    chip_dims            : 128\n",
    "    batch_size           : 2\n",
    "model:\n",
    "    class_path: MultimodalTransformer\n",
    "    init_args:\n",
    "        name        : $EXPERIMENT_NAME\n",
    "        arch_name   : smt_it_stm_p8\n",
    "        window_size : 8\n",
    "        dropout     : 0.1\n",
    "lr_scheduler:\n",
    "  class_path: torch.optim.lr_scheduler.OneCycleLR\n",
    "  init_args:\n",
    "    max_lr: $TARGET_LR\n",
    "    total_steps: $MAX_STEPS\n",
    "    anneal_strategy: cos\n",
    "    pct_start: 0.05\n",
    "optimizer:\n",
    "  class_path: torch.optim.AdamW\n",
    "  init_args:\n",
    "    lr: $TARGET_LR\n",
    "    weight_decay: $WEIGHT_DECAY\n",
    "    betas:\n",
    "      - 0.9\n",
    "      - 0.99\n",
    "trainer:\n",
    "  accumulate_grad_batches: 1\n",
    "  default_root_dir     : $DEFAULT_ROOT_DIR\n",
    "  accelerator          : cpu\n",
    "  devices              : 1\n",
    "  #devices             : 0,1\n",
    "  #strategy            : ddp\n",
    "  check_val_every_n_epoch: 1\n",
    "  enable_checkpointing: true\n",
    "  enable_model_summary: true\n",
    "  log_every_n_steps: 5\n",
    "  logger: true\n",
    "  max_steps: $MAX_STEPS\n",
    "  num_sanity_val_steps: 0\n",
    "  limit_val_batches    : 64\n",
    "  limit_train_batches  : 16\n",
    "  callbacks:\n",
    "    - class_path: pytorch_lightning.callbacks.ModelCheckpoint\n",
    "      init_args:\n",
    "        monitor: val_loss\n",
    "        mode: min\n",
    "        save_top_k: 3\n",
    "    - class_path: pytorch_lightning.callbacks.ModelCheckpoint\n",
    "      init_args:\n",
    "        monitor: train_loss\n",
    "        mode: min\n",
    "        save_top_k: 3\n",
    "torch_globals:\n",
    "    float32_matmul_precision: auto\n",
    "initializer:\n",
    "    init: noop\n",
    "\"\n",
    "\n",
    "# For more options with this particular model see:\n",
    "# python -m geowatch.tasks.fusion fit --model.help=MultimodalTransformer"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "75f55327",
   "metadata": {},
   "source": [
    "The training code will output useful information in the \"DEFAULT_ROOT_DIR\".\n",
    "\n",
    "This will include\n",
    "\n",
    "   * A set of checkpoints that score the best on validation metrics in `$DEFAULT_ROOT_DIR/lightning_logs/*/checkpoints`\n",
    "   * A monitor directory containing visualizations of train and validation batches in `$DEFAULT_ROOT_DIR/lightning_logs/*/monitor/train/batches` and `$DEFAULT_ROOT_DIR/lightning_logs/*/monitor/validate/batches`\n",
    "   * Image files containing visualized tensorboard curves in `$DEFAULT_ROOT_DIR/lightning_logs/*/monitor/tensorboard`\n",
    "       (you can start a tensorboard server if you want to)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c011757a",
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# Demo visual outputs of the training directory (use python to glob for them)\n",
    "FIRST_TRAIN_BATCH_FPATH=$(python -c \"\n",
    "import pathlib\n",
    "logs_dpath = pathlib.Path('$DEFAULT_ROOT_DIR/lightning_logs')\n",
    "print(sorted(logs_dpath.glob('*/monitor/train/batch/*.jpg'))[0])\n",
    "\")\n",
    "LAST_TRAIN_BATCH_FPATH=$(python -c \"\n",
    "import pathlib\n",
    "logs_dpath = pathlib.Path('$DEFAULT_ROOT_DIR/lightning_logs')\n",
    "print(sorted(logs_dpath.glob('*/monitor/train/batch/*.jpg'))[-1])\n",
    "\")\n",
    "echo \"Training batch at the start of training\"\n",
    "display < \"$FIRST_TRAIN_BATCH_FPATH\"\n",
    "echo \"Training batch at the end of training\"\n",
    "display < \"$LAST_TRAIN_BATCH_FPATH\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e7c450aa",
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# Demo visual outputs of the training directory (use python to glob for them)\n",
    "FIRST_VALI_BATCH_FPATH=$(python -c \"\n",
    "import pathlib\n",
    "logs_dpath = pathlib.Path('$DEFAULT_ROOT_DIR/lightning_logs')\n",
    "print(sorted(logs_dpath.glob('*/monitor/validate/batch/*.jpg'))[0])\n",
    "\")\n",
    "LAST_VALI_BATCH_FPATH=$(python -c \"\n",
    "import pathlib\n",
    "logs_dpath = pathlib.Path('$DEFAULT_ROOT_DIR/lightning_logs')\n",
    "print(sorted(logs_dpath.glob('*/monitor/validate/batch/*.jpg'))[-1])\n",
    "\")\n",
    "echo \"Training batch at the start of training\"\n",
    "display < \"$FIRST_VALI_BATCH_FPATH\"\n",
    "echo \"Training batch at the end of training\"\n",
    "display < \"$LAST_VALI_BATCH_FPATH\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6b445685",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Demo visual outputs of the training directory (use python to glob for them)\n",
    "TENSORBOARD_DPATH=$(python -c \"\n",
    "import pathlib\n",
    "logs_dpath = pathlib.Path('$DEFAULT_ROOT_DIR/lightning_logs')\n",
    "print(sorted(logs_dpath.glob('*/monitor/tensorboard'))[0])\n",
    "\")\n",
    "display < \"$TENSORBOARD_DPATH/train_loss.png\"\n",
    "display < \"$TENSORBOARD_DPATH/val_loss.png\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b378ed3c",
   "metadata": {},
   "source": [
    "In this examples we also specify a \"package_fpath\" for convenience.\n",
    "Typically you will want to repackage multiple checkpoints as torch models (see\n",
    "tutorial TODO), but for this first example, \"package_fpath\" provides an easy way to\n",
    "tell the training to always package up the final checkpoint in a serialized model.\n",
    "\n",
    "Torch models are great because they combine the weights with the code needed to\n",
    "execute the forward pass of the network.  They can also store metadata about\n",
    "how the model was trained, which is critical for performing robust analysis on\n",
    "large numbers of models.\n",
    "\n",
    "We provide a CLI tool to summarize the info contained in a torch model via\n",
    "\"geowatch torch_model_stats\". Lets try that on the model we just built."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "55c8fb06",
   "metadata": {},
   "outputs": [],
   "source": [
    "geowatch torch_model_stats \"$DEFAULT_ROOT_DIR\"/final_package.pt --stem_stats=True\n",
    "\n",
    "# NOTE: There are other model weights available in the\n",
    "# $DEFAULT_ROOT_DIR/*/*/checkpoints directory that can be converted into\n",
    "# packages using the geowatch.mlops.repackager script. The final package may not\n",
    "# be the best model."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6876649e",
   "metadata": {},
   "source": [
    "You can see the model knows what kwcoco model it was trained with as well as\n",
    "what sensors and channels it is applicable to.\n",
    "\n",
    "Furthermore it knows (1) the names of the classes that correspond to its final\n",
    "classification layer and (2) the estimated mean / std of the data. These are\n",
    "two pieces of information that are hardly ever properly bundled with models\n",
    "distributed by the ML community, and that is something that needs to change.\n",
    "\n",
    "The model itself knows how to subtract the mean / std, so the dataloader should\n",
    "never have to. Our fusion training code also knows how to estimate this for\n",
    "each new dataset, so you never have to hard code it."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d8bb932",
   "metadata": {},
   "source": [
    "Prediction\n",
    "----------\n",
    "\n",
    "Now that we have an understanding of what metadata the model contains, we can\n",
    "start to appreciate the dead simplicity of predicting with it.\n",
    "\n",
    "To use a model to predict on an unseed kwcoco dataset (in this case the toy\n",
    "test set) we simply call the \"geowatch.tasks.fusion.predict\" script and pass it:\n",
    "\n",
    "   * the kwcoco file of the dataset to predict on\n",
    "   * the path to the model we want to predict with\n",
    "   * the name of the output kwcoco file that will contain the predictions\n",
    "\n",
    "All necessary metadata you would normally have to (redundantly) specify in\n",
    "other frameworks is inferred by programmatically reading the model. You also\n",
    "have the option to overwrite prediction parameters. See --help for details, but\n",
    "for now lets just run with the defaults that match how the model was trained.\n",
    "\n",
    "Note that the test dataset contains groundtruth annotations. All annotations\n",
    "are stripped and ignored during prediction.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b9f6dd6b",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "python -m geowatch.tasks.fusion.predict \\\n",
    "    --test_dataset=\"$TEST_FPATH\" \\\n",
    "    --package_fpath=\"$DEFAULT_ROOT_DIR\"/final_package.pt  \\\n",
    "    --pred_dataset=\"$DVC_EXPT_DPATH\"/predictions/pred.kwcoco.json"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "de44d487",
   "metadata": {},
   "source": [
    "The output of the predictions is just another kwcoco file, but it augments the\n",
    "input images with new channels corresponding to predicted heatmaps. We can use\n",
    "the \"geowatch stats\" command to inspect what these new channels are."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "65a8f53e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Inspect the channels in the prediction file\n",
    "geowatch stats \"$DVC_EXPT_DPATH\"/predictions/pred.kwcoco.json"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "98d808ce",
   "metadata": {},
   "source": [
    "Running this command you can see that images now have a channels \"salient\",\n",
    "which corresponds to the BAS saliency task, and \"star\", \"eff\", and \"superstar\"\n",
    "which correspond to the classification head (for SC), and lastly the \"change\"\n",
    "channel, which is from the change head.\n",
    "\n",
    "Because these are just rasters, we can visualize them using \"geowatch\n",
    "visualize\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a54d03c5",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Visualize the channels in the prediction file\n",
    "geowatch visualize \"$DVC_EXPT_DPATH\"/predictions/pred.kwcoco.json --stack=True \\\n",
    "    --viz_dpath=\"$DVC_EXPT_DPATH\"/predictions/viz_pred \\\n",
    "    --animate=True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "08e83d36",
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "display < \"$DVC_EXPT_DPATH\"/predictions/viz_pred/toy_video_1/toy_video_1_imgs_stack.gif"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "02e30f78",
   "metadata": {},
   "source": [
    "Evaluation\n",
    "----------\n",
    "The last step in this basic tutorial is to measure how good our model is.\n",
    "We can do this with pixelwise metrics.\n",
    "\n",
    "This is done by using \"watch.tasks.fusion.evaluate\" as the main module, and\n",
    "its arguments are:\n",
    "\n",
    "    * The true kwcoco data with groundtruth annotations (i.e. the test dataset)\n",
    "    * The pred kwcoco data that we predicted earlier\n",
    "    * An output path for results\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7662113e",
   "metadata": {},
   "outputs": [],
   "source": [
    "python -m geowatch.tasks.fusion.evaluate \\\n",
    "    --true_dataset=\"$TEST_FPATH\" \\\n",
    "    --pred_dataset=\"$DVC_EXPT_DPATH\"/predictions/pred.kwcoco.json \\\n",
    "      --eval_dpath=\"$DVC_EXPT_DPATH\"/predictions/eval"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3e3f51e2",
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "display < \"$DVC_EXPT_DPATH\"/predictions/eval/curves/nocls_pr_roc.png\n",
    "display < \"$DVC_EXPT_DPATH\"/predictions/eval/curves/nocls_thresh.png"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "327cfe28",
   "metadata": {},
   "source": [
    "This will output summary pixelwise metrics to the terminal but more detailed reports\n",
    "will be written to the eval_dpath.\n",
    "\n",
    "This will include ROC curves, PR curves, and threshold curves drawn as png\n",
    "images.  These curves are also stored as serialized json files so they can be\n",
    "reparsed an replotted or used to compute different metric visualizations.\n",
    "\n",
    "It will also include visualizations of heatmaps overlaid with the truth, with\n",
    "areas of confusion higlighted.\n",
    "\n",
    "\n",
    "This concludes the basic RGB tutorial.\n",
    "\n",
    "The next tutorial lives in toy_experiments_msi.sh and will use a different\n",
    "variant of kwcoco generated data that more closely matches the watch problem by\n",
    "simulating different sensors with different channels."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Bash",
   "language": "bash",
   "name": "bash"
  },
  "language_info": {
   "codemirror_mode": "shell",
   "file_extension": ".sh",
   "mimetype": "text/x-sh",
   "name": "bash"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}