sdg and training workflow

Author: RishabhChadha-97

CLA.md

@@ -0,0 +1,58 @@
+## Individual Contributor License Agreement (CLA)
+
+**Thank you for submitting your contributions to this project.**
+
+By signing this CLA, you agree that the following terms apply to all of your past, present and future contributions
+to the project.
+
+### License.
+
+You hereby represent that all present, past and future contributions are governed by the
+[MIT License](https://opensource.org/licenses/MIT)
+copyright statement.
+
+This entails that to the extent possible under law, you transfer all copyright and related or neighboring rights
+of the code or documents you contribute to the project itself or its maintainers.
+Furthermore you also represent that you have the authority to perform the above waiver
+with respect to the entirety of you contributions.
+
+### Moral Rights.
+
+To the fullest extent permitted under applicable law, you hereby waive, and agree not to
+assert, all of your “moral rights” in or relating to your contributions for the benefit of the project.
+
+### Third Party Content.
+
+If your Contribution includes or is based on any source code, object code, bug fixes, configuration changes, tools,
+specifications, documentation, data, materials, feedback, information or other works of authorship that were not
+authored by you (“Third Party Content”) or if you are aware of any third party intellectual property or proprietary
+rights associated with your Contribution (“Third Party Rights”),
+then you agree to include with the submission of your Contribution full details respecting such Third Party
+Content and Third Party Rights, including, without limitation, identification of which aspects of your
+Contribution contain Third Party Content or are associated with Third Party Rights, the owner/author of the
+Third Party Content and Third Party Rights, where you obtained the Third Party Content, and any applicable
+third party license terms or restrictions respecting the Third Party Content and Third Party Rights. For greater
+certainty, the foregoing obligations respecting the identification of Third Party Content and Third Party Rights
+do not apply to any portion of a Project that is incorporated into your Contribution to that same Project.
+
+### Representations.
+
+You represent that, other than the Third Party Content and Third Party Rights identified by
+you in accordance with this Agreement, you are the sole author of your Contributions and are legally entitled
+to grant the foregoing licenses and waivers in respect of your Contributions. If your Contributions were
+created in the course of your employment with your past or present employer(s), you represent that such
+employer(s) has authorized you to make your Contributions on behalf of such employer(s) or such employer
+(s) has waived all of their right, title or interest in or to your Contributions.
+
+### Disclaimer.
+
+To the fullest extent permitted under applicable law, your Contributions are provided on an "as is"
+basis, without any warranties or conditions, express or implied, including, without limitation, any implied
+warranties or conditions of non-infringement, merchantability or fitness for a particular purpose. You are not
+required to provide support for your Contributions, except to the extent you desire to provide support.
+
+### No Obligation.
+
+You acknowledge that the maintainers of this project are under no obligation to use or incorporate your contributions
+into the project. The decision to use or incorporate your contributions into the project will be made at the
+sole discretion of the maintainers or their authorized delegates.

LICENSE.md

@@ -0,0 +1,20 @@
+ SPDX-FileCopyrightText: Copyright (c) 2022 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+ SPDX-License-Identifier: MIT
+
+ Permission is hereby granted, free of charge, to any person obtaining a
+ copy of this software and associated documentation files (the "Software"),
+ to deal in the Software without restriction, including without limitation
+ the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ and/or sell copies of the Software, and to permit persons to whom the
+ Software is furnished to do so, subject to the following conditions:
+
+ The above copyright notice and this permission notice shall be included in
+ all copies or substantial portions of the Software.
+
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+ THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
+ DEALINGS IN THE SOFTWARE.

README.md

@@ -0,0 +1,72 @@
+# Synthetic Data Generation and Training with Sim Ready Assets
+This project provides a workflow for Training Computer Vision models with Synthetic Data. We will use Isaac Sim with Omniverse Replicator to generate data for our use case and objects of interest. To ensure seamless compatibility with model training, the data generated is in the KITTI format. 
+
+These steps can be followed on a Cloud/remote GPU instance or locally
+
+## How to use this repository
+- [Guide](local/README.md) for running the workflow locally
+- [Guide](cloud/README.md) for running on a cloud/remote instance
+
+## Workflow Components:
+* Generating Data: Use Isaac Sim to generate data
+* Training: We will use TAO toolkit, however users can train a model in a framework of their choice with data generated
+
+### SDG 
+- Using the `palletjack` assets from the Warehouse Sim Ready Asset collection 
+
+- Carry out Domain Randomization in the scene with Replicator:
+    - Various attributes of the scene like lighting, textures, object pose and materials can be modified 
+    - Important to generate a good quality dataset to ensure model detects objects in the real world
+
+- Data output KITTI format
+    - We will use the KITTI Writer for generating annotations
+    - Possible to implement a custom writer (can be useful when data is expected in a certain format for your model)
+
+- Sample generated images:
+
+<p>
+    <img src="images/sample_synthetic/21.png" height="256"/>
+    <img src="images/sample_synthetic/653.png" height="256"/>
+</p>
+
+<p>
+    <img src="images/sample_synthetic/896.png" height="256"/>
+    <img src="images/sample_synthetic/1545.png" height="256"/>
+</p>
+
+
+
+### Training
+- TAO: Outline of steps 
+    - Generating Tfrecords
+     
+    - Model training and evaluation
+        - Model backbone selction
+        - Hyperparameters specified via `spec` file (provided with repo) 
+        - Running inference with trained model
+
+- Sample real world detections on LOCO dataset images:
+
+<p>
+    <img src="images/real_world_results/1564562568.298206.jpg" height="256"/>
+    <img src="images/real_world_results/1564562843.0618184.jpg" height="256"/>
+</p>
+
+<p>
+    <img src="images/real_world_results/593768,3659.jpg" height="256"/>
+    <img src="images/real_world_results/510196244,1362.jpg" height="256"/>
+</p>
+
+<p>
+    <img src="images/real_world_results/1574675156.7667925.jpg" height="256"/>
+    <img src="images/real_world_results/426023,9672.jpg" height="256"/>
+</p>
+
+
+### Deployment
+- Perform Optimizations: Pruning and QAT with TAO to reduce model size and improve performance
+- Deploy on NVIDIA Jetson powered Robot with Isaac ROS or Deepstream 
+
+
+## References:
+- Real world images from the [LOCO dataset](https://github.com/tum-fml/loco) are used for visualizing   model performance

cloud/README.md

@@ -0,0 +1,34 @@
+# Requirements
+- Access to a cloud/remote GPU instance (workflow tested on a `g4dn` AWS EC2 instance with T4 GPU)
+- Docker setup instructions are provided in the notebooks 
+- Entire workflow can be run in `headless` mode (SDG script and training)
+
+
+## Synthetic Data Generation
+- Use the Isaac Sim docker container for running the Data Generation [script](../palletjack_sdg/palletjack_datagen.sh) 
+- We will generate data for warehouse `palletjack` objects in KITTI format
+- Follow the steps in the `cloud_sdg` notebook
+- This generated data can be used to train your own model (framework and architecture of your choice), in this workflow we demonstrate using TAO for training
+
+
+## Training with TAO Toolkit
+- The `training/cloud_train` notebook provides a walkthrough of the steps:
+    - Setting up TAO docker container
+    - Downloading pre-trained model, we will use the `DetectNet_v2` model with a `resnet_18` backbone
+    - Running TAO training with `spec` files provided
+    - Visualizing model performance on real world data
+- Visualize model metric with Tensorboard
+<img src="../images/tensorboard/tensorboard_resized_palletjack.png"/>
+
+
+## Next steps
+
+### Generating Synthetic Data for your use case
+- Make changes in the Domain Randomization under the Synthetic Data Generation [script](../palletjack_sdg/standalone_palletjack_sdg.py)
+- Add additional objects of interest in the scene (similar to how palletjacks are added, you can add forklifts, ladders etc.) to generate dataUse different models for training with TAO (for object detection, you can use YOLO, SSD, EfficientDet) 
+- Replicator provides Semantic Segmentation, Instance Segmentation, Depth and various other ground truth annotations along with RGB. You can also write your own ground truth annotator (eg: Pose Estimation: Refer to [sample](https://docs.omniverse.nvidia.com/isaacsim/latest/tutorial_replicator_offline_pose_estimation.html) These can be used for training a model of your own framework and choice)
+- Exploring the option of using Synthetic + Real data for training a network. Can be particularly useful for generating more data around particular corner cases
+
+### Deploying Trained Models
+- The trained model can be pruned and optimized for inference with TAO
+- This can then be deployed on a robot with NVIDIA Jetson

cloud/cloud_sdg.ipynb

@@ -0,0 +1,182 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Part 1: Synthetic Data Generation and Training Workflow with Warehouse Sim Ready Assets\n",
+    "\n",
+    "This notebook is the first part of the SDG and Training Workflow. We will be focusing on generating Synthetic Data for our use case\n",
+    "\n",
+    "A high level overview of the steps:\n",
+    "* Pulling Isaac Sim Docker Container \n",
+    "* Using Replicator API for Data Generation with Domain Randomization\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Table of Contents\n",
+    "\n",
+    "This notebook shows provides an overview of generating synthetic data using Warehouse Sim Ready assets with Isaac Sim and Omniverse Replicator. We will generate data for the `palletjack` class of objects. \n",
+    "\n",
+    "1. [Set up Isaac Sim via Docker Container](#head-1)\n",
+    "2. [Generate Data for Detecting Palletjacks](#head-2)\n",
+    "3. [Deeper dive into SDG script](#head-3)\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 1. Set up Isaac Sim: Docker Container Installation <a class=\"anchor\" id=\"head-1\"></a>\n",
+    "\n",
+    "### This step can be skipped if the Isaac Sim Docker container has already been set up on your Cloud/Remote Instance\n",
+    "\n",
+    "* Follow the [instructions](https://docs.omniverse.nvidia.com/isaacsim/2022.2.1/install_container.html) for Isaac Sim Container Installation\n",
+    "* Ensure that `docker run` command on Step 7 works as expected and you are able to enter the container. \n",
+    "\n",
+    "We will use `./python.sh` in the container to run our SDG script. Please make sure you exit the container before running the next cells  "
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "tags": []
+   },
+   "source": [
+    "## 2. Generate Data for Detecting Palletjacks  <a class=\"anchor\" id=\"head-2\"></a>\n",
+    "\n",
+    "* We can find the Palletjack USDs in the Warehouse Sim Ready asset collection (`http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/DigitalTwin/Assets/Warehouse/Equipment/Pallet_Trucks`)\n",
+    "* First, we will mount our current local directory while running the docker container. This will ensure that we can run our scripts inside the Isaac Sim container. Data generated in the container will also be saved in this mounted directory."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "/home/karma/Downloads/getting_started_v4.0.1/notebooks/tao_launcher_starter_kit/detectnet_v2/sdg_and_training/sdg-and-training/palletjack_sdg\n"
+     ]
+    }
+   ],
+   "source": [
+    "import os\n",
+    "\n",
+    "# This is the directory which will be mounted into the Isaac Sim container. Make sure <path_to_repo_cloned> is updated correctly\n",
+    "# os.environ[\"MOUNT_DIR\"]=os.path.join(<path_where_repo_cloned>, \"palletjack_sdg\")\n",
+    "os.environ[\"LOCAL_PROJECT_DIR\"]=os.path.dirname(os.getcwd())\n",
+    "os.environ[\"MOUNT_DIR\"] = os.path.join(os.getenv(\"LOCAL_PROJECT_DIR\"), \"palletjack_sdg\")\n",
+    "print(os.getenv(\"MOUNT_DIR\"))"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "scrolled": true,
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "# Make sure the MOUNT_DIR location is correct, it shold have the scripts needed for SDG there\n",
+    "\n",
+    "!docker run --name isaac-sim --entrypoint bash -it --gpus all -e \"ACCEPT_EULA=Y\" --rm --network=host \\\n",
+    "    -v ~/docker/isaac-sim/cache/kit:/isaac-sim/kit/cache/Kit:rw \\\n",
+    "    -v ~/docker/isaac-sim/cache/ov:/root/.cache/ov:rw \\\n",
+    "    -v ~/docker/isaac-sim/cache/pip:/root/.cache/pip:rw \\\n",
+    "    -v ~/docker/isaac-sim/cache/glcache:/root/.cache/nvidia/GLCache:rw \\\n",
+    "    -v ~/docker/isaac-sim/cache/computecache:/root/.nv/ComputeCache:rw \\\n",
+    "    -v ~/docker/isaac-sim/logs:/root/.nvidia-omniverse/logs:rw \\\n",
+    "    -v ~/docker/isaac-sim/data:/root/.local/share/ov/data:rw \\\n",
+    "    -v ~/docker/isaac-sim/documents:/root/Documents:rw \\\n",
+    "    -v $MOUNT_DIR:/isaac-sim/palletjack_sdg \\\n",
+    "    nvcr.io/nvidia/isaac-sim:2022.2.1 \\\n",
+    "    ./palletjack_sdg/palletjack_datagen.sh\n",
+    "    \n",
+    "# Make sure $MOUNT_DIR is set correctly from the cell above"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "\n",
+    "The data generation will begin in `headless` mode. We will be generating 5k images and using a 90:10 split for training and validation. "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "# Once the data generation is complete, list the folders in the data directory\n",
+    "\n",
+    "!ls -rlt $MOUNT_DIR/palletjack_data\n",
+    "\n",
+    "# There hould be 3 folders -> 1. distractors_warehouse 2. distractors_additional 3. no_distractors "
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 3. Deeper Dive into SDG Script  <a class=\"anchor\" id=\"head-3\"></a>\n",
+    "\n",
+    "* The `standalone_palletjack_sdg.py` is the Python script which runs and generates data in headless mode inside the container.\n",
+    "* The overall flow of the script is similar to the `standalone_examples/replicator/offline_generation.py` file provided as a starting point with Isaac Sim\n",
+    "\n",
+    "\n",
+    "* We will be carrying out specific randomizations targeted to our use case. Some of them are:\n",
+    "    * Camera Pose Randomization -> Should be similar to a robot perspective in the scene\n",
+    "    * Palletjack Color Randomization -> To ensure model is robust to variations in Palletjack colors\n",
+    "    * Distractors Pose Randomization -> To enable the model to *focus* on the right object (Our object of interest: Palletjack)\n",
+    "    * Lighting Randomization-> Model robust to lights and reflections/shadows in the scene\n",
+    "    * Floor and Wall Texture Randomization -> Model more robust to changes in background textures and features <br> <br>\n",
+    "    \n",
+    "    \n",
+    "* We are only interested in the `palletjack` object class, all other semantics are removed from the stage with the `update_semantics()` function\n",
+    "\n",
+    "* You can use a model of your own choice to train with this data (Pytorch/Tensorflow or other frameworks)\n",
+    "\n",
+    "* The data is written in the KITTI Format, this allows seamless integration with TAO to train a model. Refer to `training/cloud_train.ipynb` notebook (Part 2) for training with TAO\n"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.8.10"
+  },
+  "vscode": {
+   "interpreter": {
+    "hash": "f23a2831654361cfd8b219e05b5055fdda3e37fe5c0b020e6226f740844c300a"
+   }
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}