[MXNET-102] Added tutorial on how to use data augmenters. (apache#10055)

* [MXNET-102] Added tutorial on how to use data augmenters.

* Removed Gluon from tutorial.

* Added Gluon version.

* Updated index.md

* Commit to force build.

docs/tutorials/gluon/data_augmentation.md

@@ -0,0 +1,96 @@
+# Methods of applying data augmentation (Gluon API)
+
+Data Augmentation is a regularization technique that's used to avoid overfitting when training Machine Learning models. Although the technique can be applied in a variety of domains, it's very common in Computer Vision. Adjustments are made to the original images in the training dataset before being used in training. Some example adjustments include translating, cropping, scaling, rotating, changing brightness and contrast. We do this to reduce the dependence of the model on spurious characteristics; e.g. training data may only contain faces that fill 1/4 of the image, so the model trained without data augmentation might unhelpfully learn that faces can only be of this size.
+
+In this tutorial we demonstrate a method of applying data augmentation with Gluon [`mxnet.gluon.data.Dataset`](https://mxnet.incubator.apache.org/api/python/gluon/data.html#mxnet.gluon.data.Dataset)s, specifically the [`ImageFolderDataset`](https://mxnet.incubator.apache.org/api/python/gluon/data.html#mxnet.gluon.data.vision.datasets.ImageFolderDataset).
+
+```python
+%matplotlib inline
+import mxnet as mx # used version '1.0.0' at time of writing
+import numpy as np
+from matplotlib.pyplot import imshow
+import multiprocessing
+
+mx.random.seed(42) # set seed for repeatability
+```
+
+We define a utility function below, that will be used for visualising the augmentations in the tutorial.
+
+
+```python
+def plot_mx_array(array):
+    """
+    Array expected to be height x width x 3 (channels), and values are floats between 0 and 255.
+    """
+    assert array.shape[2] == 3, "RGB Channel should be last"
+    imshow((array.clip(0, 255)/255).asnumpy())
+```
+
+```python
+!mkdir -p data/images
+!wget https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/doc/tutorials/data_aug/inputs/0.jpg -P ./data/images/
+```
+
+```python
+example_image = mx.image.imread("./data/images/0.jpg").astype("float32")
+plot_mx_array(example_image)
+```
+
+
+![png](https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/doc/tutorials/data_aug/outputs/use/output_5_0.png)
+
+
+## Quick start with [`ImageFolderDataset`](https://mxnet.incubator.apache.org/api/python/gluon/data.html#mxnet.gluon.data.vision.datasets.ImageFolderDataset)
+
+Using Gluon, it's simple to add data augmentation to your training pipeline. When creating either [`ImageFolderDataset`](https://mxnet.incubator.apache.org/api/python/gluon/data.html#mxnet.gluon.data.vision.datasets.ImageFolderDataset) or [`ImageRecordDataset`](https://mxnet.incubator.apache.org/api/python/gluon/data.html#mxnet.gluon.data.vision.datasets.ImageRecordDataset), you can pass a `transform` function that will be applied to each image in the dataset, every time it's loaded from disk. Augmentations are intended to be random, so you'll pass a slightly different version of the image to the network on each epoch.
+
+We define `aug_transform` below to perform a selection of augmentation steps and pass it to our dataset. It's worth noting that augmentations should only be applied to the training data (and not the test data), so you don't want to pass this augmentation transform function to the testing dataset.
+
+[`mxnet.image.CreateAugmenter`](https://mxnet.incubator.apache.org/api/python/image/image.html?highlight=createaugmenter#mxnet.image.CreateAugmenter) is a useful function for creating a diverse set of augmentations at once. Despite the singular `CreateAugmenter`, this function actually returns a list of Augmenters. We can then loop through this list and apply each type of augmentation one after another. Although the parameters of `CreateAugmenter` are fixed, the random augmentations (such as `rand_mirror` and `brightness`) will be different each time `aug_transform` is called.
+
+
+```python
+def aug_transform(data, label):
+    data = data.astype('float32')/255
+    augs = mx.image.CreateAugmenter(data_shape=(3, 300, 300),
+                                    rand_crop=0.5, rand_mirror=True, inter_method=10,
+                                    brightness=0.125, contrast=0.125, saturation=0.125,
+                                    pca_noise=0.02)
+    for aug in augs:
+        data = aug(data)
+    return data, label
+
+
+training_dataset = mx.gluon.data.vision.ImageFolderDataset('./data', transform=aug_transform)
+```
+
+
+We can quickly inspect the augmentations by indexing the dataset (which calls the `__getitem__` method of the dataset). When this method is called (with an index) the correct image is read from disk, and the `transform` is applied. We can see the result of the augmentations when comparing the image below with the original image above.
+
+
+```python
+sample = training_dataset[0]
+sample_data = sample[0]
+plot_mx_array(sample_data*255)
+```
+
+
+![png](https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/doc/tutorials/data_aug/outputs/use/output_10_0.png)
+
+
+In practice you should load images from a dataset with a [`mxnet.gluon.data.DataLoader`](https://mxnet.incubator.apache.org/api/python/gluon/data.html?highlight=dataloader#mxnet.gluon.data.DataLoader) to take advantage of automatic batching and shuffling. Under the hood the `DataLoader` calls `__getitem__`, but you shouldn't need to call directly for anything other than debugging. Some practitioners pre-augment their datasets by applying a fixed number of augmentations to each image and saving the outputs to disk with the aim of increased throughput. With the `num_workers` parameter of `DataLoader` you can use all CPU cores to apply the augmentations, which often mitigates the need to perform pre-augmentation; reducing complexity and saving disk space.
+
+
+```python
+batch_size = 1
+training_data_loader = mx.gluon.data.DataLoader(training_dataset, batch_size=1, shuffle=True)
+
+for data_batch, label_batch in training_data_loader:
+    plot_mx_array(data_batch[0]*255)
+    assert data_batch.shape == (1, 300, 300, 3)
+    assert label_batch.shape == (1,)
+    break
+```
+
+
+![png](https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/doc/tutorials/data_aug/outputs/use/output_12_1.png)

docs/tutorials/index.md

@@ -104,6 +104,8 @@ The Gluon and Module tutorials are in Python, but you can also find a variety of
 
 - [Training on multiple GPUs with gluon](http://gluon.mxnet.io/chapter07_distributed-learning/multiple-gpus-gluon.html)
 
+- [Applying data augmentation](/tutorials/gluon/data_augmentation.html)
+
 </div>
 
 </div> <!--end of introduction-->
@@ -126,6 +128,7 @@ The Gluon and Module tutorials are in Python, but you can also find a variety of
 - [Inference using an ONNX model](/tutorials/onnx/inference_on_onnx_model.html)
 
 - [Fine-tuning an ONNX model on Gluon](/tutorials/onnx/fine_tuning_gluon.html)
+
 </div> <!--end of applications-->
 
 </div> <!--end of gluon-->
@@ -174,6 +177,8 @@ The Gluon and Module tutorials are in Python, but you can also find a variety of
 
 - [Train a linear regression model with sparse symbols](/tutorials/sparse/train.html)
 
+- [Applying data augmentation](/tutorials/python/data_augmentation.html)
+
 - [Types of data augmentation](/tutorials/python/types_of_data_augmentation.html)
 
 </div>

docs/tutorials/python/data_augmentation.md

@@ -0,0 +1,155 @@
+# Methods of applying data augmentation (Module API)
+
+Data Augmentation is a regularization technique that's used to avoid overfitting when training Machine Learning models. Although the technique can be applied in a variety of domains, it's very common in Computer Vision. Adjustments are made to the original images in the training dataset before being used in training. Some example adjustments include translating, cropping, scaling, rotating, changing brightness and contrast. We do this to reduce the dependence of the model on spurious characteristics; e.g. training data may only contain faces that fill 1/4 of the image, so the model trained without data augmentation might unhelpfully learn that faces can only be of this size.
+
+In this tutorial we discuss the different interfaces available in MXNet to perform data augmentation if you're using the Module API. We start by showing a complete example using Module's [`ImageIter`](https://mxnet.incubator.apache.org/api/python/image/image.html?highlight=imageiter#mxnet.image.ImageIter), and then unpack the example to gain a greater understanding of the internals. In the process you'll learn about augmentation functions, [`mxnet.image.Augmenter`](https://mxnet.incubator.apache.org/api/python/image/image.html?highlight=augmen#mxnet.image.Augmenter) classes and Augmenter lists.
+
+
+```python
+%matplotlib inline
+import mxnet as mx # used version '1.0.0' at time of writing
+import numpy as np
+from matplotlib.pyplot import imshow
+import multiprocessing
+
+mx.random.seed(42) # set seed for repeatability
+```
+
+We define a utility function below, that will be used for visualising the augmentations in the tutorial.
+
+
+```python
+def plot_mx_array(array):
+    """
+    Array expected to be height x width x 3 (channels), and values are floats between 0 and 255.
+    """
+    assert array.shape[2] == 3, "RGB Channel should be last"
+    imshow((array.clip(0, 255)/255).asnumpy())
+```
+
+```python
+!mkdir -p data/images
+!wget https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/doc/tutorials/data_aug/inputs/0.jpg -P ./data/images/
+```
+
+```python
+example_image = mx.image.imread("./data/images/0.jpg").astype("float32")
+plot_mx_array(example_image)
+```
+
+
+![png](https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/doc/tutorials/data_aug/outputs/use/output_5_0.png)
+
+
+## Quick start using [`ImageIter`](https://mxnet.incubator.apache.org/api/python/image/image.html?highlight=imageiter#mxnet.image.ImageIter)
+
+One of the most convenient ways to augment your image data is via arguments of [`mxnet.image.ImageIter`](https://mxnet.incubator.apache.org/api/python/image/image.html?highlight=imageiter#mxnet.image.ImageIter), but you'll need to reference the documentation of [`mxnet.image.CreateAugmenter`](https://mxnet.incubator.apache.org/api/python/image/image.html?highlight=createaugmenter#mxnet.image.CreateAugmenter) to see a full list of available options. Under the hood, additional arguments passed to [`ImageIter`](https://mxnet.incubator.apache.org/api/python/image/image.html?highlight=imageiter#mxnet.image.ImageIter) are collected as keyword arguments (`**kwargs`), and are passed to [`mxnet.image.CreateAugmenter`](https://mxnet.incubator.apache.org/api/python/image/image.html?highlight=createaugmenter#mxnet.image.CreateAugmenter). We'll see this in more detail in the sections below, but [`mxnet.image.CreateAugmenter`](https://mxnet.incubator.apache.org/api/python/image/image.html?highlight=createaugmenter#mxnet.image.CreateAugmenter) creates a list of  [`mxnet.image.Augmenter`](https://mxnet.incubator.apache.org/api/python/image/image.html?highlight=augmen#mxnet.image.Augmenter)s corresponding to each type of augmentation (e.g. crop, flip, change of brightness, etc.), and this list will be iterated though and the augmentations applied in turn. Alternatively, you can create this list yourself and pass it to [`ImageIter`](https://mxnet.incubator.apache.org/api/python/image/image.html?highlight=imageiter#mxnet.image.ImageIter) via the `aug_list` argument.
+
+
+We show a simple example of this below, after creating an `images.lst` file used by the [`ImageIter`](https://mxnet.incubator.apache.org/api/python/image/image.html?highlight=imageiter#mxnet.image.ImageIter). Use [`tools/im2rec.py`](https://github.com/apache/incubator-mxnet/blob/master/tools/im2rec.py) to create the `images.lst` if you don't already have this for your data.
+
+```python
+!echo -e "0\t0.000000\timages/0.jpg" > ./data/images.lst
+```
+
+```python
+training_iter = mx.image.ImageIter(batch_size = 1,
+                                   data_shape = (3, 300, 300),
+                                   path_root= './data', path_imglist='./data/images.lst',
+                                   rand_crop=0.5, rand_mirror=True, inter_method=10,
+                                   brightness=0.125, contrast=0.125, saturation=0.125,
+                                   pca_noise=0.02
+                                   )
+```
+
+
+```python
+for batch in training_iter:
+    assert batch.data[0].shape == (1, 3, 300, 300)
+    assert batch.label[0].shape == (1,)
+    sample = batch.data[0][0].transpose(axes=[1,2,0])
+    plot_mx_array(sample)
+    break
+```
+
+
+![png](https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/doc/tutorials/data_aug/outputs/use/output_28_1.png)
+
+[`mxnet.image.ImageDetIter`](https://mxnet.incubator.apache.org/api/python/image/image.html?highlight=imagedetiter#mxnet.image.ImageDetIter) works similarly (with [`mxnet.image.CreateDetAugmenter`](https://mxnet.incubator.apache.org/api/python/image/image.html?highlight=createdetaugmenter#mxnet.image.CreateDetAugmenter)), but [`mxnet.io.ImageRecordIter`](https://mxnet.incubator.apache.org/api/python/io/io.html?highlight=imagerecorditer#mxnet.io.ImageRecordIter) has a slightly different interface, so reference the documentation [here](https://mxnet.incubator.apache.org/api/python/io/io.html?highlight=imagerecorditer#mxnet.io.ImageRecordIter) if you're using Record IO data format.
+
+## Manual Augmentation
+
+Although the vast majority of cases will be covered using the augmentation arguments of [`mxnet.image.ImageIter`](https://mxnet.incubator.apache.org/api/python/image/image.html?highlight=imageiter#mxnet.image.ImageIter) as we've seen above, sometime you'll want more fine grained control of augmentations. We will now dive into some of the lower level methods for image augmentation, that you can use to manually apply augmentations to images.
+
+### Augmentation Functions
+
+MXNet provides a small number of augmentation functions that are quick and easy to use, but they are limited to positional augmentations (such as [`mxnet.image.random_crop`](https://mxnet.incubator.apache.org/api/python/image/image.html?highlight=random_crop#mxnet.image.random_crop) and [`mxnet.image.resize_short`](https://mxnet.incubator.apache.org/api/python/image/image.html?highlight=random_crop#mxnet.image.resize_short) functions) as opposed to color augmentations (such as brightness jitter). Although these functions are easy to apply, the [`mxnet.image.Augmenter`](https://mxnet.incubator.apache.org/api/python/image/image.html?highlight=augmen#mxnet.image.Augmenter) classes are much more comprehensive and just as easy to use, as we'll see in the next section.
+
+
+```python
+aug_image, crop_box = mx.image.random_crop(example_image, size=(100, 100))
+plot_mx_array(aug_image)
+assert aug_image.shape == (100, 100, 3)
+```
+
+
+![png](https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/doc/tutorials/data_aug/outputs/use/output_16_0.png)
+
+
+### Augmenter Classes
+
+You can apply a wide variety of positional and color augmentations with [`mxnet.image.Augmenter`](https://mxnet.incubator.apache.org/api/python/image/image.html?highlight=augmen#mxnet.image.Augmenter) classes, and using them is the recommended approach for applying augmentations manually. After creating an instance of an Augmenter with the required parameters, you can call the Augmenter just as you would a function. Under the hood a `__call__` method is defined which applies the augmentation. Augmenters with randomness are randomized each time the Augmenter is called, so calling the same Augmenter twice will give different results on the same input.
+
+
+```python
+aug = mx.image.RandomCropAug(size=(100, 100))
+aug_image = aug(example_image)
+plot_mx_array(aug_image)
+assert aug_image.shape == (100, 100, 3)
+```
+
+
+![png](https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/doc/tutorials/data_aug/outputs/use/output_19_0.png)
+
+
+### Augmenter list
+
+Very often you'll want to apply many different types of augmentation to an image. Instead of nesting the calls of Augmenters, a natural structure for handling a large number of Augmenters is a list. You can construct this list manually, or you can use helper functions like [`mxnet.image.CreateAugmenter`](https://mxnet.incubator.apache.org/api/python/image/image.html?highlight=createaugmenter#mxnet.image.CreateAugmenter) to create these lists automatically.
+
+Object detection tasks require the same positional augmentations to be applied to the data and the label, so you should use [`mxnet.image.CreateDetAugmenter`](https://mxnet.incubator.apache.org/api/python/image/image.html?highlight=createdetaugmenter#mxnet.image.CreateDetAugmenter) which handles this case.
+
+
+```python
+# created manually
+aug_list = [mx.image.RandomCropAug(size=(100, 100)), mx.image.BrightnessJitterAug(brightness=1)]
+aug_image = example_image.copy()
+for aug in aug_list:
+    aug_image = aug(aug_image)
+plot_mx_array(aug_image)
+assert all([isinstance(a, mx.image.Augmenter) for a in aug_list])
+```
+
+
+![png](https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/doc/tutorials/data_aug/outputs/use/output_22_1.png)
+
+
+
+```python
+# created automatically
+aug_list = mx.image.CreateAugmenter(data_shape=(3, 300, 300), rand_crop=0.5,
+        rand_mirror=True, mean=True, brightness=0.125, contrast=0.125,
+        saturation=0.125, pca_noise=0.05, inter_method=10)
+aug_image = example_image.copy()
+for aug in aug_list:
+    aug_image = aug(aug_image)
+plot_mx_array(aug_image)
+assert all([isinstance(a, mx.image.Augmenter) for a in aug_list])
+```
+
+
+![png](https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/doc/tutorials/data_aug/outputs/use/output_23_1.png)
+
+
+__*Watch Out!*__ Check some examples that are output after applying all the augmentations. You may find that the augmentation steps are too severe and may actually prevent the model from learning. Some of the augmentation parameters used in this tutorial are set high for demonstration purposes (e.g. `brightness=1`); you might want to reduce them if your training error stays too high during training. Some examples of excessive augmentation are shown below:
+
+<img src="https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/doc/tutorials/data_aug/outputs/use//severe_aug.png" alt="Drawing" style="width: 700px;"/>