How to augment data to increase the number of instances being trained on?

[mohamedamrali1993]

Discussed in #3698

^{Originally posted by mohamedamrali1993 November 15, 2021}

## Instructions To Reproduce the Issue: Linux 18.04 Cuda 11.3 Pytorch Detectron2

Hi, I am having a hard time understanding "Augmentation" in the context of Detectron2. It seems like the Data Augmentation page is striving to create variations within the dataset, but not necessarily "augmenting" as in increasing the size of the dataset. i.e. if I have a dataset with 100 images and I apply RandomFlip() transformation to 5% of the data, I still end up with 100 instances to train on. Not 105! Am I understanding this correctly or is it supposed to increase the instances, and I am just doing something wrong?

Expected behavior:

I tracked down the process of data augmentation, and at no point did I see data being appended to a larger query (with original images + transformed images), which is eventually sent to the model. Here is what I understand so far:

1. The trainer calls on to the build_train_loader(), which returns the build_detection_test_loader() with the "mapper" as an argument.
2. We can custom make our own "mapper" and include the appropriate transformations we want.
3. The mapper needs to transform images sequentially, and consequently transform annotations (such as bbox, masks, and keypoints) using the utils.transform_instance_annotations(annotation, transform, size)
4. Finally utils.annotations_to_instances(annotations, size, mask_format) creates the instances that are forwarded to the model.

I would like to increase the number of images that I am training on, and their respective transformed labels. Is this doable using Detectron2 and is it a good idea for Mask R-CNN? FAIR's 2018 Mask R-CNN paper has a small section in Appendix B that talks about "Train-time augmentation" where they scale images (if I am understanding things correctly), but I am not sure if they are actually training on the original images or "original images + scaled images." I am expecting the latter!

If this can be done using Detectron2 then please advise (including code examples), if not, I would imagine this would be a good enhancement!?

I am not the first one to report/ask about this, someone else asked about it in issue #1763

[github-actions[bot]]

You've chosen to report an unexpected problem or bug. Unless you already know the root cause of it, please include details about it by filling the issue template.
The following information is missing: "Instructions To Reproduce the Issue and Full Logs"; "Your Environment";

[Sahar-DataScience]

I faced the same thing today I kept reading the docs looking for a method to "augment" data not just "transform", the use of T.transform is well explained with exemples and everyone is copy pasting the same method,but when you duve deep in the code you discover that what really happens is that the transfromed img and its transformed annotations replace the original instance in dataset_dict

[NamalJayasuriya]

I also want to increase the training dataset size by applying augmentation. I went through the API documentation, tutorials and tried them all. But, I could not find any option to increase the dataset size, it just replace the original images with transformations. If there is any update please let us know.

[mohamedamrali1993]

Hi @NamalJayasuriya,

After working with Detectron2 for quite some time, here is what I learned:

1. You can call the datasetmapper twice in the trainer instantiation and combine them to then pass them off to get batched, this essential will 2X the number of images. One version of the datasetmapper should contribute the original images, and the other can be the transformed images. I don't recommend this!
2. You transform things randomly in each batch as the documentation states and do longer training (i.e., more iteration # in the configuration file). If you think about it, let's say you're training on 1k images, and your batch size is 10. That means you'll have to do 100 iterations to cover an entire epoch. If you add random transformation, it means that some of the images will be transformed while others will be passed to the network as an original. So let's say 50% are transformed; that's 5 out of 10 images per batch. If you do longer training iterations (2 epochs instead of 1), you'll eventually go through the entire dataset's original images and the entire dataset transformed images. Much longer training epochs will ensure that the network sees a lot more variations and hence better training and generalizability of your network.

Hope this helps!