Image Augmentation

[manav-agarwal1]

So, I am trying to train an object detection model following the detectron2 tutorial and after getting a benchmark I have decided to do image data augmentation and I am stuck badly. As I have tried bunch of different things but I don't know how to get annotations out of them as annotations are available form previous data. I have made a dictionary of it as explained in tutorial and followed it. But now I am stuck at how to should I make changes. Whether I need to write the image in my training folder or how can I get annotations for them. then if I am doing random cropping then how can I obtain changed locations of boxes.
Please help I am just a beginner.

[rahulg963]

@manav-agarwal1 can you please help me understand your query further?
Are you stuck, because you are not able to get annotations for your augmented images to be used for training/validation? or You are stuck in adding runtime augmentation while training your model using detectron2?

For the first problem, I you can use a library named imgaug. You will have to provide your initial images and annotations, this library will do the necessary augmentations and will give you corresponding annotations.

For the second you can try something like this. This will do pick images and do transformations and then use those transformations in training your model using detectron2.

def mapper(dataset_dict):
    # Implement a mapper, similar to the default DatasetMapper, but with your own customizations
    dataset_dict = copy.deepcopy(dataset_dict)  # it will be modified by code below
    image = utils.read_image(dataset_dict["file_name"], format="BGR")

    image, transforms = T.apply_transform_gens([
        T.RandomFlip(prob=0.20, horizontal=True, vertical=False),
        T.RandomFlip(prob=0.20, horizontal=False, vertical=True),
        T.RandomApply(transform=T.RandomBrightness(intensity_min=0.75, intensity_max=1.25),
                      prob=0.20),
        T.RandomApply(transform=T.RandomContrast(intensity_min=0.76, intensity_max=1.25),
                      prob=0.20),
        T.RandomApply(transform=T.RandomCrop(crop_type="relative_range", crop_size=(0.8, 0.8)), 
                      prob=0.20),
        T.RandomApply(transform=T.RandomSaturation(intensity_min=0.75, intensity_max=1.25), 
                      prob=0.20),
        T.RandomApply(transform=T.Resize(shape=(800, 800)), 
                      prob=0.20),
        T.RandomApply(transform=T.RandomRotation(angle=[-30,30], expand=True, center=None, sample_style="range", interp=None), 
                      prob=0.20)

    ], image)
    
    dataset_dict["image"] = torch.as_tensor(image.transpose(2, 0, 1).astype("float32"))

    annos = [
        utils.transform_instance_annotations(obj, transforms, image.shape[:2])
        for obj in dataset_dict.pop("annotations")
        if obj.get("iscrowd", 0) == 0
    ]
    instances = utils.annotations_to_instances(annos, image.shape[:2])
    dataset_dict["instances"] = utils.filter_empty_instances(instances)
    return dataset_dict

class Trainer(DefaultTrainer):
    @classmethod
    def build_test_loader(cls, cfg, dataset_name):
        return build_detection_test_loader(cfg, dataset_name, mapper=DatasetMapper(cfg, False))

    @classmethod
    def build_train_loader(cls, cfg):
        return build_detection_train_loader(cfg, mapper=mapper)

tr = Trainer(cfg)
tr.resume_or_load(resume=True)
tr.train()

# cfg is your standard get_cfg() object of detectron2

[manav-agarwal1]

@rahulg963 Thanks for replying. I tried both of the methods you just mentioned. Now the thing was with the first one I wasn't able to get my annotations transformed as It was doing to the image. But for the second one, I tried the same fix but at runtime, something weird happened. But can you help me understand one thing that in the second mapper approach when I included the reshape, brightness transform then my prediction labels of test and Val set were looked a lot like garbage value as in an image with 15 labels it was predicting around 50 labels with all same values around 34, sometimes 20 like this. I don't know why this happened.
T.reshape((800,800))
T.RandomFlip(prob=0.4, horizontal=True),
T.RandomContrast(0.8, 3),
T.RandomBrightness(1, 1.6)

These were my transforms
I have one more question that will this also feed my original images along with transformed ones or only the augmented images.
I am currently trying to modify it so that it also includes original images in the training.

[rahulg963]

Imgaug library does transforms corresponding annotations as well. I have been using this for multiple projects of mine. Please check their example and try to do in that way. Here is a small snippet of the same.

images_aug, polygons_aug = seq(images=np.expand_dims(images, axis=0), polygons=polygons)
#here polygons refer to 'from imgaug.augmentables.polys import Polygon', convert your annotation into polygon class provided by imgaug library.

I am not very sure regarding your attempt of second approach, why you are seeing such results. However, in this augmentation, you provide probability for each augmentation, there are chances that it may use original image as well, not a guaranty though.

[rahulg963]

@ppwwyyxx I don't think this is an issue of detectron2. Can you please add relevant labels to this one.

[ppwwyyxx]

thanks @rahulg963 for helping!