Add standard ResNet data augmentation for ImageRecordIter #11027

hetong007 · 2018-05-23T01:38:07Z

Description

Add standard ResNet data augmentation for ImageRecordIter.

Specifically, this augmentation performs a "random resized crop", which crop the image with respect to a random aspect ratio, and a random area, and resize it to the desired size.

If random_resized_crop=True, existing max_random_scale and min_random_scale will be ignored.
If random_resized_crop=False, newly-added max_random_area and min_random_area will be ignored.

Besides, color jittering and pca noise are added as additional data augmentation options.

Checklist

Essentials

Please feel free to remove inapplicable items for your PR.

Changes are complete (i.e. I finished coding on this PR)
All changes have test coverage:
Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
Code is well-documented:
For user-facing API changes, API doc string has been updated.
For new C++ functions in header files, their functionalities and arguments are documented.
To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

piiswrong · 2018-05-23T18:40:19Z

Needs to add jitter pca noise etc

rahul003 · 2018-05-23T19:41:44Z

We might want to add this option to the standard image classification example: image_classification/common/data.py

…xnet into resnet_aug

piiswrong · 2018-05-24T17:50:43Z

src/io/image_aug_default.cc

@@ -218,6 +257,80 @@ class DefaultImageAugmenter : public ImageAugmenter {
      res = src;
    }

+    if (param_.random_resized_crop) {


Add checks for the mutual exclusion relationship between random_resized_crop and (resize, crop, ctc)

Do you mean unittest, or sanity checks in C++?

sanity checks

piiswrong · 2018-05-24T17:50:52Z

example/image-classification/train_imagenet.py

@@ -31,7 +31,7 @@
    data.add_data_args(parser)
    data.add_data_aug_args(parser)
    # use a large aug level
-    data.set_data_aug_level(parser, 3)
+    # data.set_data_aug_level(parser, 3)


The set_data_aug_level doesn't set standard imagenet training augmentation.

piiswrong · 2018-05-24T17:51:40Z

tests/python/train/test_resnet_aug.py

+        data_shape  = data_shape,
+        batch_size  = batch_size,
+        random_resized_crop = True,
+        max_aspect_ratio = 0.25,


what does 0.25 mean?

Is it the same as (3/4, 4/3)?

Thanks. I realized it's (3/4, 5/4), which is different. Fixing.

rahul003 · 2018-05-25T22:59:43Z

src/io/image_aug_default.cc

+    DMLC_DECLARE_FIELD(saturation).set_default(0.0f)
+        .describe("Add a random value in ``[-saturation, saturation]`` to "
+                  "the saturation of image.");
+        DMLC_DECLARE_FIELD(pca_noise).set_default(0.0f)


Could you add description of what the value would mean in this case

…o resnet_aug

…xnet into resnet_aug

rahul003 · 2018-06-08T21:22:50Z

example/image-classification/common/data.py

+                     help='max area to crop in random resized crop, whose range is [0, 1]')
+    aug.add_argument('--min-random-area', type=float, default=1,
+                     help='min area to crop in random resized crop, whose range is [0, 1]')
+    aug.add_argument('--brightness', type=float, default=0,


I think it would be better to set defaults to the values we know work well, so that this script is runnable easily. Especially since we no longer set the aug level in train_imagenet.py

My opinion is that by setting default values to perform no augmentation, users know exactly what kind of augmentation has been applied to the pipeline. It is transparent and will cause less confusion.

On the other hand, not every model uses the same augmentation as ResNet to train on ImageNet. If later another work introduces a better augmentation pipeline with different parameters, then we are unable to justify our default parameters.

How about a function like the set_aug_level with the defaults set for resnet, but that function is not automatically called?
My only concern was that we don't have a good source (for users not using GluonCv, on a good set of starting params). Your point also makes sense, so am not too strongly opinionated about this.

Just reviewed the code again, I think I'll turn off --random-crop and --random-mirror as well.

I can replace set_aug_level with set_resnet_aug. The current one has settings look kind of arbitrary to me.

This PR removed set_data_aug_level() in data.py, but the function in still used in example/image-classification/fine-tune.py and example/image-classification/train_cifar10.py. Also, shouldn't set_resnet_aug() turn on mirroring, given the default has been changed to 0? Currently:

def set_resnet_aug(aug): # standard data augmentation setting for resnet training aug.set_defaults(random_crop=0, random_resized_crop=1) aug.set_defaults(min_random_area=0.08) aug.set_defaults(max_random_aspect_ratio=4./3., min_random_aspect_ratio=3./4.) aug.set_defaults(brightness=0.4, contrast=0.4, saturation=0.4, pca_noise=0.1)

@hetong007 @ptrendx

@DickJC123 Hi Dick, I'm fixing these issues in #11533

rahul003 · 2018-06-08T21:30:40Z

example/image-classification/common/data.py

@@ -120,6 +136,7 @@ def get_rec_iter(args, kv=None):
    else:
        (rank, nworker) = (0, 1)
    rgb_mean = [float(i) for i in args.rgb_mean.split(',')]
+    random_resized_crop = args.random_resized_crop is not None


Wouldn't args.random_resized_crop never be None? action='store_true' means if not passed it is set by default to false

Right. I'll fix that.

Also, for uniformity with random-crop and random-mirror we might want to random-resized-crop be of type int

piiswrong · 2018-06-12T01:19:24Z

src/io/image_aug_default.cc

+        index_t new_cols = static_cast<index_t>(static_cast<float>(param_.data_shape[1]) /
+                                                static_cast<float>(res.rows) *
+                                                static_cast<float>(res.cols));
+        int interpolation_method = GetInterMethod(param_.inter_method, res.cols, res.rows,


Looks like the three calls to GetInterMethod can be extracted outside

hetong007 · 2018-06-12T01:22:01Z

src/io/image_aug_default.cc

@@ -218,10 +265,96 @@ class DefaultImageAugmenter : public ImageAugmenter {
      res = src;
    }

+    if (param_.random_resized_crop) {
+      // random resize crop
+      CHECK(param_.min_random_scale == 1.0f &&


@piiswrong do you think this check is good enough?

piiswrong · 2018-06-14T21:57:31Z

src/io/image_aug_default.cc

+        "min_crop_size, max_crop_size, "
+        "and rand_crop.";
+      if (param_.max_random_area != 1.0f || param_.min_random_area != 1.0f
+          || param_.max_aspect_ratio != 1.0f || param_.min_aspect_ratio != 1.0f) {


if these are all 1.0, you still need to resize the image right?

You are correct. I need to move the center crop out of this if-condition.

piiswrong · 2018-06-14T22:01:42Z

src/io/image_aug_default.cc

+            }
+            random_resized_crop_exec = true;
+      }
+    }
    // normal augmentation by affine transformation.
    if (param_.max_rotate_angle > 0 || param_.max_shear_ratio > 0.0f


this should be else if?

It is possible that users want to add rotation on top of random resized crop.

piiswrong · 2018-06-14T22:02:48Z

src/io/image_aug_default.cc

@@ -277,7 +413,8 @@ class DefaultImageAugmenter : public ImageAugmenter {
    }

    // crop logic
-    if (param_.max_crop_size != -1 || param_.min_crop_size != -1) {
+    if (!param_.random_resized_crop &&


This looks weird. Why not resize in the random sized crop section?

random resized crop considers sample the relative area and the aspect ratio of the to-be-cropped part.

here's random crop, it randomly crop a square with its edge length sampled from [min_crop_size, max_crop_size]

… resized crop

hetong007 · 2018-06-15T01:27:08Z

@piiswrong I have made the following changes:

set min_aspect_ratio to optional<float>
moved rotation and padding before random_resized_crop and random_crop
if neither random_resized_crop nor random_crop has been applied, perform center_crop

piiswrong · 2018-06-18T23:27:05Z

example/image-classification/common/data.py

-                     help='max change of aspect ratio, whose range is [0, 1]')
+    aug.add_argument('--min-random-aspect-ratio', type=float, default=1,
+                     help='min value of aspect ratio, whose value should be positive.')
+    aug.add_argument('--max-random-aspect-ratio', type=float, default=1,


should the default be max-random-aspect-ratio = 0 and min-random-aspect-ratio = None so that we don't break previous API?

* add resnet augmentation * add test * fix scope * fix warning * fix lint * fix lint * add color jitter and pca noise * fix center crop * merge * fix lint * Trigger CI * fix * fix augmentation implementation * add checks for parameters * modify training script * fix compile error * Trigger CI * Trigger CI * modify error message * Trigger CI * Trigger CI * Trigger CI * improve script in example * fix script * clear code * Trigger CI * set min_aspect_ratio to optional, move rotation and pad before random resized crop * fix * Trigger CI * Trigger CI * Trigger CI * fix default values * Trigger CI

TccccD · 2018-09-17T07:30:45Z

Although it is standard during training, it cannot be resized to 256*256 and center crop in validation.
@hetong007 @rahul003 @piiswrong

hetong007 · 2018-09-17T17:32:26Z

@TccccD What is the exact issue you have? FYI for training you can set up the pipeline like this: https://github.com/dmlc/gluon-cv/blob/master/scripts/classification/imagenet/train_imagenet.py#L183

hetong007 added 6 commits May 22, 2018 17:40

add resnet augmentation

987a112

add test

626aa7d

fix scope

252188e

fix warning

b699b75

fix lint

883c58e

fix lint

3580a0d

hetong007 added 5 commits May 23, 2018 12:58

add color jitter and pca noise

6cc27cc

fix center crop

8906933

merge

a693eac

fix lint

662bbf7

merge

f6a7177

hetong007 requested a review from szha as a code owner May 23, 2018 23:12

hetong007 added 3 commits May 23, 2018 17:36

Trigger CI

a777cb9

fix

0c1c89b

Merge branch 'resnet_aug' of https://github.com/hetong007/incubator-m…

74ee0c4

…xnet into resnet_aug

piiswrong suggested changes May 24, 2018

View reviewed changes

fix augmentation implementation

5e99e7c

rahul003 reviewed May 25, 2018

View reviewed changes

hetong007 and others added 10 commits June 5, 2018 16:21

Merge remote-tracking branch 'upstream/master' into resnet_aug

e171803

add checks for parameters

6b388ac

Merge branch 'resnet_aug' of github.com:hetong007/incubator-mxnet int…

9615bf6

…o resnet_aug

modify training script

9e372b0

merge with upstream

8edf384

fix compile error

bae03f4

Trigger CI

03b8f9f

Trigger CI

54c077e

modify error message

37f65fe

Merge branch 'resnet_aug' of https://github.com/hetong007/incubator-m…

b6bc840

…xnet into resnet_aug

Trigger CI

12e0009

rahul003 reviewed Jun 8, 2018

View reviewed changes

hetong007 added 2 commits June 8, 2018 16:33

improve script in example

4c79a06

fix script

27b99e3

piiswrong reviewed Jun 12, 2018

View reviewed changes

hetong007 commented Jun 12, 2018

View reviewed changes

hetong007 added 2 commits June 11, 2018 21:22

clear code

c8a3bfa

Trigger CI

966360c

piiswrong reviewed Jun 14, 2018

View reviewed changes

hetong007 added 2 commits June 14, 2018 16:41

set min_aspect_ratio to optional, move rotation and pad before random…

ecdb44a

… resized crop

fix

6a848bb

hetong007 added 4 commits June 14, 2018 20:39

Trigger CI

f4a0195

Trigger CI

3161574

Trigger CI

0fd347a

Merge remote-tracking branch 'upstream/master' into resnet_aug

22719b1

piiswrong reviewed Jun 18, 2018

View reviewed changes

fix default values

43c71e8

hetong007 mentioned this pull request Jun 19, 2018

Image classification training with RecordIter dmlc/gluon-cv#161

Merged

Trigger CI

a0e3015

piiswrong merged commit ccee176 into apache:master Jun 19, 2018

rahul003 mentioned this pull request Jun 28, 2018

Performance regression in augmentation #11469

Closed

rahul003 mentioned this pull request Jul 19, 2018

Fix image classification scripts and Improve Fp16 tutorial #11533

Merged

7 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add standard ResNet data augmentation for ImageRecordIter #11027

Add standard ResNet data augmentation for ImageRecordIter #11027

hetong007 commented May 23, 2018 •

edited

Loading

piiswrong commented May 23, 2018

rahul003 commented May 23, 2018

piiswrong May 24, 2018

hetong007 May 24, 2018

piiswrong Jun 12, 2018

piiswrong May 24, 2018

hetong007 May 24, 2018

piiswrong May 24, 2018

piiswrong May 24, 2018

hetong007 May 24, 2018

rahul003 May 25, 2018

rahul003 Jun 8, 2018 •

edited

Loading

hetong007 Jun 8, 2018

rahul003 Jun 8, 2018 •

edited

Loading

hetong007 Jun 8, 2018

DickJC123 Jul 6, 2018

rahul003 Jul 19, 2018

rahul003 Jun 8, 2018

hetong007 Jun 8, 2018

rahul003 Jun 8, 2018

piiswrong Jun 12, 2018

hetong007 Jun 12, 2018

piiswrong Jun 14, 2018

hetong007 Jun 14, 2018

piiswrong Jun 14, 2018

hetong007 Jun 14, 2018

piiswrong Jun 14, 2018

hetong007 Jun 14, 2018

hetong007 commented Jun 15, 2018

piiswrong Jun 18, 2018

TccccD commented Sep 17, 2018

hetong007 commented Sep 17, 2018

Add standard ResNet data augmentation for ImageRecordIter #11027

Add standard ResNet data augmentation for ImageRecordIter #11027

Conversation

hetong007 commented May 23, 2018 • edited Loading

Description

Checklist

Essentials

piiswrong commented May 23, 2018

rahul003 commented May 23, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rahul003 Jun 8, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rahul003 Jun 8, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hetong007 commented Jun 15, 2018

Choose a reason for hiding this comment

TccccD commented Sep 17, 2018

hetong007 commented Sep 17, 2018

hetong007 commented May 23, 2018 •

edited

Loading

rahul003 Jun 8, 2018 •

edited

Loading

rahul003 Jun 8, 2018 •

edited

Loading