[Feature] Add MultiImageMixDataset #1105

lkm2835 · 2021-12-06T10:18:33Z

Modification

mmseg/datasets/builder.py: Add (if cfg['type'] == 'MultiImageMixDataset'])
mmseg/datasets/dataset_wrappers.py: Add class MultiImageMixDataset
tests/test_data/test_dataset.py: Add unittests

Use cases (Optional)

train_pipeline = [
    dict(type='Mosaic'),
    dict(type='Resize', img_scale=(1024, 512), keep_ratio=True),
    dict(type='RandomFlip', prob=0.5),
    dict(type='Normalize', **img_norm_cfg),
    dict(type='DefaultFormatBundle'),
    dict(type='Collect', keys=['img', 'gt_semantic_seg']),
]

train_dataset = dict(
    type='MultiImageMixDataset',
    dataset=dict(
        classes=classes,
        palette=palette,
        type=dataset_type,
        reduce_zero_label=False, 
        img_dir=data_root + "images/train",
        ann_dir=data_root + "annotations/train",
        pipeline=[
            dict(type='LoadImageFromFile'),
            dict(type='LoadAnnotations'),
        ]  
    ),
    pipeline=train_pipeline
)

use case in mmdet

Original code: MultiImageMixDataset in mmdet
Related: Issue#1045, Pull Request#1093

* remove dynamic_scale & add palette * modify retrieve_data_cfg method * modify retrieve_data_cfg func

CLAassistant · 2021-12-06T10:18:37Z

All committers have signed the CLA.

codecov · 2021-12-06T11:06:09Z

Codecov Report

Merging #1105 (664d5da) into master (91cbe06) will increase coverage by 0.45%.
The diff coverage is 84.00%.

@@            Coverage Diff             @@
##           master    #1105      +/-   ##
==========================================
+ Coverage   89.57%   90.03%   +0.45%     
==========================================
  Files         120      125       +5     
  Lines        6717     7314     +597     
  Branches     1122     1219      +97     
==========================================
+ Hits         6017     6585     +568     
- Misses        496      524      +28     
- Partials      204      205       +1

Flag	Coverage Δ
unittests	`90.03% <84.00%> (+0.45%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
mmseg/datasets/dataset_wrappers.py	`92.12% <81.39%> (-5.55%)`	⬇️
mmseg/datasets/__init__.py	`100.00% <100.00%> (ø)`
mmseg/datasets/builder.py	`87.80% <100.00%> (+0.79%)`	⬆️
mmseg/models/segmentors/base.py	`57.85% <0.00%> (-4.10%)`	⬇️
mmseg/models/backbones/swin.py	`83.62% <0.00%> (-0.12%)`	⬇️
mmseg/models/losses/__init__.py	`100.00% <0.00%> (ø)`
mmseg/models/losses/dice_loss.py	`100.00% <0.00%> (ø)`
mmseg/models/backbones/__init__.py	`100.00% <0.00%> (ø)`
mmseg/models/decode_heads/__init__.py	`100.00% <0.00%> (ø)`
mmseg/models/backbones/twins.py	`99.40% <0.00%> (ø)`
... and 9 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 91cbe06...664d5da. Read the comment docs.

MengzhangLI · 2021-12-06T11:57:52Z

Hi, @lkm2835 thanks for your pr.

We would review it ASAP.

Best,

Younghoon-Lee · 2021-12-06T12:59:17Z

This is the sample image that is built by MultiImageMixDataset from our custom dataset.
As you can see, four images come together to form a single square.

Junjun2016 · 2021-12-06T13:45:17Z

This is the sample image that is built by MultiImageMixDataset from our custom dataset. As you can see, four images come together to form a single square.

Could you please provide some results on your custom dataset?

Younghoon-Lee · 2021-12-06T16:33:18Z

Could you please provide some results on your custom dataset?

Okay I see . We will do some tests with and without Mosaic. Please note that our custom dataset has a few noises in annotation.

mmseg/datasets/dataset_wrappers.py

tools/browse_dataset.py

Junjun2016 · 2021-12-06T17:07:43Z

This is the sample image that is built by MultiImageMixDataset from our custom dataset. As you can see, four images come together to form a single square.

Could you please provide some results on your custom dataset?

It seems that you have set the opacity to 0, can set it to 0.5 to also show the label annotations.

Junjun2016 · 2021-12-06T17:09:54Z

Please improve the unittests coverage.

Junjun2016 · 2021-12-06T17:32:10Z

Can browse more images and label annotations to check the correctness of mosaic augmentation.

Junjun2016 · 2021-12-06T17:34:24Z

Hi @RockeyCoss
Please review it and do some ablation studies.

* add cfg-options

Younghoon-Lee · 2021-12-08T15:12:13Z

We made changes you requested. And I browsed about 100 images and annotations and they seemed all fine :)

Junjun2016 · 2021-12-08T15:44:37Z

We made changes you requested. And I browsed about 100 images and annotations and they seemed all fine :)

Thanks for your hard work.
We could do some ablation studies together.
Looking forward to the result of your project.

RockeyCoss · 2021-12-09T06:37:15Z

Thanks for your contribution. I will combine it with Mosaic data augmentation and do some experiments.

lkm2835 · 2021-12-11T17:15:45Z

mmseg/datasets/dataset_wrappers.py line 247 ~ 259

When it merged with Mosaic, it will be easy to improve the unittests coverage.

Younghoon-Lee · 2021-12-16T16:00:21Z

Thanks for your hard work. We could do some ablation studies together. Looking forward to the result of your project.

Sorry for taking so late. I brought some results below.
All results were applied of 512 img scale for Mosaic augmentation.

Mosaic_prob	best	best epoch	last	last epoch
0	0.5752	25	0.5685	50
0.5	0.5728	35	0.5696	50
1.0	0.5814	49	0.579	50

We have done different img scales test for Mosaic augmentation as well, but 512x512 showed best performance. (Mosaic prob =1.0)

img_scale	best	best epoch	last	last epoch
384	0.5297	38	0.5269	50
448	0.5632	49	0.5627	50
512	0.5814	49	0.579	50
576	0.5475	41	0.5405	50
640	0.544	46	0.5411	50

According to test results, Mosaic augmentation seems to show better performance on a few label images.

Junjun2016 · 2021-12-16T16:09:00Z

Thanks for your hard work. We could do some ablation studies together. Looking forward to the result of your project.

Sorry for taking so late. I brought some results below. All results were applied of 512 img scale for Mosaic augmentation.

Mosaic_prob best best epoch last last epoch
0 0.5752 25 0.5685 50
0.5 0.5728 35 0.5696 50
1.0 0.5814 49 0.579 50
We have done different img scales test for Mosaic augmentation as well, but 512x512 showed best performance. (Mosaic prob =1.0)

img_scale best best epoch last last epoch
384 0.5297 38 0.5269 50
448 0.5632 49 0.5627 50
512 0.5814 49 0.579 50
576 0.5475 41 0.5405 50
640 0.544 46 0.5411 50
According to test results, Mosaic augmentation seems to show better performance on a few label images.

What's the best result on your custom dataset (SOTA)?

Younghoon-Lee · 2021-12-16T16:30:31Z

What's the best result on your custom dataset (SOTA)?

The best result is 0.5814 ,which is 512x512 img scale for Mosaic(prob=1) augmentation.

Here is config script for more detail.

train_pipeline = [
    dict(type='RandomMosaic', prob=1, img_scale=(512,512)),
    dict(type='RandomCrop', crop_size=(512, 512)),
    dict(type='RandomFlip', prob=0.5),
    dict(type='Normalize', **img_norm_cfg),
    dict(type='DefaultFormatBundle'),
    dict(type='Collect', keys=['img', 'gt_semantic_seg']),
]
test_pipeline = [
    dict(type='LoadImageFromFile'),
    dict(
        type='MultiScaleFlipAug',
        img_scale=(512,512),
        flip=False,
        transforms=[
            dict(type='Resize', keep_ratio=True),
            dict(type='RandomFlip'),
            dict(type='Normalize', **img_norm_cfg),
            dict(type='ImageToTensor', keys=['img']),
            dict(type='Collect', keys=['img']),
        ])
]

(segformer_mit-b0_512x512)

Junjun2016 · 2021-12-16T16:36:50Z

What's the best result on your custom dataset (SOTA)?

The best result is 0.5814 ,which is 512x512 img scale for Mosaic(prob=1) augmentation.

Here is config script for more detail.

train_pipeline = [
    dict(type='RandomMosaic', prob=1, img_scale=(512,512)),
    dict(type='RandomCrop', crop_size=(512, 512)),
    dict(type='RandomFlip', prob=0.5),
    dict(type='Normalize', **img_norm_cfg),
    dict(type='DefaultFormatBundle'),
    dict(type='Collect', keys=['img', 'gt_semantic_seg']),
]
test_pipeline = [
    dict(type='LoadImageFromFile'),
    dict(
        type='MultiScaleFlipAug',
        img_scale=(512,512),
        flip=False,
        transforms=[
            dict(type='Resize', keep_ratio=True),
            dict(type='RandomFlip'),
            dict(type='Normalize', **img_norm_cfg),
            dict(type='ImageToTensor', keys=['img']),
            dict(type='Collect', keys=['img']),
        ])
]

(segformer_mit-b0_512x512)

I mean the best result you got before.
Can mosaic augmentation boost the best result?

Younghoon-Lee · 2021-12-16T17:19:11Z

I mean the best result you got before. Can mosaic augmentation boost the best result?

Oh I see.

Unfortunately, now we only have the best result which had been applied many tricks(due to competition),so it is hard to tell the best result of the model itself.

We will do test right away with same condition as before .

Junjun2016 · 2021-12-18T14:27:38Z

I mean the best result you got before. Can mosaic augmentation boost the best result?

Oh I see.

Unfortunately, now we only have the best result which had been applied many tricks(due to competition),so it is hard to tell the best result of the model itself.

We will do test right away with same condition as before .

That may not be a good data augmentation strategy for segmentation.

Younghoon-Lee · 2021-12-18T14:35:12Z

I mean the best result you got before. Can mosaic augmentation boost the best result?

Oh I see.
Unfortunately, now we only have the best result which had been applied many tricks(due to competition),so it is hard to tell the best result of the model itself.
We will do test right away with same condition as before .

That may not be a good data augmentation strategy for segmentation.

So we are just using SOTA model(without any tricks) we used and doing some augmentation tests only.

Junjun2016 · 2021-12-18T14:41:21Z

I mean the best result you got before. Can mosaic augmentation boost the best result?

Oh I see.
Unfortunately, now we only have the best result which had been applied many tricks(due to competition),so it is hard to tell the best result of the model itself.
We will do test right away with same condition as before .

That may not be a good data augmentation strategy for segmentation.

So we are just using SOTA model(without any tricks) we used and doing some augmentation tests only.

Make sense.

Younghoon-Lee · 2021-12-20T10:41:30Z

Here's some results we have done.

exp_num	method	mIoU best	best epoch	mIoU last	last epoch
1(SOTA)	512x512_epoch18	0.7627	11	0.7611	18
2	512x512_epoch18_mosaic_prob_0.2	0.7588	11	0.7571	18
3	512x512_epoch18_mosaic_prob_0.3	0.7577	14	0.7558	18
4	512x512_epoch18_mosaic_prob_0.5	0.7567	13	0.7566	18
5	512x512_epoch18_mosaic_prob_1.0	0.755	18	0.755	18

Actually, we did several more tests such as without crop, changing center ratio, img scale etc .
But It seemed that Mosaic couldn't boost our best result.

mmseg/datasets/dataset_wrappers.py

Co-authored-by: Miao Zheng <76149310+MeowZheng@users.noreply.github.com>

* Fix typo in usage example * original MultiImageMixDataset code in mmdet * Add MultiImageMixDataset unittests in test_dataset_wrapper * fix lint error * fix value name ann_file to ann_dir * modify retrieve_data_cfg (#1) * remove dynamic_scale & add palette * modify retrieve_data_cfg method * modify retrieve_data_cfg func * fix error * improve the unittests coverage * fix unittests error * Dataset (#2) * add cfg-options * Add unittest in test_build_dataset * add blank line * add blank line * add a blank line Co-authored-by: Miao Zheng <76149310+MeowZheng@users.noreply.github.com> Co-authored-by: Younghoon-Lee <72462227+Younghoon-Lee@users.noreply.github.com> Co-authored-by: MeowZheng <meowzheng@outlook.com> Co-authored-by: Miao Zheng <76149310+MeowZheng@users.noreply.github.com>

lkm2835 and others added 14 commits October 29, 2021 00:12

Fix typo in usage example

525bb6b

Merge branch 'open-mmlab:master' into master

e34d8b4

Merge branch 'open-mmlab:master' into master

0328021

Merge branch 'open-mmlab:master' into master

13a3b00

Merge branch 'open-mmlab:master' into master

4e1f1e3

Merge branch 'open-mmlab:master' into master

bcc4399

Merge branch 'open-mmlab:master' into master

d186b2a

original MultiImageMixDataset code in mmdet

46c4e92

Add MultiImageMixDataset unittests in test_dataset_wrapper

4ea0432

fix lint error

dbcf753

fix value name ann_file to ann_dir

1d37504

modify retrieve_data_cfg (#1)

8ba8e2a

* remove dynamic_scale & add palette * modify retrieve_data_cfg method * modify retrieve_data_cfg func

fix error

acfb228

Merge 'origin/dataset' into dataset

ad65763

Junjun2016 reviewed Dec 6, 2021

View reviewed changes

mmseg/datasets/dataset_wrappers.py Outdated Show resolved Hide resolved

Junjun2016 reviewed Dec 6, 2021

View reviewed changes

tools/browse_dataset.py Show resolved Hide resolved

lkm2835 and others added 3 commits December 7, 2021 10:49

improve the unittests coverage

f62f781

fix unittests error

dc0878c

Dataset (#2)

17d9e57

* add cfg-options

Add unittest in test_build_dataset

08b5c33

lkm2835 mentioned this pull request Dec 16, 2021

[Feature] Add Mosaic transform #1093

Merged

Junjun2016 approved these changes Dec 29, 2021

View reviewed changes

MeowZheng reviewed Jan 7, 2022

View reviewed changes

mmseg/datasets/dataset_wrappers.py Show resolved Hide resolved

MeowZheng approved these changes Jan 7, 2022

View reviewed changes

MeowZheng and others added 3 commits January 7, 2022 23:01

add blank line

fc05df0

add blank line

cf9b38a

add a blank line

664d5da

Co-authored-by: Miao Zheng <76149310+MeowZheng@users.noreply.github.com>

RockeyCoss mentioned this pull request Jan 11, 2022

[Docs] Add MultiImageMixDataset tutorial #1194

Merged

Junjun2016 merged commit 6c3e63e into open-mmlab:master Jan 11, 2022

wjkim81 pushed a commit to wjkim81/mmsegmentation that referenced this pull request Dec 3, 2023

fix ap-10k dataset reference (open-mmlab#1105)

76060f4

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature] Add MultiImageMixDataset #1105

[Feature] Add MultiImageMixDataset #1105

lkm2835 commented Dec 6, 2021 •

edited

Loading

CLAassistant commented Dec 6, 2021 •

edited

Loading

codecov bot commented Dec 6, 2021 •

edited

Loading

MengzhangLI commented Dec 6, 2021

Younghoon-Lee commented Dec 6, 2021 •

edited

Loading

Junjun2016 commented Dec 6, 2021

Younghoon-Lee commented Dec 6, 2021

Junjun2016 commented Dec 6, 2021

Junjun2016 commented Dec 6, 2021

Junjun2016 commented Dec 6, 2021

Junjun2016 commented Dec 6, 2021

Younghoon-Lee commented Dec 8, 2021

Junjun2016 commented Dec 8, 2021

RockeyCoss commented Dec 9, 2021

lkm2835 commented Dec 11, 2021 •

edited

Loading

Younghoon-Lee commented Dec 16, 2021 •

edited

Loading

Junjun2016 commented Dec 16, 2021

Younghoon-Lee commented Dec 16, 2021

Junjun2016 commented Dec 16, 2021

Younghoon-Lee commented Dec 16, 2021 •

edited

Loading

Junjun2016 commented Dec 18, 2021

Younghoon-Lee commented Dec 18, 2021

Junjun2016 commented Dec 18, 2021

Younghoon-Lee commented Dec 20, 2021 •

edited

Loading

[Feature] Add MultiImageMixDataset #1105

[Feature] Add MultiImageMixDataset #1105

Conversation

lkm2835 commented Dec 6, 2021 • edited Loading

Modification

Use cases (Optional)

CLAassistant commented Dec 6, 2021 • edited Loading

codecov bot commented Dec 6, 2021 • edited Loading

Codecov Report

MengzhangLI commented Dec 6, 2021

Younghoon-Lee commented Dec 6, 2021 • edited Loading

Junjun2016 commented Dec 6, 2021

Younghoon-Lee commented Dec 6, 2021

Junjun2016 commented Dec 6, 2021

Junjun2016 commented Dec 6, 2021

Junjun2016 commented Dec 6, 2021

Junjun2016 commented Dec 6, 2021

Younghoon-Lee commented Dec 8, 2021

Junjun2016 commented Dec 8, 2021

RockeyCoss commented Dec 9, 2021

lkm2835 commented Dec 11, 2021 • edited Loading

Younghoon-Lee commented Dec 16, 2021 • edited Loading

Junjun2016 commented Dec 16, 2021

Younghoon-Lee commented Dec 16, 2021

Junjun2016 commented Dec 16, 2021

Younghoon-Lee commented Dec 16, 2021 • edited Loading

Junjun2016 commented Dec 18, 2021

Younghoon-Lee commented Dec 18, 2021

Junjun2016 commented Dec 18, 2021

Younghoon-Lee commented Dec 20, 2021 • edited Loading

lkm2835 commented Dec 6, 2021 •

edited

Loading

CLAassistant commented Dec 6, 2021 •

edited

Loading

codecov bot commented Dec 6, 2021 •

edited

Loading

Younghoon-Lee commented Dec 6, 2021 •

edited

Loading

lkm2835 commented Dec 11, 2021 •

edited

Loading

Younghoon-Lee commented Dec 16, 2021 •

edited

Loading

Younghoon-Lee commented Dec 16, 2021 •

edited

Loading

Younghoon-Lee commented Dec 20, 2021 •

edited

Loading