[Feature] Support eval concate dataset and add tool to show dataset #833

FreyWang · 2021-08-27T16:10:48Z

This pr

add tool to show mask
support concate dataset eval and format result
fix bug of metric nan when pre_eval=False

Add tool to show origin train set and augmented train set

Add file tools/browse_dataset.py, ConcatDataset and RepeatDataset is also supported.

usage

python tools/browse_dataset.py {CONFIG}
it will save augmented train image to args.output-dir
python tools/browse_dataset.py {CONFIG} --show-origin
it will save origin train image to args.output-dir

Support eval concate dataset

this version is compatible with progressive eval #PR709

usage

The format of concate val and test dataset is same as train dataset. If separate_eval=True, it will eval every sub dataset separately, else eval as a whole dataset.

    test=dict(
        type=dataset_type,
        data_root=data_root,
        img_dir=['images1/validation',
                 'images2/validation'],
        ann_dir=['annotations1/validation',
                 'annotations2/validation'],
        separate_eval=False,
        pipeline=test_pipeline))

CityscapesDataset is not support in concat dataset

Modification

mmseg/datasets/custom.py

Add 'gt_seg_maps' argument, used in evaluation of concate dataset
Assert self.CLASSES is not None in test mode to avoid call generator gt_seg_maps repeatedly

mmseg/datasets/dataset_wrapper.py

Add evaluate() to ConcatDataset. When separate_eval=True, each subset is evaluated separately. When separate_eval=False, the generator gt_seg_maps of each subset is merged to calculate whole result.
Add pre_eval() to be compatible with progressive eval
Add format_results() to save each result, all image result from different subset will be saved to imgfile_prefix/{dataset_idx}

Some numerical results

Use configs/fcn/fcn_r50-d8_512x1024_40k_cityscapes.py and its trained checkpoint as demo, when repeat test set twice

separate_eval=True

zsh tools/dist_test.sh configs/fcn/fcn_r50-d8_512x512_80k_ade20k.py fcn_r50-d8_512x512_80k_ade20k_20200614_144016-f8ac5082.pth 8 --eval mIoU  --options data.test.img_dir="[images/validation,images/validation]" data.test.ann_dir="[annotations/validation,annotations/validation]"  data.test.separate_eval=True

It will eval twice, each 2000 image result is 35.94
sub set 1:

+-------+-------+-------+
|  aAcc |  mIoU |  mAcc |
+-------+-------+-------+
| 77.39 | 35.94 | 45.69 |
+-------+-------+-------+

sub set 2:

+-------+-------+-------+
|  aAcc |  mIoU |  mAcc |
+-------+-------+-------+
| 77.39 | 35.94 | 45.69 |
+-------+-------+-------+

separate_eval=False

zsh tools/dist_test.sh configs/fcn/fcn_r50-d8_512x512_80k_ade20k.py fcn_r50-d8_512x512_80k_ade20k_20200614_144016-f8ac5082.pth 8 --eval mIoU  --options data.test.img_dir="[images/validation,images/validation]" data.test.ann_dir="[annotations/validation,annotations/validation]"  data.test.separate_eval=False

It will eval 2000*2 image as a whole, the result is alse 35.94

[>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 4000/4000, 9.4 task/s, elapsed: 426s, ETA:     0sper class results:

+---------------------+-------+-------+
|        Class        |  IoU  |  Acc  |
+---------------------+-------+-------+
Summary:

+-------+-------+-------+
|  aAcc |  mIoU |  mAcc |
+-------+-------+-------+
| 77.39 | 35.94 | 45.69 |
+-------+-------+-------+

Junjun2016 · 2021-08-28T06:59:46Z

Hi @FreyWang
Thanks for updating.
We will review it ASAP.

Junjun2016 · 2021-08-28T07:03:25Z

Please use pre-commit to fix the lint error.

Junjun2016 · 2021-08-28T07:06:39Z

Please also use pytest to check the code error or compatibility.

Junjun2016 · 2021-08-30T05:44:52Z

mmseg/datasets/dataset_wrappers.py

@@ -15,10 +20,107 @@ class ConcatDataset(_ConcatDataset):
        datasets (list[:obj:`Dataset`]): A list of datasets.
    """

-    def __init__(self, datasets):
+    def __init__(self, datasets, separate_eval=True):


Docstring for separate_eval.

hi, I hava fix the issue and add unittest for it, does I need to submit a new PR or not?

Junjun2016 · 2021-08-30T09:42:58Z

mmseg/datasets/custom.py

@@ -99,6 +99,9 @@ def __init__(self,
        self.label_map = None
        self.CLASSES, self.PALETTE = self.get_classes_and_palette(
            classes, palette)
+        if test_mode:
+            assert self.CLASSES is not None, \
+                '`cls.CLASSES` or `classes` should be specified when testing'


It seems that this modify leads to failed github CI (checked).
Could you please add some unittests and fix the failed unitsests ?

https://github.com/open-mmlab/mmsegmentation/pull/833/checks?check_run_id=3449373682

https://github.com/open-mmlab/mmsegmentation/pull/833/checks?check_run_id=3449373726

OK, I will find time to fix the issue above 😂

Junjun2016 · 2021-08-30T11:47:58Z

mmseg/datasets/dataset_wrappers.py

+            raise NotImplementedError(
+                'All the datasets should have same types when self.separate_eval=False')
+        else:
+            gt_seg_maps = chain(*[dataset.get_gt_seg_maps()


If the results are pre_eval results, we do not need gt_seg_maps.

Please refer to https://github.com/open-mmlab/mmsegmentation/blob/master/mmseg/datasets/custom.py#L365-L386.

If the results are pre_eval results, we do not need gt_seg_maps.

yes, but if pre_eval = False when training, it may case error

mmsegmentation/configs/_base_/schedules/schedule_20k.py

Line 9 in a7461d9

evaluation = dict(interval=2000, metric='mIoU', pre_eval=True)

I mean if the results are pre_eval results, we do not need gt_seg_maps and set gt_seg_maps=None.
We only need to collect gt_seg_maps when the results are eval results.

got it 😯

Junjun2016 · 2021-08-30T11:59:41Z

mmseg/datasets/dataset_wrappers.py

+    def format_results(self, results, imgfile_prefix, indices=None, **kwargs):
+        """format result for every sample of ConcatDataset """
+        ret_res = []
+        for i, indice in enumerate(indices):


How about indices=None？
Maybe we need handle this case.

How about indices=None？
Maybe we need handle this case.

you are right, I will fix it

Junjun2016 · 2021-09-02T05:45:50Z

mmseg/datasets/dataset_wrappers.py

+            gt_seg_maps = chain(*[dataset.get_gt_seg_maps()
+                                  for dataset in self.datasets])


Suggested change

gt_seg_maps = chain(*[dataset.get_gt_seg_maps()

for dataset in self.datasets])

if mmcv.is_list_of(results, np.ndarray) or mmcv.is_list_of(

results, str):

gt_seg_maps = chain(*[dataset.get_gt_seg_maps()

for dataset in self.datasets])

else:

gt_seg_maps = None

Does this work?
Please have a check.

ok，i will

Signed-off-by: FreyWang <wangwxyz@qq.com>

Junjun2016 · 2021-09-02T14:03:05Z

Update in this PR. 发自网易邮箱大师 ---- 回复的原邮件 ---- 发件人 ***@***.***> 日期 2021年09月02日 21:53 收件人 ***@***.***> 抄送至 ***@***.******@***.***> 主题 Re: [open-mmlab/mmsegmentation] [Feature] Support eval concate dataset and add tool to show dataset (#833) @FreyWang commented on this pull request. In mmseg/datasets/dataset_wrappers.py: > @@ -15,10 +20,107 @@ class ConcatDataset(_ConcatDataset): datasets (list[:obj:`Dataset`]): A list of datasets. """ - def __init__(self, datasets): + def __init__(self, datasets, separate_eval=True): hi, I hava fix the issue and add unittest for it, does I need to submit a new PR or not? — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

FreyWang · 2021-09-02T14:22:44Z

Update in this PR. 发自网易邮箱大师 ---- 回复的原邮件 ---- 发件人 @.> 日期 2021年09月02日 21:53 收件人 @.> 抄送至 @.@.> 主题 Re: [open-mmlab/mmsegmentation] [Feature] Support eval concate dataset and add tool to show dataset (#833) @FreyWang commented on this pull request. In mmseg/datasets/dataset_wrappers.py: > @@ -15,10 +20,107 @@ class ConcatDataset(_ConcatDataset): datasets (list[:obj:Dataset]): A list of datasets. """ - def init(self, datasets): + def init(self, datasets, separate_eval=True): hi, I hava fix the issue and add unittest for it, does I need to submit a new PR or not? — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

Done. Additionally, one bug fix have been commited, please review it too. bf48690
len(list(gt_seg_maps)) will lead the generator to be empty and return nan metric when pre_eval=False

codecov · 2021-09-02T14:57:17Z

Codecov Report

Merging #833 (28e3bd2) into master (d35fbbd) will increase coverage by 0.10%.
The diff coverage is 97.33%.

@@            Coverage Diff             @@
##           master     #833      +/-   ##
==========================================
+ Coverage   88.90%   89.00%   +0.10%     
==========================================
  Files         110      110              
  Lines        5928     5992      +64     
  Branches      950      966      +16     
==========================================
+ Hits         5270     5333      +63     
- Misses        465      466       +1     
  Partials      193      193

Flag	Coverage Δ
unittests	`88.98% <97.33%> (+0.10%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
mmseg/core/evaluation/metrics.py	`90.42% <ø> (-0.20%)`	⬇️
mmseg/datasets/dataset_wrappers.py	`97.67% <97.01%> (-2.33%)`	⬇️
mmseg/datasets/builder.py	`89.61% <100.00%> (+0.13%)`	⬆️
mmseg/datasets/custom.py	`92.09% <100.00%> (-0.05%)`	⬇️
mmseg/datasets/ade.py	`93.93% <0.00%> (+3.03%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d35fbbd...28e3bd2. Read the comment docs.

Junjun2016 · 2021-09-02T15:26:53Z

mmseg/core/evaluation/metrics.py

@@ -112,8 +112,6 @@ def total_intersect_and_union(results,
         ndarray: The prediction histogram on all classes.
         ndarray: The ground truth histogram on all classes.
    """
-    num_imgs = len(results)
-    assert len(list(gt_seg_maps)) == num_imgs


Why remove these assert?

list(gt_seg_maps) will loop the generator and then gt_seg_maps will be empty,

mmsegmentation/mmseg/core/evaluation/metrics.py

Line 121 in 4981ff6

for result, gt_seg_map in zip(results, gt_seg_maps):

will case error, lead metric to be nan, I have add unittest in bf48690.

There are still missing some lines.
You can view it through files changed.

If the assert is not remove,

mmsegmentation/tests/test_data/test_dataset.py

Line 172 in 4981ff6

eval_results = train_dataset.evaluate(pseudo_results, metric=['mIoU'])

The eval result will be nan

Junjun2016 · 2021-09-02T15:31:28Z

Could you please fix the lint error and add more unittests to improve the coverage?

FreyWang · 2021-09-02T15:36:36Z

Could you please fix the lint error and add more unittests to improve the coverage?

OK, I will check again. Actually I did use pre-commit to refactor the code😕

Junjun2016 · 2021-09-02T15:38:13Z

Could you please fix the lint error and add more unittests to improve the coverage?

OK, I will check again. Actually I did use pre-commit to refactor the code

xvjiarui · 2021-09-02T16:53:26Z

mmseg/datasets/builder.py

@@ -30,6 +30,7 @@ def _concat_dataset(cfg, default_args=None):
    img_dir = cfg['img_dir']
    ann_dir = cfg.get('ann_dir', None)
    split = cfg.get('split', None)
+    separate_eval = cfg.get('separate_eval', True)


We may pop separate_eval here?

let me check😢

xvjiarui · 2021-09-02T16:56:36Z

Please fix the lint

Signed-off-by: FreyWang <wangwxyz@qq.com>

Signed-off-by: FreyWang <wangwxyz@qq.com> # Conflicts: # tools/test.py

FreyWang

Could you please fix the lint error and add more unittests to improve the coverage?

Done

FreyWang · 2021-09-03T03:01:55Z

mmseg/datasets/builder.py

@@ -49,6 +50,9 @@ def _concat_dataset(cfg, default_args=None):
    datasets = []
    for i in range(num_dset):
        data_cfg = copy.deepcopy(cfg)
+        # pop 'separate_eval' since it is not a valid key for common datasets.
+        if 'separate_eval' in data_cfg:
+            data_cfg.pop('separate_eval')


separate_eval has been poped here for every subset @xvjiarui

Can we use separate_eval = cfg.pop('separate_eval', True) in L33?

Can we use separate_eval = cfg.pop('separate_eval', True) in L33?

Sure, I think it will be better

updated 28e3bd2

Junjun2016

Thank you for your efforts, LGTM.

Junjun2016 · 2021-09-03T11:30:52Z

Hi @xvjiarui
Please review it again.

openmmlab-bot · 2021-09-07T17:01:07Z

Task linked: CU-gva479 Support eval concate dataset and add tool to show datase

…pen-mmlab#833) * [Feature] Add tool to show origin or augmented train data * [Feature] Support eval concate dataset * Add docstring and modify evaluate of concate dataset Signed-off-by: FreyWang <wangwxyz@qq.com> * format concat dataset in subfolder of imgfile_prefix Signed-off-by: FreyWang <wangwxyz@qq.com> * add unittest of concate dataset Signed-off-by: FreyWang <wangwxyz@qq.com> * update unittest for eval dataset with CLASSES is None Signed-off-by: FreyWang <wangwxyz@qq.com> * [FIX] bug of generator, which lead metric to nan when pre_eval=False Signed-off-by: FreyWang <wangwxyz@qq.com> * format code Signed-off-by: FreyWang <wangwxyz@qq.com> * add more unittest * add more unittest * optim concat dataset builder

Patch Release: 0.5.1

jason102811 · 2023-03-29T04:29:28Z

freywang，您好！您在MMSeg项目中给我们提的PR非常重要，感谢您付出私人时间帮助改进开源项目，相信很多开发者会从你的PR中受益。
我们非常期待与您继续合作，OpenMMLab专门成立了贡献者组织MMSIG，为贡献者们提供开源证书、荣誉体系和专享好礼，可通过添加微信：openmmlabwx 联系我们（请备注mmsig+GitHub id），由衷希望您能加入！
Dear freywang,
First of all, we want to express our gratitude for your significant PR in the MMSeg project. Your contribution is highly appreciated, and we are grateful for your efforts in helping improve this open-source project during your personal time. We believe that many developers will benefit from your PR.
We are looking forward to continuing our collaboration with you. OpenMMLab has established a special contributors' organization called MMSIG, which provides contributors with open-source certificates, a recognition system, and exclusive rewards. You can contact us by adding our WeChat（if you have WeChat): openmmlabwx, or join in our discord： https://discord.gg/qH9fysxPDW. We sincerely hope you will join us!
Best regards！ @FreyWang

…b#833)

* correct tpn sthv1 testing * Update tpn_tsm_r50_1x1x8_150e_sthv1_rgb.py

FreyWang mentioned this pull request Aug 27, 2021

[Feature] Support eval concate dataset and add tool to show dataset #781

Closed

Junjun2016 reviewed Aug 30, 2021

View reviewed changes

Junjun2016 self-requested a review August 30, 2021 12:06

FreyWang added 2 commits September 2, 2021 11:17

[Feature] Add tool to show origin or augmented train data

6fdd5ed

[Feature] Support eval concate dataset

8e75af9

Junjun2016 reviewed Sep 2, 2021

View reviewed changes

FreyWang added 6 commits September 2, 2021 17:35

Add docstring and modify evaluate of concate dataset

bc781f1

Signed-off-by: FreyWang <wangwxyz@qq.com>

format concat dataset in subfolder of imgfile_prefix

17cf91a

Signed-off-by: FreyWang <wangwxyz@qq.com>

add unittest of concate dataset

5bc4e58

Signed-off-by: FreyWang <wangwxyz@qq.com>

update unittest for eval dataset with CLASSES is None

e6501a2

Signed-off-by: FreyWang <wangwxyz@qq.com>

[FIX] bug of generator, which lead metric to nan when pre_eval=False

bf48690

Signed-off-by: FreyWang <wangwxyz@qq.com>

Merge commit '4981ff68c2ae05ccfe83e340d528bb4c3c130740' into pr/temp

980e3bf

FreyWang force-pushed the pr1 branch from 98b60c1 to 980e3bf Compare September 2, 2021 13:47

Junjun2016 reviewed Sep 2, 2021

View reviewed changes

xvjiarui reviewed Sep 2, 2021

View reviewed changes

FreyWang added 2 commits September 3, 2021 11:07

format code

26570de

Signed-off-by: FreyWang <wangwxyz@qq.com>

add more unittest

51ac8af

Merge commit 'd35fbbdb47383906399df3a23e4f9f1db1e68b2b' into pr/temp

5784eb0

Signed-off-by: FreyWang <wangwxyz@qq.com> # Conflicts: # tools/test.py

FreyWang commented Sep 3, 2021

View reviewed changes

FreyWang added 2 commits September 3, 2021 17:12

add more unittest

9352d60

optim concat dataset builder

28e3bd2

Junjun2016 approved these changes Sep 3, 2021

View reviewed changes

xvjiarui approved these changes Sep 9, 2021

View reviewed changes

Junjun2016 merged commit 872e544 into open-mmlab:master Sep 9, 2021

Junjun2016 mentioned this pull request Nov 2, 2021

[Feature] Add SOD datasets #913

Closed

MeowZheng mentioned this pull request Mar 11, 2023

Why not Cityscapes evaluation for concat dataset? #2731

Open

aravind-h-v pushed a commit to aravind-h-v/mmsegmentation that referenced this pull request Mar 27, 2023

Release 0 5 1 (open-mmlab#833)

e48ca0f

Patch Release: 0.5.1

wjkim81 pushed a commit to wjkim81/mmsegmentation that referenced this pull request Dec 3, 2023

Remove opencv-python-headless dependency by albumentations (open-mmla…

77d78f0

…b#833)

sibozhang pushed a commit to sibozhang/mmsegmentation that referenced this pull request Mar 22, 2024

use twice sample for tpn_tsm sthv1 testing (open-mmlab#833)

d32e2df

* correct tpn sthv1 testing * Update tpn_tsm_r50_1x1x8_150e_sthv1_rgb.py

		gt_seg_maps = chain(*[dataset.get_gt_seg_maps()
		for dataset in self.datasets])

[Feature] Support eval concate dataset and add tool to show dataset #833

[Feature] Support eval concate dataset and add tool to show dataset #833

Conversation

FreyWang commented Aug 27, 2021 • edited Loading

Add tool to show origin train set and augmented train set

usage

Support eval concate dataset

usage

Modification

mmseg/datasets/custom.py

mmseg/datasets/dataset_wrapper.py

Some numerical results

Junjun2016 commented Aug 28, 2021

Junjun2016 commented Aug 28, 2021

Junjun2016 commented Aug 28, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Junjun2016 commented Sep 2, 2021 via email

FreyWang commented Sep 2, 2021

codecov bot commented Sep 2, 2021 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Junjun2016 commented Sep 2, 2021

FreyWang commented Sep 2, 2021

Junjun2016 commented Sep 2, 2021

Choose a reason for hiding this comment

FreyWang Sep 3, 2021 • edited Loading

Choose a reason for hiding this comment

xvjiarui commented Sep 2, 2021

FreyWang left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Junjun2016 left a comment

Choose a reason for hiding this comment

Junjun2016 commented Sep 3, 2021

openmmlab-bot commented Sep 7, 2021

jason102811 commented Mar 29, 2023

FreyWang commented Aug 27, 2021 •

edited

Loading

codecov bot commented Sep 2, 2021 •

edited

Loading

FreyWang Sep 3, 2021 •

edited

Loading