[Refactor] Support progressive test with fewer memory cost #709

clownrat6 · 2021-07-16T13:34:39Z

Support progressive test

Calculate Metrics

The current test tool require a middle list type variable to keep all of the model predict masks. When there are large number of test samples, many bugs will occur. We add dataset.pre_eval function which can avoid keeping all of the model prediction masks and only keep the pre-evaluation results. So, we just need to use pre_eval to calculate regular metrics, like, IoU, Dice.

Cityscapes metric may need to keep middle model predict masks. We make a judgement for Cityscapes to ensure keeping middle predict masks for metric calculation.

Format Results

We also provide a progressive mode for format_results. But the dataset.format_results must support indices argument.

ps:

usage:

python tools/test.py [config_path] [ckpt_path] --eval mIoU # Default progressive test.
python tools/test.py [config_path] [ckpt_path] --eval cityscapes # Test on formatting results.
python tools/test.py [config_path] [ckpt_path] --format-only # Default progressive test.

Modifications Abstract

mmseg/apis/test.py

Deprecate efficient_test argument; (efficient_test is still valid but the return value of test api will be prediction save paths)
Add pre_eval argument; (for progressive metric calculation)
Add format_only argument; (for progressive results format)
Add format_args argument; (for progressive results format)

tools/test.py

Add eval_on_format_results middle variable for cityscapes metric;
Deprecate efficient_test argument;

mmseg/core/evaluation/eval_hooks.py

Deprecated efficient_test argument;
Add pre_eval argument to control evaluation action of test api when training;

codecov · 2021-07-16T13:44:08Z

Codecov Report

Merging #709 (469ced8) into master (f72727c) will increase coverage by 1.54%.
The diff coverage is 78.70%.

@@            Coverage Diff             @@
##           master     #709      +/-   ##
==========================================
+ Coverage   86.09%   87.64%   +1.54%     
==========================================
  Files         108      108              
  Lines        5862     5885      +23     
  Branches      953      957       +4     
==========================================
+ Hits         5047     5158     +111     
+ Misses        630      535      -95     
- Partials      185      192       +7

Flag	Coverage Δ
unittests	`87.62% <78.70%> (+1.54%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
mmseg/apis/test.py	`46.34% <48.27%> (+15.67%)`	⬆️
mmseg/datasets/ade.py	`90.90% <72.72%> (+50.36%)`	⬆️
mmseg/datasets/cityscapes.py	`85.00% <75.00%> (+63.65%)`	⬆️
mmseg/datasets/custom.py	`92.09% <93.10%> (+2.27%)`	⬆️
mmseg/core/evaluation/__init__.py	`100.00% <100.00%> (ø)`
mmseg/core/evaluation/eval_hooks.py	`72.54% <100.00%> (+4.36%)`	⬆️
mmseg/core/evaluation/metrics.py	`90.62% <100.00%> (+1.33%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f72727c...469ced8. Read the comment docs.

mmseg/apis/test.py

mmseg/datasets/custom.py

tools/test.py

mmseg/apis/test.py

mmseg/datasets/custom.py

tools/deploy_test.py

DCNSW · 2021-08-12T15:07:54Z

MMdet3d can be supported by this pr successfully. No BC breaking issues.

* [Feature]Segformer re-implementation * Using act_cfg and norm_cfg to control activation and normalization * Split this PR into several little PRs * Fix lint error * Remove SegFormerHead * [Feature] Add segformer decode head and related train config * Add ade20K trainval support for segformer 1. Add related train and val configs; 2. Add AlignedResize; * Set arg: find_unused_parameters = True * parameters init refactor * 1. Refactor segformer backbone parameters init; 2. Remove rebundant functions and unit tests; * Remove rebundant codes * Replace Linear Layer to 1X1 Conv * Use nn.ModuleList to refactor segformer head. * Remove local to_xtuple * 1. Remove rebundant codes; 2. Modify module name; * Refactor the backbone of segformer using mmcv.cnn.bricks.transformer.py * Fix some code logic bugs. * Add mit_convert.py to match pretrain keys of segformer. * Resolve some comments. * 1. Add some assert to ensure right params; 2. Support flexible peconv position; * Add pe_index assert and fix unit test. * 1. Add doc string for MixVisionTransformer; 2. Add some unit tests for MixVisionTransformer; * Use hw_shape to pass shape of feature map. * 1. Fix doc string of MixVisionTransformer; 2. Simplify MixFFN; 3. Modify H, W to hw_shape; * Add more unit tests. * Add doc string for shape convertion functions. * Add some unit tests to improve code coverage. * Fix Segformer backbone pretrain weights match bug. * Modify configs of segformer. * resolve the shape convertion functions doc string. * Add pad_to_patch_size arg. * Support progressive test with fewer memory cost. * Modify default value of pad_to_patch_size arg. * Temp code * Using processor to refactor evaluation workflow. * refactor eval hook. * Fix process bar. * Fix middle save argument. * Modify some variable name of dataset evaluate api. * Modify some viriable name of eval hook. * Fix some priority bugs of eval hook. * Fix some bugs about model loading and eval hook. * Add ade20k 640x640 dataset. * Fix related segformer configs. * Depreciated efficient_test. * Fix training progress blocked by eval hook. * Depreciated old test api. * Modify error patch size. * Fix pretrain of mit_b0 * Fix the test api error. * Modify dataset base config. * Fix test api error. * Modify outer api. * Build a sampler test api. * TODO: Refactor format_results. * Modify variable names. * Fix num_classes bug. * Fix sampler index bug. * Fix grammaly bug. * Add part of benchmark results. * Support batch sampler. * More readable test api. * Remove some command arg and fix eval hook bug. * Support format-only arg. * Modify format_results of datasets. * Modify tool which use test apis. * Update readme. * Update readme of segformer. * Updata readme of segformer. * Update segformer readme and fix segformer mit_b4. * Update readme of segformer. * Clean AlignedResize related config. * Clean code from pr #709 * Clean code from pr #709 * Add 512x512 segformer_mit-b5. * Fix lint. * Fix some segformer head bugs. * Add segformer unit tests. * Replace AlignedResize to ResizeToMultiple. * Modify readme of segformer. * Fix bug of ResizeToMultiple. * Add ResizeToMultiple unit tests. * Resolve conflict. * Simplify the implementation of ResizeToMultiple. * Update test results. * Fix multi-scale test error when resize_ratio=1.75 and input size=640x640. * Update segformer results. * Update Segformer results. * Fix some url bugs and pipelines bug. * Move ckpt convertion to tools. * Add segformer official pretrain weights usage. * Clean redundant codes. * Remove redundant codes. * Unfied format. * Add description for segformer converter. * Update workers.

mmseg/apis/train.py

mmseg/core/evaluation/eval_hooks.py

tests/test_apis/test_single_gpu.py

mmseg/apis/train.py

…b#709) * Support progressive test with fewer memory cost. * Temp code * Using processor to refactor evaluation workflow. * refactor eval hook. * Fix process bar. * Fix middle save argument. * Modify some variable name of dataset evaluate api. * Modify some viriable name of eval hook. * Fix some priority bugs of eval hook. * Depreciated efficient_test. * Fix training progress blocked by eval hook. * Depreciated old test api. * Fix test api error. * Modify outer api. * Build a sampler test api. * TODO: Refactor format_results. * Modify variable names. * Fix num_classes bug. * Fix sampler index bug. * Fix grammaly bug. * Support batch sampler. * More readable test api. * Remove some command arg and fix eval hook bug. * Support format-only arg. * Modify format_results of datasets. * Modify tool which use test apis. * support cityscapes eval * fixed cityscapes * 1. Add comments for batch_sampler; 2. Keep eval hook api same and add deprecated warning; 3. Add doc string for dataset.pre_eval; * Add efficient_test doc string. * Modify test tool to compat old version. * Modify eval hook to compat with old version. * Modify test api to compat old version api. * Sampler explanation. * update warning * Modify deploy_test.py * compatible with old output, add efficient test back * clear logic of exclusive * Warning about efficient_test. * Modify format_results save folder. * Fix bugs of format_results. * Modify deploy_test.py. * Update doc * Fix deploy test bugs. * Fix custom dataset unit tests. * Fix dataset unit tests. * Fix eval hook unit tests. * Fix some imcompatible. * Add pre_eval argument for eval hooks. * Update eval hook doc string. * Make pre_eval false in default. * Add unit tests for dataset format_results. * Fix some comments and bc-breaking bug. * Fix pre_eval set cfg field. * Remove redundant codes. Co-authored-by: Jiarui XU <xvjiarui0826@gmail.com>

…lab#599) * [Feature]Segformer re-implementation * Using act_cfg and norm_cfg to control activation and normalization * Split this PR into several little PRs * Fix lint error * Remove SegFormerHead * [Feature] Add segformer decode head and related train config * Add ade20K trainval support for segformer 1. Add related train and val configs; 2. Add AlignedResize; * Set arg: find_unused_parameters = True * parameters init refactor * 1. Refactor segformer backbone parameters init; 2. Remove rebundant functions and unit tests; * Remove rebundant codes * Replace Linear Layer to 1X1 Conv * Use nn.ModuleList to refactor segformer head. * Remove local to_xtuple * 1. Remove rebundant codes; 2. Modify module name; * Refactor the backbone of segformer using mmcv.cnn.bricks.transformer.py * Fix some code logic bugs. * Add mit_convert.py to match pretrain keys of segformer. * Resolve some comments. * 1. Add some assert to ensure right params; 2. Support flexible peconv position; * Add pe_index assert and fix unit test. * 1. Add doc string for MixVisionTransformer; 2. Add some unit tests for MixVisionTransformer; * Use hw_shape to pass shape of feature map. * 1. Fix doc string of MixVisionTransformer; 2. Simplify MixFFN; 3. Modify H, W to hw_shape; * Add more unit tests. * Add doc string for shape convertion functions. * Add some unit tests to improve code coverage. * Fix Segformer backbone pretrain weights match bug. * Modify configs of segformer. * resolve the shape convertion functions doc string. * Add pad_to_patch_size arg. * Support progressive test with fewer memory cost. * Modify default value of pad_to_patch_size arg. * Temp code * Using processor to refactor evaluation workflow. * refactor eval hook. * Fix process bar. * Fix middle save argument. * Modify some variable name of dataset evaluate api. * Modify some viriable name of eval hook. * Fix some priority bugs of eval hook. * Fix some bugs about model loading and eval hook. * Add ade20k 640x640 dataset. * Fix related segformer configs. * Depreciated efficient_test. * Fix training progress blocked by eval hook. * Depreciated old test api. * Modify error patch size. * Fix pretrain of mit_b0 * Fix the test api error. * Modify dataset base config. * Fix test api error. * Modify outer api. * Build a sampler test api. * TODO: Refactor format_results. * Modify variable names. * Fix num_classes bug. * Fix sampler index bug. * Fix grammaly bug. * Add part of benchmark results. * Support batch sampler. * More readable test api. * Remove some command arg and fix eval hook bug. * Support format-only arg. * Modify format_results of datasets. * Modify tool which use test apis. * Update readme. * Update readme of segformer. * Updata readme of segformer. * Update segformer readme and fix segformer mit_b4. * Update readme of segformer. * Clean AlignedResize related config. * Clean code from pr open-mmlab#709 * Clean code from pr open-mmlab#709 * Add 512x512 segformer_mit-b5. * Fix lint. * Fix some segformer head bugs. * Add segformer unit tests. * Replace AlignedResize to ResizeToMultiple. * Modify readme of segformer. * Fix bug of ResizeToMultiple. * Add ResizeToMultiple unit tests. * Resolve conflict. * Simplify the implementation of ResizeToMultiple. * Update test results. * Fix multi-scale test error when resize_ratio=1.75 and input size=640x640. * Update segformer results. * Update Segformer results. * Fix some url bugs and pipelines bug. * Move ckpt convertion to tools. * Add segformer official pretrain weights usage. * Clean redundant codes. * Remove redundant codes. * Unfied format. * Add description for segformer converter. * Update workers.

…b#709) * Support progressive test with fewer memory cost. * Temp code * Using processor to refactor evaluation workflow. * refactor eval hook. * Fix process bar. * Fix middle save argument. * Modify some variable name of dataset evaluate api. * Modify some viriable name of eval hook. * Fix some priority bugs of eval hook. * Depreciated efficient_test. * Fix training progress blocked by eval hook. * Depreciated old test api. * Fix test api error. * Modify outer api. * Build a sampler test api. * TODO: Refactor format_results. * Modify variable names. * Fix num_classes bug. * Fix sampler index bug. * Fix grammaly bug. * Support batch sampler. * More readable test api. * Remove some command arg and fix eval hook bug. * Support format-only arg. * Modify format_results of datasets. * Modify tool which use test apis. * support cityscapes eval * fixed cityscapes * 1. Add comments for batch_sampler; 2. Keep eval hook api same and add deprecated warning; 3. Add doc string for dataset.pre_eval; * Add efficient_test doc string. * Modify test tool to compat old version. * Modify eval hook to compat with old version. * Modify test api to compat old version api. * Sampler explanation. * update warning * Modify deploy_test.py * compatible with old output, add efficient test back * clear logic of exclusive * Warning about efficient_test. * Modify format_results save folder. * Fix bugs of format_results. * Modify deploy_test.py. * Update doc * Fix deploy test bugs. * Fix custom dataset unit tests. * Fix dataset unit tests. * Fix eval hook unit tests. * Fix some imcompatible. * Add pre_eval argument for eval hooks. * Update eval hook doc string. * Make pre_eval false in default. * Add unit tests for dataset format_results. * Fix some comments and bc-breaking bug. * Fix pre_eval set cfg field. * Remove redundant codes. Co-authored-by: Jiarui XU <xvjiarui0826@gmail.com>

fix docstring fixes open-mmlab#709

* add TopDownH36MDataset * add support in inference * add configs

Support progressive test with fewer memory cost.

f3aaecc

clownrat6 requested a review from xvjiarui July 16, 2021 13:36

xvjiarui reviewed Jul 18, 2021

View reviewed changes

mmseg/apis/test.py Outdated Show resolved Hide resolved

mmseg/apis/test.py Outdated Show resolved Hide resolved

mmseg/apis/test.py Outdated Show resolved Hide resolved

mmseg/datasets/custom.py Outdated Show resolved Hide resolved

xvjiarui reviewed Jul 19, 2021

View reviewed changes

tools/test.py Outdated Show resolved Hide resolved

Temp code

7c944cf

clownrat6 force-pushed the progressive_test branch from 9dfe1c0 to 7c944cf Compare July 22, 2021 07:45

sennnnn added 2 commits July 22, 2021 16:41

Using processor to refactor evaluation workflow.

3a97df6

refactor eval hook.

e6be6b4

clownrat6 force-pushed the progressive_test branch from 7c944cf to e6be6b4 Compare July 22, 2021 12:38

sennnnn added 10 commits July 22, 2021 22:07

Fix multi gpu bug when index out of len(dataset).

fdb6b49

Fix process bar.

8653c96

Fix middle save argument.

842bdef

Modify some variable name of dataset evaluate api.

d389665

Modify some viriable name of eval hook.

9349970

Fix some priority bugs of eval hook.

3a3b3ec

Depreciated efficient_test.

e03ee57

Fix training progress blocked by eval hook.

c3c491d

Depreciated old test api.

0cb97f4

Fix test api error.

25943c8

xvjiarui reviewed Jul 26, 2021

View reviewed changes

mmseg/apis/test.py Outdated Show resolved Hide resolved

xvjiarui reviewed Jul 26, 2021

View reviewed changes

mmseg/apis/test.py Outdated Show resolved Hide resolved

xvjiarui reviewed Jul 26, 2021

View reviewed changes

mmseg/datasets/custom.py Outdated Show resolved Hide resolved

xvjiarui reviewed Jul 26, 2021

View reviewed changes

tools/deploy_test.py Outdated Show resolved Hide resolved

sennnnn added 6 commits July 27, 2021 12:00

Modify outer api.

be13436

Build a sampler test api.

6bb1c91

TODO: Refactor format_results.

25a7475

Modify variable names.

25cc8ba

Fix num_classes bug.

af54432

Fix sampler index bug.

1b7c976

sennnnn added 7 commits August 10, 2021 16:50

Fix custom dataset unit tests.

89e182c

Fix dataset unit tests.

1773f99

Fix eval hook unit tests.

b844145

Fix some imcompatible.

3f11345

Add pre_eval argument for eval hooks.

8662e69

Update eval hook doc string.

8dea091

Make pre_eval false in default.

fd28ab3

Add unit tests for dataset format_results.

ebea350

xvjiarui reviewed Aug 17, 2021

View reviewed changes

mmseg/apis/train.py Outdated Show resolved Hide resolved

xvjiarui reviewed Aug 17, 2021

View reviewed changes

mmseg/core/evaluation/eval_hooks.py Outdated Show resolved Hide resolved

xvjiarui reviewed Aug 17, 2021

View reviewed changes

mmseg/core/evaluation/eval_hooks.py Outdated Show resolved Hide resolved

xvjiarui reviewed Aug 17, 2021

View reviewed changes

tests/test_apis/test_single_gpu.py Outdated Show resolved Hide resolved

sennnnn added 3 commits August 19, 2021 10:10

Fix some comments and bc-breaking bug.

d0a3528

Merge Master.

5ddf909

Fix pre_eval set cfg field.

f474721

xvjiarui reviewed Aug 19, 2021

View reviewed changes

mmseg/apis/train.py Outdated Show resolved Hide resolved

Remove redundant codes.

469ced8

xvjiarui approved these changes Aug 20, 2021

View reviewed changes

xvjiarui merged commit e235c1a into open-mmlab:master Aug 20, 2021

Junjun2016 mentioned this pull request Aug 20, 2021

[Feature] Support eval concate dataset and add tool to show dataset #781

Closed

MengzhangLI mentioned this pull request Aug 26, 2021

Is there a problem with the mmsegmentation Miou calculation? #788

Closed

FreyWang mentioned this pull request Aug 27, 2021

[Feature] Support eval concate dataset and add tool to show dataset #833

Merged

zhouzaida mentioned this pull request Nov 26, 2021

Is there any trick to boost EvalHook? open-mmlab/mmcv#1348

Closed

aravind-h-v pushed a commit to aravind-h-v/mmsegmentation that referenced this pull request Mar 27, 2023

[Docs] fix docstring for issue open-mmlab#709 (open-mmlab#710)

f1b9ee7

fix docstring fixes open-mmlab#709

wjkim81 pushed a commit to wjkim81/mmsegmentation that referenced this pull request Dec 3, 2023

Add H36M dataset for 2D keypoint detection (open-mmlab#709)

1c1bcaf

* add TopDownH36MDataset * add support in inference * add configs

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Refactor] Support progressive test with fewer memory cost #709

[Refactor] Support progressive test with fewer memory cost #709

clownrat6 commented Jul 16, 2021 •

edited

Loading

codecov bot commented Jul 16, 2021 •

edited

Loading

DCNSW commented Aug 12, 2021

[Refactor] Support progressive test with fewer memory cost #709

[Refactor] Support progressive test with fewer memory cost #709

Conversation

clownrat6 commented Jul 16, 2021 • edited Loading

Support progressive test

Calculate Metrics

Format Results

Modifications Abstract

mmseg/apis/test.py

tools/test.py

mmseg/core/evaluation/eval_hooks.py

codecov bot commented Jul 16, 2021 • edited Loading

Codecov Report

DCNSW commented Aug 12, 2021

clownrat6 commented Jul 16, 2021 •

edited

Loading

codecov bot commented Jul 16, 2021 •

edited

Loading