[Enhance] Add extra dataloader settings in configs #1435

MeowZheng · 2022-03-30T12:54:21Z

Thanks for your contribution and we appreciate it a lot. The following instructions would make your pull request more healthy and more easily get feedback. If you do not understand some items, don't worry, just make the pull request and seek help from maintainers.

Motivation

ref: open-mmlab/mmpretrain#752
In some situation, we need to use different dataloader settings like samples_per_gpu in train/val/test dataloader.

Modification

Use train_dataloader, val_dataloader and test_dataloader settings in the data field to specify different arguments.

Use cases (Optional)

data = dict(
    samples_per_gpu=64,
    workers_per_gpu=4,
    train=dict(type='xxx', ...),
    val=dict(type='xxx', ...),
    test=dict(type='xxx', ...),
    # Use different batch size during inference.
    val_dataloader=dict(samples_per_gpu=8, workers_per_gpu=2),
    test_dataloader=dict(samples_per_gpu=8, workers_per_gpu=2),
)

Checklist

Pre-commit or other linting tools are used to fix the potential lint issues.
The modification is covered by complete unit tests. If not, please add more unit test to ensure the correctness.
If the modification has potential influence on downstream projects, this PR should be tested with downstream projects, like MMDet or MMDet3D.
The documentation has been modified accordingly, like docstring or example tutorials.

codecov · 2022-03-30T14:55:23Z

Codecov Report

Merging #1435 (cfdfc0e) into master (30864ea) will increase coverage by 0.09%.
The diff coverage is 91.45%.

@@            Coverage Diff             @@
##           master    #1435      +/-   ##
==========================================
+ Coverage   90.22%   90.31%   +0.09%     
==========================================
  Files         136      139       +3     
  Lines        7946     8289     +343     
  Branches     1323     1392      +69     
==========================================
+ Hits         7169     7486     +317     
- Misses        552      567      +15     
- Partials      225      236      +11

Flag	Coverage Δ
unittests	`90.31% <91.45%> (+0.09%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
mmseg/apis/train.py	`20.25% <0.00%> (-1.09%)`	⬇️
mmseg/core/layer_decay_optimizer_constructor.py	`88.37% <88.37%> (ø)`
mmseg/models/backbones/beit.py	`92.33% <92.33%> (ø)`
mmseg/core/__init__.py	`100.00% <100.00%> (ø)`
mmseg/models/backbones/__init__.py	`100.00% <100.00%> (ø)`
mmseg/models/losses/cross_entropy_loss.py	`82.71% <100.00%> (+3.30%)`	⬆️
mmseg/models/losses/utils.py	`87.50% <100.00%> (+5.92%)`	⬆️
mmseg/models/necks/__init__.py	`100.00% <100.00%> (ø)`
mmseg/models/necks/featurepyramid.py	`100.00% <100.00%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a82ebad...cfdfc0e. Read the comment docs.

mmseg/apis/train.py

tools/test.py

mmseg/apis/train.py

tools/test.py

Junjun2016

LGTM

* [Enhance] Add extra dataloader settings in configs * val default samples * val default samples * del unuse * del unused

* Ensure Flax pipeline always returns numpy array. * Clarify documentation.

[Enhance] Add extra dataloader settings in configs

baa1897