Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] Support dataset initialization with file_client #1402

Merged
merged 3 commits into from
Mar 28, 2022

Conversation

MeowZheng
Copy link
Collaborator

@MeowZheng MeowZheng commented Mar 20, 2022

Thanks for your contribution and we appreciate it a lot. The following instructions would make your pull request more healthy and more easily get feedback. If you do not understand some items, don't worry, just make the pull request and seek help from maintainers.

Motivation

Support to initialize dataset from file_client

Modification

  1. Add file_client_args in __init__ with default value dict(backend='disk')
  2. scan directory with file_client.list_dir_or_file in load_annotations
  3. Read split file with mmcv.list_from_file in load_annotations

Use cases (Optional)

file_client_args = dict(
    backend='petrel',
    path_mapping=dict({
        '.data/cityscapes/': 'openmmlab:s3://openmmlab/datasets/segmentation/cityscapes/',
        'data/cityscapes/': 'openmmlab:s3://openmmlab/datasets/segmentation/cityscapes/'
    }))

data = dict(
...
    train=dict(
    ...
           file_client_args=file_client_args),
    val=dict(
    ...
        gt_seg_map_loader_cfg=dict(file_client_args=file_client_args),
        file_client_args=file_client_args),
    test=dict(
    ...
        gt_seg_map_loader_cfg=dict(file_client_args=file_client_args),
        file_client_args=file_client_args))

Checklist

  1. Pre-commit or other linting tools are used to fix the potential lint issues.
  2. The modification is covered by complete unit tests. If not, please add more unit test to ensure the correctness.
  3. If the modification has potential influence on downstream projects, this PR should be tested with downstream projects, like MMDet or MMDet3D.
  4. The documentation has been modified accordingly, like docstring or example tutorials.

@codecov
Copy link

codecov bot commented Mar 20, 2022

Codecov Report

Merging #1402 (919d3e7) into master (3d0c2eb) will decrease coverage by 0.00%.
The diff coverage is 70.58%.

@@            Coverage Diff             @@
##           master    #1402      +/-   ##
==========================================
- Coverage   90.39%   90.39%   -0.01%     
==========================================
  Files         133      133              
  Lines        7906     7903       -3     
  Branches     1318     1318              
==========================================
- Hits         7147     7144       -3     
  Misses        536      536              
  Partials      223      223              
Flag Coverage Δ
unittests 90.39% <70.58%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
mmseg/datasets/chase_db1.py 77.77% <0.00%> (-2.23%) ⬇️
mmseg/datasets/drive.py 77.77% <0.00%> (-2.23%) ⬇️
mmseg/datasets/hrf.py 77.77% <0.00%> (-2.23%) ⬇️
mmseg/datasets/pascal_context.py 75.00% <0.00%> (-1.48%) ⬇️
mmseg/datasets/custom.py 94.73% <100.00%> (+0.05%) ⬆️
mmseg/datasets/isaid.py 93.93% <100.00%> (-0.18%) ⬇️
mmseg/core/evaluation/class_names.py 88.33% <0.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3d0c2eb...919d3e7. Read the comment docs.

@RockeyCoss
Copy link
Contributor

LGTM

Copy link
Collaborator

@Junjun2016 Junjun2016 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.
This month, we can do regression benchmarks with ceph to verify this PR.

@MeowZheng MeowZheng merged commit 30864ea into open-mmlab:master Mar 28, 2022
@MeowZheng MeowZheng deleted the file-client-dataset branch March 28, 2022 15:53
mob5566 pushed a commit to mob5566/mmsegmentation that referenced this pull request Apr 13, 2022
…#1402)

* [Feature] Support file_client in dataset initialization

* docstring

* revise osp.exist
ZhimingNJ pushed a commit to AetrexTechnology/mmsegmentation that referenced this pull request Jun 29, 2022
…#1402)

* [Feature] Support file_client in dataset initialization

* docstring

* revise osp.exist
aravind-h-v pushed a commit to aravind-h-v/mmsegmentation that referenced this pull request Mar 27, 2023
* Upscaling fixed

* up

* more fixes

* fix

* more fixes

* finish again

* up
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants