Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

added if statement to account for IterableDatasets doing distributed … #2151

Merged
merged 1 commit into from
Oct 8, 2022

Conversation

ShirleyWangCVR
Copy link

@ShirleyWangCVR ShirleyWangCVR commented Oct 8, 2022

Motivation

Distributed training currently doesn't work if the dataset is an IterableDataset, due to always specifying the DistributedSampler if the distributed flag is on. IterableDatasets require no Sampler when creating a dataloader over them.

Modification

I added an if statement to the build_dataloader function to check for the case where the dataset is an IterableDataset, and if so, to use no sampler.

BC-breaking (Optional)

I don't believe it should, unless people were doing distributed training with IterableDatasets beforehand and somehow made the DistributedSampler work for them when pytorch itself doesn't support it.

Use cases (Optional)

Specific case of distributed training with IterableDataset

Checklist

  1. Pre-commit or other linting tools are used to fix the potential lint issues.
  2. The modification is covered by complete unit tests. If not, please add more unit test to ensure the correctness.
  3. If the modification has potential influence on downstream projects, this PR should be tested with downstream projects, like MMDet or MMDet3D.
  4. The documentation has been modified accordingly, like docstring or example tutorials.

@CLAassistant
Copy link

CLAassistant commented Oct 8, 2022

CLA assistant check
All committers have signed the CLA.

Copy link
Collaborator

@MeowZheng MeowZheng left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Many thanks for your kind reminder and contribution, we are working on testing this modification.

@codecov
Copy link

codecov bot commented Oct 8, 2022

Codecov Report

Base: 89.14% // Head: 89.09% // Decreases project coverage by -0.05% ⚠️

Coverage data is based on head (997e2ec) compared to base (6c746fa).
Patch coverage: 28.57% of modified lines in pull request are covered.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #2151      +/-   ##
==========================================
- Coverage   89.14%   89.09%   -0.06%     
==========================================
  Files         145      145              
  Lines        8716     8721       +5     
  Branches     1471     1472       +1     
==========================================
  Hits         7770     7770              
- Misses        704      708       +4     
- Partials      242      243       +1     
Flag Coverage Δ
unittests 89.09% <28.57%> (-0.06%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
mmseg/datasets/builder.py 82.02% <28.57%> (-4.89%) ⬇️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

@MeowZheng MeowZheng merged commit 9d2312b into open-mmlab:master Oct 8, 2022
aravind-h-v pushed a commit to aravind-h-v/mmsegmentation that referenced this pull request Mar 27, 2023
@jason102811
Copy link

shiirleywangcvr,您好!您在MMSeg项目中给我们提的PR非常重要,感谢您付出私人时间帮助改进开源项目,相信很多开发者会从你的PR中受益。
我们非常期待与您继续合作,OpenMMLab专门成立了贡献者组织MMSIG,为贡献者们提供开源证书、荣誉体系和专享好礼,可通过添加微信:openmmlabwx 联系我们(请备注mmsig+GitHub id),由衷希望您能加入!
Dear shiirleywangcvr,
First of all, we want to express our gratitude for your significant PR in the MMSeg project. Your contribution is highly appreciated, and we are grateful for your efforts in helping improve this open-source project during your personal time. We believe that many developers will benefit from your PR.
We are looking forward to continuing our collaboration with you. OpenMMLab has established a special contributors' organization called MMSIG, which provides contributors with open-source certificates, a recognition system, and exclusive rewards. You can contact us by adding our WeChat(if you have WeChat): openmmlabwx, or join in our discord: https://discord.gg/qH9fysxPDW. We sincerely hope you will join us!
Best regards! @ShirleyWangCVR

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants