Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

avoid unnecessary workers with sequential CombinedLoader #17639

Merged
merged 12 commits into from
May 30, 2023
Merged

avoid unnecessary workers with sequential CombinedLoader #17639

merged 12 commits into from
May 30, 2023

Conversation

mukhery
Copy link
Contributor

@mukhery mukhery commented May 16, 2023 •
edited
Loading

What does this PR do?

This PR relates to behavior exhibited when training with multiple validation dataloaders. In this scenario, Lightning wraps the multiple dataloaders with a CombinedLoader. This CombinedLoader currently initializes all of the dataloaders causing all of their workers to start up and prefetch data potentially introducing substantial overhead.

Fixes #17641

Before submitting
  • Was this discussed/agreed via a GitHub issue? (not for typos and docs)
  • Did you read the contributor guideline, Pull Request section?
  • Did you make sure your PR does only one thing, instead of bundling different changes together?
  • Did you make sure to update the documentation with your changes? (if necessary)
  • Did you write any new necessary tests? (not for typos and docs)
  • Did you verify new and existing tests pass locally with your changes?
  • Did you list all the breaking changes introduced by this pull request?
  • Did you update the CHANGELOG? (not for typos, docs, test updates, or minor internal changes/refactors)

PR review

Anyone in the community is welcome to review the PR.
Before you start reviewing, make sure you have read the review guidelines. In short, see the following bullet-list:

Reviewer checklist
  • Is this pull request ready for review? (if not, please submit in draft mode)
  • Check that all items from Before submitting are resolved
  • Make sure the title is self-explanatory and the description concisely explains the PR
  • Add labels and milestones (and optionally projects) to the PR so it can be classified

@github-actions github-actions bot added the pl Generic label for PyTorch Lightning package label May 16, 2023
@mukhery mukhery marked this pull request as ready for review May 29, 2023 13:57
@carmocca carmocca added bug Something isn't working data handling Generic data-related topic labels May 29, 2023
@carmocca carmocca added this to the 2.0.x milestone May 29, 2023
@carmocca carmocca added the community This PR is from the community label May 29, 2023
carmocca
carmocca reviewed May 29, 2023
View reviewed changes
Copy link
Contributor

@carmocca carmocca left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch! Can you add a test in tests/tests_pytorch/utilities/test_combined_loader.py. Also a "Fixed" entry in src/lightning/pytorch/CHANGELOG.md

src/lightning/pytorch/utilities/combined_loader.py Outdated Show resolved Hide resolved
@mukhery mukhery changed the title avoid unnecessary workers with sequential CombinedLoader WIP: avoid unnecessary workers with sequential CombinedLoader May 29, 2023
@mukhery mukhery marked this pull request as draft May 29, 2023 19:01
@mukhery mukhery changed the title WIP: avoid unnecessary workers with sequential CombinedLoader avoid unnecessary workers with sequential CombinedLoader May 29, 2023
@mukhery mukhery marked this pull request as ready for review May 29, 2023 19:35
@mukhery
Copy link
Contributor Author

mukhery commented May 29, 2023

Good catch! Can you add a test in tests/tests_pytorch/utilities/test_combined_loader.py. Also a "Fixed" entry in src/lightning/pytorch/CHANGELOG.md

Alright, I think I've addressed these.

@Borda Borda changed the title avoid unnecessary workers with sequential CombinedLoader avoid unnecessary workers with sequential CombinedLoader May 29, 2023
carmocca
carmocca reviewed May 30, 2023
View reviewed changes
tests/tests_pytorch/loops/test_loops.py Show resolved Hide resolved
tests/tests_pytorch/loops/test_loops.py Outdated Show resolved Hide resolved
tests/tests_pytorch/loops/test_loops.py Outdated Show resolved Hide resolved
tests/tests_pytorch/utilities/test_combined_loader.py Outdated Show resolved Hide resolved
carmocca
carmocca approved these changes May 30, 2023
View reviewed changes
Copy link
Contributor

@carmocca carmocca left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for fixing this!

@mergify mergify bot added the ready PRs ready to be merged label May 30, 2023
@awaelchli awaelchli enabled auto-merge (squash) May 30, 2023 03:41
@awaelchli awaelchli merged commit c3ad756 into Lightning-AI:master May 30, 2023
@mukhery mukhery deleted the bugfix/combined_worker_overload branch May 30, 2023 12:08
Borda pushed a commit that referenced this pull request Jun 2, 2023
@mukhery @Borda
avoid unnecessary workers with sequential CombinedLoader (#17639)
0e42998 
Co-authored-by: Carlos Mocholí <carlossmocholi@gmail.com>
Co-authored-by: Adrian Wälchli <aedu.waelchli@gmail.com>

(cherry picked from commit c3ad756)
lantiga pushed a commit that referenced this pull request Jun 2, 2023
@mukhery @lantiga
avoid unnecessary workers with sequential CombinedLoader (#17639)
64d84cc 
Co-authored-by: Carlos Mocholí <carlossmocholi@gmail.com>
Co-authored-by: Adrian Wälchli <aedu.waelchli@gmail.com>

(cherry picked from commit c3ad756)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@awaelchli awaelchli awaelchli approved these changes

@Borda Borda Borda approved these changes

@carmocca carmocca carmocca approved these changes

@williamFalcon williamFalcon Awaiting requested review from williamFalcon

@justusschock justusschock Awaiting requested review from justusschock justusschock is a code owner

Assignees
No one assigned
Labels
bug Something isn't working community This PR is from the community data handling Generic data-related topic pl Generic label for PyTorch Lightning package ready PRs ready to be merged
Projects
None yet
Milestone
2.0.x
Development

Successfully merging this pull request may close these issues.

potentially substantial overhead starting all dataloaders in CombinedLoader
4 participants
@mukhery @awaelchli @Borda @carmocca