Fix min-epochs and early-stopping triggering too many validation runs #16719

awaelchli · 2023-02-10T15:00:58Z

What does this PR do?

There is an unfortunate interaction between the early stopping trigger mechanism and min_epochs not being reached, that then leads to the validation being triggered on every subsequent training step due to this condition here:

https://github.com/Lightning-AI/lightning/blob/5196eaa5264c7b95316718aa2d173dd42c5d9936/src/lightning/pytorch/loops/epoch/training_epoch_loop.py#L392-L394

This then manifests in a big runtime increase for subsequent epochs.

Before submitting

Was this discussed/approved via a GitHub issue? (not for typos and docs)
Did you read the contributor guideline, Pull Request section?
Did you make sure your PR does only one thing, instead of bundling different changes together?
Did you make sure to update the documentation with your changes? (if necessary)
Did you write any new necessary tests? (not for typos and docs)
Did you verify new and existing tests pass locally with your changes?
Did you update the CHANGELOG? (not for typos, docs, test updates, or internal minor changes/refactorings)

PR review

Anyone in the community is free to review the PR once the tests have passed.
Before you start reviewing make sure you have read Review guidelines. In short, see the following bullet-list:

Is this pull request ready for review? (if not, please submit in draft mode)
Check that all items from Before submitting are resolved
Make sure the title is self-explanatory and the description concisely explains the PR
Add labels and milestones (and optionally projects) to the PR so it can be classified

Did you have fun?

I made sure I had fun coding 🙃

cc @Borda @carmocca @awaelchli @justusschock

for more information, see https://pre-commit.ci

…o bugfix/early-stop-min-steps

github-actions · 2023-02-11T01:49:16Z

⚡ Required checks status: All passing 🟢

Groups summary

🟢 pytorch_lightning: Tests workflow

Check ID	Status
pl-cpu (macOS-11, lightning, 3.8, 1.11)	success	✅
pl-cpu (macOS-11, lightning, 3.9, 1.12)	success	✅
pl-cpu (macOS-11, lightning, 3.10, 1.13)	success	✅
pl-cpu (macOS-11, lightning, 3.8, 1.11, oldest)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.9, 1.11)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.10, 1.12)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.10, 1.13)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.8, 1.11, oldest)	success	✅
pl-cpu (windows-2022, lightning, 3.9, 1.11)	success	✅
pl-cpu (windows-2022, lightning, 3.10, 1.12)	success	✅
pl-cpu (windows-2022, lightning, 3.10, 1.13)	success	✅
pl-cpu (windows-2022, lightning, 3.8, 1.11, oldest)	success	✅
pl-cpu (macOS-11, pytorch, 3.8, 1.13)	success	✅
pl-cpu (ubuntu-20.04, pytorch, 3.8, 1.13)	success	✅
pl-cpu (windows-2022, pytorch, 3.8, 1.13)	success	✅

These checks are required after the changes to src/lightning/pytorch/loops/epoch/training_epoch_loop.py, src/lightning/pytorch/loops/fit_loop.py, tests/tests_pytorch/loops/epoch/test_training_epoch_loop.py, tests/tests_pytorch/loops/test_training_loop.py.

🟢 pytorch_lightning: Azure GPU

Check ID	Status
pytorch-lightning (GPUs)	success	✅

These checks are required after the changes to src/lightning/pytorch/loops/epoch/training_epoch_loop.py, src/lightning/pytorch/loops/fit_loop.py, tests/tests_pytorch/loops/epoch/test_training_epoch_loop.py, tests/tests_pytorch/loops/test_training_loop.py.

🟢 pytorch_lightning: Azure HPU

Check ID	Status
pytorch-lightning (HPUs)	success	✅

These checks are required after the changes to src/lightning/pytorch/loops/epoch/training_epoch_loop.py, src/lightning/pytorch/loops/fit_loop.py, tests/tests_pytorch/loops/epoch/test_training_epoch_loop.py, tests/tests_pytorch/loops/test_training_loop.py.

🟢 pytorch_lightning: Azure IPU

Check ID	Status
pytorch-lightning (IPUs)	success	✅

These checks are required after the changes to src/lightning/pytorch/loops/epoch/training_epoch_loop.py, src/lightning/pytorch/loops/fit_loop.py, tests/tests_pytorch/loops/epoch/test_training_epoch_loop.py, tests/tests_pytorch/loops/test_training_loop.py.

🟢 pytorch_lightning: Docs

Check ID	Status
make-doctest (pytorch)	success	✅
make-html (pytorch)	success	✅

These checks are required after the changes to src/lightning/pytorch/loops/epoch/training_epoch_loop.py, src/lightning/pytorch/loops/fit_loop.py.

🟢 mypy

Check ID	Status
mypy	success	✅

These checks are required after the changes to src/lightning/pytorch/loops/epoch/training_epoch_loop.py, src/lightning/pytorch/loops/fit_loop.py.

🟢 install

Check ID	Status
install-pkg (ubuntu-22.04, app, 3.8)	success	✅
install-pkg (ubuntu-22.04, app, 3.10)	success	✅
install-pkg (ubuntu-22.04, fabric, 3.8)	success	✅
install-pkg (ubuntu-22.04, fabric, 3.10)	success	✅
install-pkg (ubuntu-22.04, pytorch, 3.8)	success	✅
install-pkg (ubuntu-22.04, pytorch, 3.10)	success	✅
install-pkg (ubuntu-22.04, lightning, 3.8)	success	✅
install-pkg (ubuntu-22.04, lightning, 3.10)	success	✅
install-pkg (ubuntu-22.04, notset, 3.8)	success	✅
install-pkg (ubuntu-22.04, notset, 3.10)	success	✅
install-pkg (macOS-12, app, 3.8)	success	✅
install-pkg (macOS-12, app, 3.10)	success	✅
install-pkg (macOS-12, fabric, 3.8)	success	✅
install-pkg (macOS-12, fabric, 3.10)	success	✅
install-pkg (macOS-12, pytorch, 3.8)	success	✅
install-pkg (macOS-12, pytorch, 3.10)	success	✅
install-pkg (macOS-12, lightning, 3.8)	success	✅
install-pkg (macOS-12, lightning, 3.10)	success	✅
install-pkg (macOS-12, notset, 3.8)	success	✅
install-pkg (macOS-12, notset, 3.10)	success	✅
install-pkg (windows-2022, app, 3.8)	success	✅
install-pkg (windows-2022, app, 3.10)	success	✅
install-pkg (windows-2022, fabric, 3.8)	success	✅
install-pkg (windows-2022, fabric, 3.10)	success	✅
install-pkg (windows-2022, pytorch, 3.8)	success	✅
install-pkg (windows-2022, pytorch, 3.10)	success	✅
install-pkg (windows-2022, lightning, 3.8)	success	✅
install-pkg (windows-2022, lightning, 3.10)	success	✅
install-pkg (windows-2022, notset, 3.8)	success	✅
install-pkg (windows-2022, notset, 3.10)	success	✅

These checks are required after the changes to src/lightning/pytorch/loops/epoch/training_epoch_loop.py, src/lightning/pytorch/loops/fit_loop.py.

🟢 link-check

Check ID	Status
markdown-link-check	success	✅

These checks are required after the changes to src/lightning/pytorch/CHANGELOG.md.

Thank you for your contribution! 💜

Note
This comment is automatically generated and updates for 60 minutes every 180 seconds. If you have any other questions, contact carmocca for help.

src/lightning/pytorch/loops/epoch/training_epoch_loop.py

…#16719) Co-authored-by: Jirka Borovec <6035284+Borda@users.noreply.github.com>

* Add .git-blame-ignore-revs (#16709) Co-authored-by: Jirka Borovec <6035284+Borda@users.noreply.github.com> * Fix strategy type validation in connectors (#16693) * Disable strict loading in multiprocessing launcher (#16365) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Jirka <jirka.borovec@seznam.cz> * Fix min-epochs and early-stopping triggering too many validation runs (#16719) Co-authored-by: Jirka Borovec <6035284+Borda@users.noreply.github.com> * Update hydra-core requirement from <1.3.0,>=1.0.5 to >=1.0.5,<1.4.0 in /requirements (#16736) Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * [App] Add support for private data (#16738) Co-authored-by: thomas <thomas@thomass-MacBook-Pro.local> * [App] Add rm one level below project level (#16740) Co-authored-by: Ethan Harris <ethanwharris@gmail.com> Co-authored-by: Justus Schock <12886177+justusschock@users.noreply.github.com> Co-authored-by: thomas <thomas@thomass-MacBook-Pro.local> * ci: cleaning caches (#16752) * CI: Update colossalai version (#16747) Co-authored-by: Carlos Mocholí <carlossmocholi@gmail.com> type * Update version and changelog for 1.9.2 --------- Co-authored-by: Akihiro Nitta <nitta@akihironitta.com> Co-authored-by: Jirka Borovec <6035284+Borda@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Jirka <jirka.borovec@seznam.cz> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: thomas chaton <thomas@grid.ai> Co-authored-by: thomas <thomas@thomass-MacBook-Pro.local> Co-authored-by: Ethan Harris <ethanwharris@gmail.com> Co-authored-by: Justus Schock <12886177+justusschock@users.noreply.github.com>

see: Lightning-AI/pytorch-lightning#16719

bugfix

6441474

awaelchli added bug Something isn't working loops Related to the Loop API labels Feb 10, 2023

github-actions bot added the pl Generic label for PyTorch Lightning package label Feb 10, 2023

awaelchli added the callback: early stopping label Feb 10, 2023

awaelchli added this to the v1.9.x milestone Feb 10, 2023

awaelchli and others added 5 commits February 11, 2023 02:35

add a test

b310b56

update test

658d142

[pre-commit.ci] auto fixes from pre-commit.com hooks

abbbeaf

for more information, see https://pre-commit.ci

changelog

feb6e3a

Merge remote-tracking branch 'origin/bugfix/early-stop-min-steps' int…

f5a2b60

…o bugfix/early-stop-min-steps

awaelchli marked this pull request as ready for review February 11, 2023 01:48

awaelchli requested review from carmocca, justusschock and williamFalcon as code owners February 11, 2023 01:48

clarify comment

981d619

awaelchli changed the title ~~WIP: Fix min-epochs and early-stopping triggering too many validation runs~~ Fix min-epochs and early-stopping triggering too many validation runs Feb 11, 2023

carmocca approved these changes Feb 11, 2023

View reviewed changes

src/lightning/pytorch/loops/epoch/training_epoch_loop.py Outdated Show resolved Hide resolved

Borda approved these changes Feb 11, 2023

View reviewed changes

Merge branch 'master' into bugfix/early-stop-min-steps

0d43ae2

mergify bot added the ready PRs ready to be merged label Feb 11, 2023

Update CHANGELOG.md

0625cf7

Borda approved these changes Feb 11, 2023

View reviewed changes

rename

ee51293

awaelchli enabled auto-merge (squash) February 11, 2023 02:38

awaelchli merged commit 5340d96 into master Feb 11, 2023

awaelchli deleted the bugfix/early-stop-min-steps branch February 11, 2023 03:02

awaelchli added a commit that referenced this pull request Feb 11, 2023

Fix min-epochs and early-stopping triggering too many validation runs (…

6db6431

…#16719) Co-authored-by: Jirka Borovec <6035284+Borda@users.noreply.github.com>

This was referenced Jul 23, 2023

Trainer stuck in infinite validation loop #17813

Closed

Add warning to 1.8.x users about Early Stopping and min_epochs interaction #18139

Closed

thesofakillers added a commit to thesofakillers/nlgoals that referenced this pull request Jul 23, 2023

PL>=1.9.2 -> handle early stopping with min_epochs

0aacb67

see: Lightning-AI/pytorch-lightning#16719

awaelchli mentioned this pull request Aug 16, 2023

Bug combining min_epochs and EarlyStopping #18308

Closed

awaelchli mentioned this pull request Dec 29, 2023

Early Stopping that triggers before min_epochs is reached increasingly slows training down #18063

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix min-epochs and early-stopping triggering too many validation runs #16719

Fix min-epochs and early-stopping triggering too many validation runs #16719

awaelchli commented Feb 10, 2023 •

edited by github-actions bot

Loading

github-actions bot commented Feb 11, 2023 •

edited

Loading

Fix min-epochs and early-stopping triggering too many validation runs #16719

Fix min-epochs and early-stopping triggering too many validation runs #16719

Conversation

awaelchli commented Feb 10, 2023 • edited by github-actions bot Loading

What does this PR do?

Before submitting

PR review

Did you have fun?

github-actions bot commented Feb 11, 2023 • edited Loading

⚡ Required checks status: All passing 🟢

Groups summary

awaelchli commented Feb 10, 2023 •

edited by github-actions bot

Loading

github-actions bot commented Feb 11, 2023 •

edited

Loading