loop customization docs #9557

awaelchli · 2021-09-16T10:45:35Z

What does this PR do?

Adds a user guide for loops in Lightning. Contains:

Explanation of default loop structure in Lightning
How loop customization works
An example of a "yield loop" that transforms the training step into a generator

Also adds the loop base class to the API reference section.

You can check the docs build by clicking here:

Then go to Artifacts and navigate to loops.html.

Or if you have access to circle-ci: https://129946-178626720-gh.circle-artifacts.com/0/html/advanced/loops.html

Part of #7938

Before submitting

Was this discussed/approved via a GitHub issue? (not for typos and docs)
Did you read the contributor guideline, Pull Request section?
Did you make sure your PR does only one thing, instead of bundling different changes together?
Did you make sure to update the documentation with your changes? (if necessary)
Did you write any new necessary tests? (not for typos and docs)
Did you verify new and existing tests pass locally with your changes?
Did you update the CHANGELOG? (not for typos, docs, test updates, or internal minor changes/refactorings)

PR review

Anyone in the community is free to review the PR once the tests have passed.
Before you start reviewing make sure you have read Review guidelines. In short, see the following bullet-list:

Is this pull request ready for review? (if not, please submit in draft mode)
Check that all items from Before submitting are resolved
Make sure the title is self-explanatory and the description concisely explains the PR
Add labels and milestones (and optionally projects) to the PR so it can be classified

Did you have fun?

I made sure I had fun coding 🙃

docs/source/advanced/loop_customization.rst

awaelchli · 2021-09-16T12:33:37Z

docs/source/advanced/loops.rst

+---------------------------------------------------------------+-----------------------------------------------------------------------+-------------------------------------------------------------------------------+
+| Entry point                                                   | Trainer attribute                                                     | Loop class                                                                    |


this is supposed to be a table, sorry this may look messed up in your browser when viewing the diff

Neat ! Good idea to show entry points !

this may look messed up in your browser

For better text readability, we could use .. list-table:: as we do in Bolts maybe?

source: https://github.com/PyTorchLightning/lightning-bolts/blame/master/docs/source/deprecated/models/self_supervised.rst#L148
built page: https://lightning-bolts.readthedocs.io/en/latest/deprecated/models/self_supervised.html#cifar-10-and-stl-10-baselines

looks better on the source code. let's do it

github-actions · 2021-09-16T13:45:40Z

Build Error! No Linked Issue found. Please link an issue or mention it in the body using #<issue_id>

awaelchli · 2021-09-16T13:46:39Z

hey @tchaton can we disable the bot for documentation updates?

codecov · 2021-09-16T14:06:47Z

Codecov Report

Merging #9557 (4add03c) into master (23450e2) will decrease coverage by 0%.
The diff coverage is n/a.

@@          Coverage Diff           @@
##           master   #9557   +/-   ##
======================================
- Coverage      93%     93%   -0%     
======================================
  Files         180     180           
  Lines       15090   15116   +26     
======================================
+ Hits        14019   14043   +24     
- Misses       1071    1073    +2

for more information, see https://pre-commit.ci

docs/source/advanced/loops.rst

justusschock · 2021-09-16T15:21:32Z

Can we also add this to the Optional Extension section. I searched for it there and common usecases is IMO not the best fit.

Also in general: This is part of the public facing lightning API. I find it strange, that the API section only covers trainer and LightningModule.

tchaton · 2021-09-17T08:01:32Z

hey @tchaton can we disable the bot for documentation updates?

Yes, I need to implement this. I will try this weekend.

Best,
T.C

Co-authored-by: Carlos Mocholí <carlossmocholi@gmail.com>

tchaton

Awesome work !

docs/source/advanced/loops.rst

tchaton · 2021-09-17T08:05:43Z

docs/source/advanced/loops.rst

+- how you can create a custom loop for a new training step flavor,
+- and how you can connect the custom loop and run it.
+
+Most importantly, we will also provide guidelines when to use loop customization and when NOT to use it.


Suggested change

Most importantly, we will also provide guidelines when to use loop customization and when NOT to use it.

Most importantly, we will provide guidelines when to use loop customization and when NOT to use it.

tchaton · 2021-09-17T08:06:09Z

docs/source/advanced/loops.rst

+---------------------------------------------------------------+-----------------------------------------------------------------------+-------------------------------------------------------------------------------+
+| Entry point                                                   | Trainer attribute                                                     | Loop class                                                                    |


Neat ! Good idea to show entry points !

tchaton · 2021-09-17T08:06:46Z

docs/source/advanced/loops.rst

+| :meth:`~pytorch_lightning.trainer.trainer.Trainer.predict`    | :attr:`~pytorch_lightning.trainer.trainer.Trainer.predict_loop`       | :class:`~pytorch_lightning.loops.dataloader.prediction_loop.PredictionLoop`   |
+---------------------------------------------------------------+-----------------------------------------------------------------------+-------------------------------------------------------------------------------+
+
+When the user calls :code:`Trainer.method`, it redirects to the corresponding :code:`Trainer.loop.run()` which implements the main logic of that particular Lightning loop.


Suggested change

When the user calls :code:`Trainer.method`, it redirects to the corresponding :code:`Trainer.loop.run()` which implements the main logic of that particular Lightning loop.

When the user calls :code:`Trainer.method`, it redirects to the corresponding :code:`Trainer.loop.run()` which implements the associated logic of this particular Lightning Loop.

docs/source/advanced/loops.rst

tchaton · 2021-09-17T09:13:59Z

docs/source/advanced/loops.rst

+    Set the environment variable `PL_FAULT_TOLERANT_TRAINING = 1` to enable saving the progress of loops.
+    Read more about :doc:`fault-tolerant training training <../advanced/fault_tolerant_training>`.
+
+An interesting property of the abstract loop interface is that it can maintain a state.


Suggested change

An interesting property of the abstract loop interface is that it can maintain a state.

The PyTorch Lightning Loop state management design is highly inspired from the PyTorch nn.Module and relies on the same underlying logic.

actually all I want to express here is that the benefit of having loops as objects is that they can have state.

I know, but I built the state management exactly as pytorch ones. Advanced users are already familiar with it, and it means they will understand faster how it works. Just a personal suggestion.

tchaton · 2021-09-17T09:16:29Z

docs/source/advanced/loops.rst

+    Read more about :doc:`fault-tolerant training training <../advanced/fault_tolerant_training>`.
+
+An interesting property of the abstract loop interface is that it can maintain a state.
+It can save its state to a checkpoint through corresponding hooks and if implemented accordingly, resume it's state of exectuion at the appropriate place.


Suggested change

It can save its state to a checkpoint through corresponding hooks and if implemented accordingly, resume it's state of exectuion at the appropriate place.

As a result, the loops can save their state and their children within the model checkpoint through corresponding hooks and if implemented accordingly, resume their state of execution at the appropriate place.

tchaton · 2021-09-17T09:18:19Z

docs/source/advanced/loops.rst

+
+An interesting property of the abstract loop interface is that it can maintain a state.
+It can save its state to a checkpoint through corresponding hooks and if implemented accordingly, resume it's state of exectuion at the appropriate place.
+This design is particularly interesting for fault-tolerant training which is an experimental feature released in Lightning v1.5.


Suggested change

This design is particularly interesting for fault-tolerant training which is an experimental feature released in Lightning v1.5.

As a matter of facts, all the Lightning loops implements an advanced state management which is the underlying mechanism of a long waited feature: Fault Tolerant Training which is an experimental feature released in Lightning v1.5.

docs/source/advanced/loops.rst

tchaton · 2021-09-17T09:21:14Z

docs/source/advanced/loops.rst

+
+**How do I make sure a given LightningModule is compatible with my custom loop?**
+
+To restrict the compatibility of a LightningModule to a particular loop type, we recommend to define a specific class mixin for this purpose.


Dope ! Really neat :)

awaelchli · 2021-09-17T09:27:28Z

@justusschock sounds good, can move it to "Optional Extensions" section under plugins if others agree @carmocca @tchaton .

Also in general: This is part of the public facing lightning API. I find it strange, that the API section only covers trainer and LightningModule.

We have two sections, one is just called "Lightning API" and the other "API Reference" which is supposed to contain all public facing api. The difference is that the former is more a user guide and the latter is just the api reference nothing more. we could probably do a better naming there.

Co-authored-by: thomas chaton <thomas@grid.ai>

for more information, see https://pre-commit.ci

tchaton · 2021-09-18T11:18:31Z

docs/source/advanced/loops.rst

+    from pytorch_lightning.loops.utilities import _build_training_step_kwargs
+
+
+    class YieldLoop(OptimizerLoop):


@awaelchli, would mind making sure the code isn't too long. Maybe 80 lines. It doesn't render very well in the doc.

awaelchli · 2021-09-18T21:40:46Z

will rewrite the docs and send new pr.

awaelchli added 7 commits September 15, 2021 15:05

add introduction draft

f6fe9ee

trainer entry point section

3836641

add api reference

15d74ff

base class section draft

40bcfb6

faq drat and notes

762f4e9

default loop description

d439db7

loop customization section

97ad8d7

awaelchli added the docs Documentation related label Sep 16, 2021

awaelchli added this to the v1.5 milestone Sep 16, 2021

awaelchli commented Sep 16, 2021

View reviewed changes

docs/source/advanced/loop_customization.rst Outdated Show resolved Hide resolved

awaelchli added 5 commits September 16, 2021 12:47

remove notes

e2626ae

rename file

fd30f8b

add loop page to index

4cf9d04

fix typo

fad1106

persisting state

2e887cb

awaelchli commented Sep 16, 2021

View reviewed changes

awaelchli added 5 commits September 16, 2021 14:37

fix typo in code-block

ff784a7

add FAQ

4710ce9

fix underline

16f46fc

fix todo's for links

6ff22c5

update Q/A format

c140209

awaelchli marked this pull request as ready for review September 16, 2021 13:44

awaelchli requested review from Borda, carmocca, edenlightning, justusschock and tchaton as code owners September 16, 2021 13:44

awaelchli and others added 3 commits September 16, 2021 16:25

blacken docs

d56f20e

make comments fit on the page

c0edf98

[pre-commit.ci] auto fixes from pre-commit.com hooks

79967b3

for more information, see https://pre-commit.ci

carmocca reviewed Sep 16, 2021

View reviewed changes

backticks and typos

773f1b6

Co-authored-by: Carlos Mocholí <carlossmocholi@gmail.com>

tchaton reviewed Sep 17, 2021

View reviewed changes

awaelchli added 2 commits September 17, 2021 11:19

add missing backticks

6919352

remove "-"

f6c83af

tchaton reviewed Sep 17, 2021

View reviewed changes

awaelchli and others added 2 commits September 17, 2021 11:52

several rewordings and typos

9c9bd78

Co-authored-by: thomas chaton <thomas@grid.ai>

[pre-commit.ci] auto fixes from pre-commit.com hooks

4add03c

for more information, see https://pre-commit.ci

tchaton reviewed Sep 18, 2021

View reviewed changes

awaelchli closed this Sep 18, 2021

awaelchli mentioned this pull request Sep 20, 2021

loop customization docs #9609

Merged

12 tasks

carmocca deleted the docs/loop-customization branch September 21, 2021 22:00

awaelchli mentioned this pull request Oct 3, 2021

Allow users to specify custom fit_loop in Trainer #9792

Closed

12 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

loop customization docs #9557

loop customization docs #9557

awaelchli commented Sep 16, 2021 •

edited by carmocca

Loading

awaelchli Sep 16, 2021

tchaton Sep 17, 2021

akihironitta Sep 18, 2021

awaelchli Sep 18, 2021

github-actions bot commented Sep 16, 2021

awaelchli commented Sep 16, 2021

codecov bot commented Sep 16, 2021 •

edited

Loading

justusschock commented Sep 16, 2021

tchaton commented Sep 17, 2021

tchaton left a comment •

edited

Loading

tchaton Sep 17, 2021

tchaton Sep 17, 2021

tchaton Sep 17, 2021

tchaton Sep 17, 2021

awaelchli Sep 17, 2021

tchaton Sep 17, 2021

tchaton Sep 17, 2021

tchaton Sep 17, 2021

tchaton Sep 17, 2021

awaelchli commented Sep 17, 2021

tchaton Sep 18, 2021

awaelchli commented Sep 18, 2021

		+---------------------------------------------------------------+-----------------------------------------------------------------------+-------------------------------------------------------------------------------+
		\| Entry point \| Trainer attribute \| Loop class \|

	Most importantly, we will also provide guidelines when to use loop customization and when NOT to use it.
	Most importantly, we will provide guidelines when to use loop customization and when NOT to use it.

	When the user calls :code:`Trainer.method`, it redirects to the corresponding :code:`Trainer.loop.run()` which implements the main logic of that particular Lightning loop.
	When the user calls :code:`Trainer.method`, it redirects to the corresponding :code:`Trainer.loop.run()` which implements the associated logic of this particular Lightning Loop.

	An interesting property of the abstract loop interface is that it can maintain a state.
	The PyTorch Lightning Loop state management design is highly inspired from the PyTorch nn.Module and relies on the same underlying logic.

	It can save its state to a checkpoint through corresponding hooks and if implemented accordingly, resume it's state of exectuion at the appropriate place.
	As a result, the loops can save their state and their children within the model checkpoint through corresponding hooks and if implemented accordingly, resume their state of execution at the appropriate place.

	This design is particularly interesting for fault-tolerant training which is an experimental feature released in Lightning v1.5.
	As a matter of facts, all the Lightning loops implements an advanced state management which is the underlying mechanism of a long waited feature: Fault Tolerant Training which is an experimental feature released in Lightning v1.5.


		How do I make sure a given LightningModule is compatible with my custom loop?

		To restrict the compatibility of a LightningModule to a particular loop type, we recommend to define a specific class mixin for this purpose.

		from pytorch_lightning.loops.utilities import _build_training_step_kwargs


		class YieldLoop(OptimizerLoop):

loop customization docs #9557

loop customization docs #9557

Conversation

awaelchli commented Sep 16, 2021 • edited by carmocca Loading

What does this PR do?

Before submitting

PR review

Did you have fun?

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Sep 16, 2021

awaelchli commented Sep 16, 2021

codecov bot commented Sep 16, 2021 • edited Loading

Codecov Report

justusschock commented Sep 16, 2021

tchaton commented Sep 17, 2021

tchaton left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

awaelchli commented Sep 17, 2021

Choose a reason for hiding this comment

awaelchli commented Sep 18, 2021

awaelchli commented Sep 16, 2021 •

edited by carmocca

Loading

codecov bot commented Sep 16, 2021 •

edited

Loading

tchaton left a comment •

edited

Loading