Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CLI] Shorthand notation to instantiate callbacks [3/3] #8815

Merged
merged 88 commits into from
Sep 17, 2021
Merged
Show file tree
Hide file tree
Changes from 34 commits
Commits
Show all changes
88 commits
Select commit Hold shift + click to select a range
d7f00be
add registries
tchaton Aug 9, 2021
a07d305
simplify LightningCLI with defaults
tchaton Aug 9, 2021
ce39c47
cleanup
tchaton Aug 9, 2021
3081475
update
tchaton Aug 9, 2021
51f82d5
updates
tchaton Aug 10, 2021
7197d6e
cleanup
tchaton Aug 10, 2021
9a6e81e
update on comments
tchaton Aug 10, 2021
41f5d78
update
tchaton Aug 10, 2021
06e4999
cleanup
tchaton Aug 10, 2021
e91ea47
update on comments
tchaton Aug 10, 2021
3f35ecd
Merge branch 'master' into lightning_cli_registries
tchaton Aug 10, 2021
705c0bd
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Aug 10, 2021
78a4398
add docs
tchaton Aug 10, 2021
e96dc28
doc updates
tchaton Aug 10, 2021
631aa72
update
tchaton Aug 10, 2021
2fc3c0a
update
tchaton Aug 10, 2021
43dd8b4
resolve comments
tchaton Aug 10, 2021
c6ae669
comment
tchaton Aug 10, 2021
5c21b1c
add comment
tchaton Aug 10, 2021
b370deb
typo
tchaton Aug 10, 2021
f8e7ca7
update on comments
tchaton Aug 11, 2021
e428d2f
resolve bug
tchaton Aug 11, 2021
3e97905
typo
tchaton Aug 11, 2021
3b1bdb6
update
tchaton Aug 11, 2021
0d1db29
resolve comments
tchaton Aug 11, 2021
3d35c82
add unittesting
tchaton Aug 11, 2021
4c0f960
resolve tests
tchaton Aug 11, 2021
d3a62ca
resolve comments
tchaton Aug 12, 2021
39781a1
update on comments
tchaton Aug 13, 2021
68c03de
doc updates
tchaton Aug 13, 2021
b01828b
update
tchaton Aug 13, 2021
d213c73
Merge branch 'master' into lightning_cli_registries
tchaton Aug 13, 2021
5935ec4
update on comments
tchaton Aug 17, 2021
b6616f0
Merge branch 'lightning_cli_registries' of https://github.com/PyTorch…
tchaton Aug 17, 2021
0d89423
Merge branch 'master' into lightning_cli_registries
carmocca Aug 19, 2021
37fd679
Fix mypy
carmocca Aug 19, 2021
f16db3d
Revert unrelated change which had broken mypy
carmocca Aug 19, 2021
572488c
Convert to staticmethod
carmocca Aug 19, 2021
2fc4608
Replace context managers for functional static transformations
carmocca Aug 19, 2021
9f383dc
Split tests
carmocca Aug 19, 2021
2a7dfa8
Refactor optimizer tests
carmocca Aug 19, 2021
423ab7b
Cleaning tests
carmocca Aug 19, 2021
7c2e39e
Delete broken test
carmocca Aug 19, 2021
048e159
Docs improvements
carmocca Aug 19, 2021
86fce55
Docs improvements
carmocca Aug 19, 2021
624b0d8
Restructure docs
carmocca Aug 19, 2021
2cc0dc5
Docs for callbacks
carmocca Aug 19, 2021
f9b49fe
Add reload test when add_optimizer_args is added by the user
carmocca Aug 19, 2021
afcc4ba
Add failing config test - needs to be fixed
carmocca Aug 19, 2021
9f41b88
Merge branch 'master' into lightning_cli_registries
carmocca Aug 28, 2021
0ed4ae8
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Aug 28, 2021
4dd0732
Use property
carmocca Aug 19, 2021
e0fae4f
Fixes after merge
carmocca Aug 28, 2021
4f053bb
Merge branch 'master' into lightning_cli_registries
carmocca Sep 15, 2021
a22fdb3
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Sep 15, 2021
160b3f6
Update jsonargparse version
carmocca Sep 15, 2021
f185c2d
Use properties in registry
carmocca Sep 15, 2021
803385c
Keep hacks together
carmocca Sep 15, 2021
8eb8b05
Add FIXMEs
carmocca Sep 15, 2021
9d84127
add_class_choices
carmocca Sep 15, 2021
33ff2f4
Merge branch 'master' into lightning_cli_registries
carmocca Sep 15, 2021
cf82e1a
Remove contains registry. Avoid nested_key clash for optimizers and l…
carmocca Sep 15, 2021
b1cd083
Remove sanitize argv
carmocca Sep 15, 2021
95d31a7
Better support for new callback format
carmocca Sep 16, 2021
231e0ed
Avoid evaluating
carmocca Sep 16, 2021
2af596f
Minor cleaning
carmocca Sep 16, 2021
6add619
Mark argv as private
carmocca Sep 16, 2021
525358a
Fix mypy
carmocca Sep 16, 2021
84b8120
Fix mypy
carmocca Sep 16, 2021
7e48c0e
Fix mypy
carmocca Sep 16, 2021
40ce3c7
Merge branch 'master' into lightning_cli_registries
carmocca Sep 16, 2021
3e77e8e
Support shorthand notation to instantiate optimizers and learning rat…
carmocca Sep 16, 2021
1512a80
Update CHANGELOG
carmocca Sep 16, 2021
c6b86b1
Fix install
carmocca Sep 16, 2021
6f1600c
Fix install
carmocca Sep 16, 2021
a3a791f
Use release
carmocca Sep 16, 2021
f67a90f
Merge branch 'feat/cli-shorthand-optimizers' into lightning_cli_regis…
carmocca Sep 16, 2021
fedae46
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Sep 16, 2021
ee7a068
Introduce set_choices
carmocca Sep 16, 2021
6e67617
Undo change
carmocca Sep 16, 2021
e7f6d61
Replace add_class_choices with set_choices
carmocca Sep 16, 2021
8e87359
Replace add_class_choices with set_choices
carmocca Sep 16, 2021
c74426b
Merge
carmocca Sep 16, 2021
66cdb52
Docstrings
carmocca Sep 16, 2021
9217304
Merge branch 'master' into lightning_cli_registries
carmocca Sep 17, 2021
7b50401
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Sep 17, 2021
1406be9
Fix mypy
carmocca Sep 17, 2021
a000446
Undo change
carmocca Sep 17, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,9 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
* Added `FastForwardSampler` and `CaptureIterableDataset` injection to data loading utilities ([#8366](https://github.com/PyTorchLightning/pytorch-lightning/pull/8366))


- Added simplification for handling optimizers, schedulers and callbacks with `LightningCLI` ([#8815](https://github.com/PyTorchLightning/pytorch-lightning/pull/8815))


- Added `CheckpointIO` to expose checkpoint IO from training type plugin ([#8743](https://github.com/PyTorchLightning/pytorch-lightning/pull/8743))


Expand Down
100 changes: 100 additions & 0 deletions docs/source/common/lightning_cli.rst
Original file line number Diff line number Diff line change
Expand Up @@ -312,6 +312,42 @@ Similar to the callbacks, any arguments in :class:`~pytorch_lightning.trainer.tr
:class:`~pytorch_lightning.core.datamodule.LightningDataModule` classes that have as type hint a class can be configured
the same way using :code:`class_path` and :code:`init_args`.

Lightning optionally simplifies the user command line so that only the :class:`~pytorch_lightning.callbacks.Callback` name is required.
tchaton marked this conversation as resolved.
Show resolved Hide resolved
The argument's order matters and the user needs to pass the arguments in the following way.
This is supported only for PyTorch Lightning built-in :class:`~pytorch_lightning.callbacks.Callback`.

.. code-block:: bash

$ python ... --trainer.callbacks={CALLBACK_NAME_1} --trainer.callbacks.{CALLBACK_1_ARGS_1}=... --trainer.callbacks.{CALLBACK_1_ARGS_2}=... --trainer.callbacks={CALLBACK_N} --trainer.callbacks.{CALLBACK_N_ARGS_1}=...

Here is an example:

.. code-block:: bash

$ python ... --trainer.callbacks=EarlyStopping --trainer.callbacks.patience=5 --trainer.callbacks=LearningRateMonitor

Register your callbacks
^^^^^^^^^^^^^^^^^^^^^^^

Lightning provides registries for you to add your own callbacks and benefit from the command line simplification as described above:

.. code-block:: python

from pytorch_lightning.utilities.cli import CALLBACK_REGISTRY
from pytorch_lightning.callbacks import Callback


@CALLBACK_REGISTRY
class CustomCallback(Callback):
...


cli = LightningCLI(...)

.. code-block:: bash

$ python ... --trainer.callbacks=CustomCallback ...


Multiple models and/or datasets
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Expand Down Expand Up @@ -683,6 +719,12 @@ And the same through command line:

$ python train.py --optimizer.class_path=torch.optim.Adam --optimizer.init_args.lr=0.01

Optionally, the command line can be simplified for PyTorch built-in `optimizers` and `schedulers`:
tchaton marked this conversation as resolved.
Show resolved Hide resolved

.. code-block:: bash

$ python train.py --optimizer=Adam --optimizer.lr=0.01

The automatic implementation of :code:`configure_optimizers` can be disabled by linking the configuration group. An
example can be :code:`ReduceLROnPlateau` which requires to specify a monitor. This would be:

Expand Down Expand Up @@ -717,13 +759,71 @@ example can be :code:`ReduceLROnPlateau` which requires to specify a monitor. Th

cli = MyLightningCLI(MyModel)


For both possibilities of using :meth:`pytorch_lightning.utilities.cli.LightningArgumentParser.add_optimizer_args` with
a single class or a tuple of classes, the value given to :code:`optimizer_init` will always be a dictionary including
:code:`class_path` and :code:`init_args` entries. The function
:func:`~pytorch_lightning.utilities.cli.instantiate_class` takes care of importing the class defined in
:code:`class_path` and instantiating it using some positional arguments, in this case :code:`self.parameters()`, and the
:code:`init_args`. Any number of optimizers and learning rate schedulers can be added when using :code:`link_to`.

Built in schedulers & optimizers and registering your own
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

For code simplification, the LightningCLI provides properties with PyTorch's built-in `optimizers` and `schedulers` already registered.

Only the optimizer or scheduler name needs to be passed along its arguments.

.. code-block:: bash

$ python train.py --optimizer=Adam --optimizer.lr=0.01 --lr_scheduler=CosineAnnealingLR

If your model requires multiple optimizers, the LightningCLI provides already registered optimizers and schedulers under the properties `registered_optimizers` and `registered_lr_schedulers`

.. code-block::

class MyLightningCLI(LightningCLI):
def add_arguments_to_parser(self, parser):
parser.add_optimizer_args(
self.registered_optimizers,
nested="gen_optimizer",
link_to="model.optimizer_init",
)
parser.add_optimizer_args(
self.registered_optimizers,
nested="gen_discriminator",
link_to="model.optimizer_init",
)

.. code-block:: bash

$ python train.py --gen_optimizer=Adam --optimizer.lr=0.01 -gen_discriminator=Adam --optimizer.lr=0.0001

Furthermore, a user can register their own optimizers or schedulers as follows.

.. code-block:: python

import torch
from pytorch_lightning.utilities.cli import OPTIMIZER_REGISTRY, LR_SCHEDULER_REGISTRY
from pytorch_lightning.callbacks import Callback


@OPTIMIZER_REGISTRY
class CustomAdam(torch.optim.Adam):
pass


@LR_SCHEDULER_REGISTRY
class CustomCosineAnnealingLR(torch.optim.lr_scheduler.CosineAnnealingLR):
pass


cli = LightningCLI(...)

.. code-block:: bash

$ python train.py --optimizer=CustomAdam --optimizer.lr=0.01 --lr_scheduler=CustomCosineAnnealingLR


Notes related to reproducibility
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Expand Down
3 changes: 1 addition & 2 deletions pytorch_lightning/plugins/plugins_registry.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,6 @@
# limitations under the License.
import importlib
import inspect
from collections import UserDict
from inspect import getmembers, isclass
from pathlib import Path
from typing import Any, Callable, Dict, List, Optional
Expand All @@ -22,7 +21,7 @@
from pytorch_lightning.utilities.exceptions import MisconfigurationException


class _TrainingTypePluginsRegistry(UserDict):
class _TrainingTypePluginsRegistry(dict):
"""
This class is a Registry that stores information about the Training Type Plugins.

Expand Down
Loading