From 17c7c86d5f1065d26f6ca78a82833c3f25effe55 Mon Sep 17 00:00:00 2001 From: shayaharon Date: Thu, 26 Jan 2023 16:33:35 +0200 Subject: [PATCH 1/5] final version --- documentation/assets/Losses.md | 153 +++++++++++++++++++++++++++++++++ 1 file changed, 153 insertions(+) create mode 100644 documentation/assets/Losses.md diff --git a/documentation/assets/Losses.md b/documentation/assets/Losses.md new file mode 100644 index 0000000000..b2f0ebedc1 --- /dev/null +++ b/documentation/assets/Losses.md @@ -0,0 +1,153 @@ +# Losses in SG + +SuperGradients provides multiple Loss function implementations for various tasks: + + cross_entropy + mse + r_squared_loss + shelfnet_ohem_loss + shelfnet_se_loss + yolox_loss + yolox_fast_loss + ssd_loss + stdc_loss + bce_dice_loss + kd_loss + dice_ce_edge_loss + + +Usage: + +- When using train_from_recipe (or similar, when the underlying train method that is being called is Trainer.train_from_config(...): + - In your `my_yolox_training_hyperparams.yaml` file: + ```yaml + ... + ... + + loss: yolox_loss + + criterion_params: + strides: [8, 16, 32] # output strides of all yolo outputs + num_classes: 80 + ``` + + `criterion_params` dictionary will be unpacked to the underlying `yolox_loss` class constructor. + + +- When using a direct Trainer.train(...) call: + - In your `my_training_script.py` + ```python + ... + trainer = Trainer("external_criterion_test") + train_dataloader = ... + valid_dataloader = ... + model = ... + + train_params = { + ... + "loss": "cross_entropy", + "criterion_params": {} + ... + } + trainer.train(model=model, training_params=train_params, train_loader=dataloader, valid_loader=dataloader) + ``` + Note that object names in SG are not case-sensitive nor symbol-sensitive, so `"CrossEntropy` could have been passed as well. + Since most IDEs support auto-completion, for your convenience, you can use our object_names module: + ```python + from super_gradients.common.object_names import Losses + ``` + Then simply instead of "cross_entropy", use + ```python + Losses.CROSS_ENTROPY + ``` + +## Using Your Own Loss +SuperGradients also supports user-defined loss functions assuming they are torch.nn.Module inheritors, and that their `forward` signature is in the form: + +```python +import torch.nn + +MyLoss(torch.nn.Module): +... +forward(preds, target): +... + +``` +And as the argument names suggest- the first argument is the model's output, and target is the label/ground truth (argument naming is arbitrary and does not need to be specifically 'preds' or 'target'). +Loss functions accepting additional arguments in their `forward` method will be supported in the future. +- When using train_from_recipe (or similar, when the underlying train method that is being called is Trainer.train_from_config(...): + - In your ``my_loss.py``, register your loss class by decorating the class with `register_loss`: + ```python + import torch.nn + from super_gradients.common.registry import register_loss + + @register_loss("my_loss") + class MyLoss(torch.nn.Module): + ... + ``` + - Then, in your `my_training_hyperparams.yaml`, use `"my_loss"` in the same way as any other loss supported in SG: + ```yaml + ... + ... + + loss: my_loss + + criterion_params: + ... + ``` + - Last, in your ``my_train_from_recipe_script.py`` file, just import the newly registered class (even though the class itself is unused, just to trigger the registry): + + ```python + from omegaconf import DictConfig + import hydra + import pkg_resources + from my_loss import MyLoss + from super_gradients import Trainer, init_trainer + + + @hydra.main(config_path=pkg_resources.resource_filename("super_gradients.recipes", ""), version_base="1.2") + def main(cfg: DictConfig) -> None: + Trainer.train_from_config(cfg) + + + def run(): + init_trainer() + main() + + + if __name__ == "__main__": + run() + + + + ``` +## Passing Instantiated nn.Module Objects as Loss Functions + +SuperGradients also supports passing instantiated nn.Module Objects as demonstrated below: +- When using a direct Trainer.train(...) call: + - In you `my_training_script.py` + ```python + ... + trainer = Trainer("external_criterion_test") + train_dataloader = ... + valid_dataloader = ... + model = ... + + train_params = { + ... + "loss": torch.nn.CrossEntropy() + ... + } + trainer.train(model=model, training_params=train_params, train_loader=dataloader, valid_loader=dataloader) + ``` + - Though not as convenient as using `register_loss`, one can also equivalently instantiate objects when using train_from_recipe (or similar, when the underlying train method being called is Trainer.train_from_config(...) + As demonstrated below: + - In your `my_training_hyperparams.yaml` file: + ```yaml + ... + ... + loss: + _target_: torch.nn.CrossEntropy + + ``` + Note that when passing an instantiated loss object, `criterion_params` will be ignored. From fcdba498252290fe4fb54a1229ee13df6b1d7604 Mon Sep 17 00:00:00 2001 From: shayaharon Date: Thu, 26 Jan 2023 17:24:14 +0200 Subject: [PATCH 2/5] removed bullitins, added more text explanations --- documentation/assets/Losses.md | 100 +++++++++++++++++---------------- 1 file changed, 51 insertions(+), 49 deletions(-) diff --git a/documentation/assets/Losses.md b/documentation/assets/Losses.md index b2f0ebedc1..78770c6da0 100644 --- a/documentation/assets/Losses.md +++ b/documentation/assets/Losses.md @@ -15,11 +15,14 @@ SuperGradients provides multiple Loss function implementations for various tasks kd_loss dice_ce_edge_loss +All of the above, are just string aliases for the underlying torch.nn.Module classes, implementing the specified loss functions. -Usage: +##Basic Usage of Implemented Loss Functions in SG: -- When using train_from_recipe (or similar, when the underlying train method that is being called is Trainer.train_from_config(...): - - In your `my_yolox_training_hyperparams.yaml` file: +When using configuration files, for example training using train_from_recipe (or similar, when the underlying train method that is being called is Trainer.train_from_config(...)): + + +In your `my_yolox_training_hyperparams.yaml` file: ```yaml ... ... @@ -34,7 +37,7 @@ Usage: `criterion_params` dictionary will be unpacked to the underlying `yolox_loss` class constructor. -- When using a direct Trainer.train(...) call: +Another usage case, is when using a direct Trainer.train(...) call: - In your `my_training_script.py` ```python ... @@ -49,7 +52,7 @@ Usage: "criterion_params": {} ... } - trainer.train(model=model, training_params=train_params, train_loader=dataloader, valid_loader=dataloader) + trainer.train(model=model, training_params=train_params, train_loader=train_dataloader, valid_loader=valid_dataloader) ``` Note that object names in SG are not case-sensitive nor symbol-sensitive, so `"CrossEntropy` could have been passed as well. Since most IDEs support auto-completion, for your convenience, you can use our object_names module: @@ -75,29 +78,29 @@ forward(preds, target): ``` And as the argument names suggest- the first argument is the model's output, and target is the label/ground truth (argument naming is arbitrary and does not need to be specifically 'preds' or 'target'). Loss functions accepting additional arguments in their `forward` method will be supported in the future. -- When using train_from_recipe (or similar, when the underlying train method that is being called is Trainer.train_from_config(...): - - In your ``my_loss.py``, register your loss class by decorating the class with `register_loss`: - ```python - import torch.nn - from super_gradients.common.registry import register_loss - - @register_loss("my_loss") - class MyLoss(torch.nn.Module): +When using configuration files, for example training using train_from_recipe (or similar, when the underlying train method that is being called is Trainer.train_from_config(...)), In your ``my_loss.py``, register your loss class by decorating the class with `register_loss`: +```python + import torch.nn + from super_gradients.common.registry import register_loss + + @register_loss("my_loss") + class MyLoss(torch.nn.Module): ... - ``` - - Then, in your `my_training_hyperparams.yaml`, use `"my_loss"` in the same way as any other loss supported in SG: +``` +Then, in your `my_training_hyperparams.yaml`, use `"my_loss"` in the same way as any other loss supported in SG: ```yaml - ... - ... - - loss: my_loss - - criterion_params: - ... +... +... + +loss: my_loss + +criterion_params: + ... ``` - - Last, in your ``my_train_from_recipe_script.py`` file, just import the newly registered class (even though the class itself is unused, just to trigger the registry): +Last, in your ``my_train_from_recipe_script.py`` file, just import the newly registered class (even though the class itself is unused, just to trigger the registry): - ```python +```python + from omegaconf import DictConfig import hydra import pkg_resources @@ -119,35 +122,34 @@ Loss functions accepting additional arguments in their `forward` method will be run() - - ``` +``` ## Passing Instantiated nn.Module Objects as Loss Functions SuperGradients also supports passing instantiated nn.Module Objects as demonstrated below: -- When using a direct Trainer.train(...) call: - - In you `my_training_script.py` - ```python - ... - trainer = Trainer("external_criterion_test") - train_dataloader = ... - valid_dataloader = ... - model = ... - - train_params = { - ... - "loss": torch.nn.CrossEntropy() - ... - } - trainer.train(model=model, training_params=train_params, train_loader=dataloader, valid_loader=dataloader) - ``` - - Though not as convenient as using `register_loss`, one can also equivalently instantiate objects when using train_from_recipe (or similar, when the underlying train method being called is Trainer.train_from_config(...) - As demonstrated below: - - In your `my_training_hyperparams.yaml` file: - ```yaml +When using a direct Trainer.train(...) call, in your `my_training_script.py` simply pass the instantiated nn.Module under the "loss" key inside training_params: +```python +... +trainer = Trainer("external_criterion_test") +train_dataloader = ... +valid_dataloader = ... +model = ... + +train_params = { ... + "loss": torch.nn.CrossEntropy() ... - loss: - _target_: torch.nn.CrossEntropy +} +trainer.train(model=model, training_params=train_params, train_loader=dataloader, valid_loader=dataloader) + ``` +Though not as convenient as using `register_loss`, one can also equivalently instantiate objects when using train_from_recipe (or similar, when the underlying train method is Trainer.train_from_config(...) as demonstrated below: - ``` + +In your `my_training_hyperparams.yaml` file: +```yaml + ... + ... + loss: + _target_: torch.nn.CrossEntropy + +``` Note that when passing an instantiated loss object, `criterion_params` will be ignored. From 5f91928342bd08e0c3e3e4c147016ca62d8e8deb Mon Sep 17 00:00:00 2001 From: shayaharon Date: Thu, 26 Jan 2023 17:33:20 +0200 Subject: [PATCH 3/5] my training hyperparams renamed --- documentation/assets/Losses.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/documentation/assets/Losses.md b/documentation/assets/Losses.md index 78770c6da0..b6075b0003 100644 --- a/documentation/assets/Losses.md +++ b/documentation/assets/Losses.md @@ -22,7 +22,7 @@ All of the above, are just string aliases for the underlying torch.nn.Module cla When using configuration files, for example training using train_from_recipe (or similar, when the underlying train method that is being called is Trainer.train_from_config(...)): -In your `my_yolox_training_hyperparams.yaml` file: +In your `my_training_hyperparams.yaml` file: ```yaml ... ... From e6cc16d897d78a095f5fce559fa70957a8ddd783 Mon Sep 17 00:00:00 2001 From: shayaharon Date: Sun, 29 Jan 2023 11:47:46 +0200 Subject: [PATCH 4/5] lint --- documentation/assets/Losses.md | 179 +++++++++++++++++++++++---------- 1 file changed, 125 insertions(+), 54 deletions(-) diff --git a/documentation/assets/Losses.md b/documentation/assets/Losses.md index b6075b0003..03ad26d033 100644 --- a/documentation/assets/Losses.md +++ b/documentation/assets/Losses.md @@ -1,6 +1,6 @@ # Losses in SG -SuperGradients provides multiple Loss function implementations for various tasks: +SuperGradients can support any PyTorch-based loss function. Additionally, multiple Loss function implementations for various tasks are also supported: cross_entropy mse @@ -15,30 +15,13 @@ SuperGradients provides multiple Loss function implementations for various tasks kd_loss dice_ce_edge_loss -All of the above, are just string aliases for the underlying torch.nn.Module classes, implementing the specified loss functions. +All the above, are just string aliases for the underlying torch.nn.Module classes, implementing the specified loss functions. -##Basic Usage of Implemented Loss Functions in SG: +## Basic Usage of Implemented Loss Functions in SG: -When using configuration files, for example training using train_from_recipe (or similar, when the underlying train method that is being called is Trainer.train_from_config(...)): - - -In your `my_training_hyperparams.yaml` file: - ```yaml - ... - ... - - loss: yolox_loss - - criterion_params: - strides: [8, 16, 32] # output strides of all yolo outputs - num_classes: 80 - ``` - - `criterion_params` dictionary will be unpacked to the underlying `yolox_loss` class constructor. +The most basic use case is when using a direct Trainer.train(...) call: - -Another usage case, is when using a direct Trainer.train(...) call: - - In your `my_training_script.py` +In your `my_training_script.py`: ```python ... trainer = Trainer("external_criterion_test") @@ -64,7 +47,57 @@ Another usage case, is when using a direct Trainer.train(...) call: Losses.CROSS_ENTROPY ``` + +Another use case is when using configuration files. For example, when training using train_from_recipe (or similar, when the underlying train method that is being called is Trainer.train_from_config(...)). + +When doing so, in your `my_training_hyperparams.yaml` file: + ```yaml + ... + ... + + loss: yolox_loss + + criterion_params: + strides: [8, 16, 32] # output strides of all yolo outputs + num_classes: 80 + ``` + + `criterion_params` dictionary will be unpacked to the underlying `yolox_loss` class constructor. + +## Passing Instantiated nn.Module Objects as Loss Functions + +SuperGradients also supports passing instantiated nn.Module Objects as demonstrated below: +When using a direct Trainer.train(...) call, in your `my_training_script.py` simply pass the instantiated nn.Module under the "loss" key inside training_params: +```python +... +trainer = Trainer("external_criterion_test") +train_dataloader = ... +valid_dataloader = ... +model = ... + +train_params = { + ... + "loss": torch.nn.CrossEntropy() + ... +} +trainer.train(model=model, training_params=train_params, train_loader=dataloader, valid_loader=dataloader) + ``` +Though not as convenient as using `register_loss` (discussed further into detail in the next sub-section), one can also equivalently instantiate objects when using train_from_recipe (or similar, when the underlying train method is Trainer.train_from_config(...) as demonstrated below: + + +In your `my_training_hyperparams.yaml` file: +```yaml + ... + ... + loss: + _target_: torch.nn.CrossEntropy + +``` + Note that when passing an instantiated loss object, `criterion_params` will be ignored. + + ## Using Your Own Loss + SuperGradients also supports user-defined loss functions assuming they are torch.nn.Module inheritors, and that their `forward` signature is in the form: ```python @@ -76,9 +109,77 @@ forward(preds, target): ... ``` -And as the argument names suggest- the first argument is the model's output, and target is the label/ground truth (argument naming is arbitrary and does not need to be specifically 'preds' or 'target'). +And as the argument names suggest, the first argument is the model's output, and target is the label/ground truth (argument naming is arbitrary and does not need to be specifically 'preds' or 'target'). Loss functions accepting additional arguments in their `forward` method will be supported in the future. -When using configuration files, for example training using train_from_recipe (or similar, when the underlying train method that is being called is Trainer.train_from_config(...)), In your ``my_loss.py``, register your loss class by decorating the class with `register_loss`: + +### Using Your Own Loss- Logging Loss Outputs + +In the most common case, where the loss function returns a single item for backprop the loss output will appear in +the logs, training logs (i.e Tensorboards and any other supported SGLogger, for more information on SGLoggers click [here](https://github.com/Deci-AI/super-gradients)), over epochs under . + +forward(...) should return a (loss, loss_items) tuple where loss is the tensor used +for backprop (i.e what your original loss function returns), and loss_items should be a tensor of +shape (n_items) consisting of values computed during the forward pass which we desire to log over the +entire epoch. For example- the loss itself should always be logged. Another example is a scenario +where the computed loss is the sum of a few components we would like to log. + +For example: +```python +class MyLoss(_Loss): + ... + def forward(self, inputs, targets): + ... + total_loss = comp1 + comp2 + loss_items = torch.cat((total_loss.unsqueeze(0),comp1.unsqueeze(0), comp2.unsqueeze(0)).detach() + return total_loss, loss_items + ... + +Trainer.train(... + train_params={"loss":MyLoss(), + ... + "metric_to_watch": "MyLoss2/loss_0"} + +``` + + +The above snippet will log `MyLoss2/loss_0`, `MyLoss2/loss_1` and `MyLoss2/loss_2` as they have been named by their positional index in loss_items. +Note we also defined "MyLoss2/loss_0" to be our watched metric which means we save our checkpoint every epoch we reach the best loss score. + +For more visibility, you can also set a "component_names" property in the loss class, +to be a list of strings, of length n_items whose ith element is the name of the ith entry in loss_items. +Then each item will be logged, rendered on the tensorboard, and "watched" (i.e saving model checkpoints +according to it) under `/`. + +For example: +```python + class MyLoss(_Loss): + ... + def forward(self, inputs, targets): + ... + total_loss = comp1 + comp2 + loss_items = torch.cat((total_loss.unsqueeze(0),comp1.unsqueeze(0), comp2.unsqueeze(0)).detach() + return total_loss, loss_items + ... + @property + def component_names(self): + return ["total_loss", "my_1st_component", "my_2nd_component"] + +Trainer.train(... + train_params={"loss":MyLoss(), + ... + "metric_to_watch": "MyLoss/my_1st_component"} +``` + + +The above code will log and monitor `MyLoss/total_loss`, `MyLoss/my_1st_component` and `MyLoss/my_2nd_component`. + + +Since running logs will save the loss_items in some internal state, it is recommended to +detach loss_items from their computational graph for memory efficiency. + +### Using Your Own Loss- Training with Configuration Files + +When using configuration files, for example, training using train_from_recipe (or similar, when the underlying train method that is being called is Trainer.train_from_config(...)), In your ``my_loss.py``, register your loss class by decorating the class with `register_loss`: ```python import torch.nn from super_gradients.common.registry import register_loss @@ -123,33 +224,3 @@ Last, in your ``my_train_from_recipe_script.py`` file, just import the newly reg ``` -## Passing Instantiated nn.Module Objects as Loss Functions - -SuperGradients also supports passing instantiated nn.Module Objects as demonstrated below: -When using a direct Trainer.train(...) call, in your `my_training_script.py` simply pass the instantiated nn.Module under the "loss" key inside training_params: -```python -... -trainer = Trainer("external_criterion_test") -train_dataloader = ... -valid_dataloader = ... -model = ... - -train_params = { - ... - "loss": torch.nn.CrossEntropy() - ... -} -trainer.train(model=model, training_params=train_params, train_loader=dataloader, valid_loader=dataloader) - ``` -Though not as convenient as using `register_loss`, one can also equivalently instantiate objects when using train_from_recipe (or similar, when the underlying train method is Trainer.train_from_config(...) as demonstrated below: - - -In your `my_training_hyperparams.yaml` file: -```yaml - ... - ... - loss: - _target_: torch.nn.CrossEntropy - -``` - Note that when passing an instantiated loss object, `criterion_params` will be ignored. From 2d799e1723e68f36e89b1117bc8f5d8fd237f0d0 Mon Sep 17 00:00:00 2001 From: shayaharon Date: Sun, 29 Jan 2023 11:58:13 +0200 Subject: [PATCH 5/5] comment resolved --- documentation/assets/Losses.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/documentation/assets/Losses.md b/documentation/assets/Losses.md index 03ad26d033..58d0cf62fb 100644 --- a/documentation/assets/Losses.md +++ b/documentation/assets/Losses.md @@ -61,8 +61,8 @@ When doing so, in your `my_training_hyperparams.yaml` file: strides: [8, 16, 32] # output strides of all yolo outputs num_classes: 80 ``` - - `criterion_params` dictionary will be unpacked to the underlying `yolox_loss` class constructor. + +Note that two `training_params` parameters define the loss function: `loss` which defines the type of the loss, and`criterion_params` dictionary which will be unpacked to the underlying `yolox_loss` class constructor. ## Passing Instantiated nn.Module Objects as Loss Functions