Support deterministic="warn" in Trainer for Pytorch 1.11+

[weiji14]

What does this PR do?

In Pytorch 1.11, operations that do not have a deterministic implementation can be set to throw a warning instead of an error when ran in deterministic mode. See https://pytorch.org/docs/1.11/generated/torch.use_deterministic_algorithms.html

Fixes #<issue_number>

Does your PR introduce any breaking changes? If yes, please list them.

Before submitting

• Was this discussed/approved via a GitHub issue? (not for typos and docs)
• Did you read the contributor guideline, Pull Request section?
• Did you make sure your PR does only one thing, instead of bundling different changes together?
• Did you make sure to update the documentation with your changes? (if necessary)
• Did you write any new necessary tests? (not for typos and docs)
• Did you verify new and existing tests pass locally with your changes?
• Did you list all the breaking changes introduced by this pull request?
• Did you update the CHANGELOG? (not for typos, docs, test updates, or internal minor changes/refactorings)

PR review

Anyone in the community is welcome to review the PR.
Before you start reviewing make sure you have read Review guidelines. In short, see the following bullet-list:

• Is this pull request ready for review? (if not, please submit in draft mode)
• Check that all items from Before submitting are resolved
• Make sure the title is self-explanatory and the description concisely explains the PR
• Add labels and milestones (and optionally projects) to the PR so it can be classified

Did you have fun?

Make sure you had fun coding 🙃

[carmocca]

Cool! Pushed a commit with some minor changes

[akihironitta]

Nice addition!

---

The type check is failing in CI: https://github.com/PyTorchLightning/pytorch-lightning/runs/5853483314?check_suite_focus=true

Once the failure is resolved, LGTM!

[weiji14]

Cool, thanks everyone for sorting out the type hints! Great to learn about "Literal" types 😆

[awaelchli]

I find this is a confusing feature. Either I want determinism or not. Something in-between like "warn" does not exist for me. Perhaps I'm missing the use case here.

[weiji14]

I find this is a confusing feature. Either I want determinism or not. Something in-between like "warn" does not exist for me. Perhaps I'm missing the use case here.

This thread might be a useful read: pytorch/pytorch#64883. Ideally yes, full determinism would be desirable, but in practice, not every algorithm has a deterministic implementation and this gets us 90% of the way there (depending on the model being used). I'm coming at it more from a reproducibility standpoint, it's better to be able to reproduce a model's result by 90% than to not be able to reproduce it at all because of all the randomness going on.

[mareikethies]

Hi, nice update which is useful for me. Could it be that the add_argparse_args() method in Trainer needs to be adapted as well? Currently, it accepts only boolean values for the deterministic keyword. At least I don't know how to pass the warn flag using argparse. Thanks!

[weiji14]

Hmm, looking at add_argparse_args(), I don't quite know where a change needs to happen. That said, what version of pytorch-lightning are you using? Don't think this feature is in the recent PyPI 1.6.2 release, might need to wait for 1.7.0 or install from GitHub.

[mareikethies]

Yes, I did install from github today. I have not had a look at the source code, but I observe that when I run something like python train.py --deterministic warn using Trainer.add_argparse_args() in train.py, I get this error: error: argument --deterministic: invalid str_to_bool value: 'warn'. But I might miss something here or pass it the wrong way. Just plain python train.py --deterministic works.

[mauvilsa]

@mareikethies any particular reason you want to use add_argparse_args? If you use LightningCLI this will not be a problem.

[weiji14]

Yes, I did install from github today. I have not had a look at the source code, but I observe that when I run something like python train.py --deterministic warn using Trainer.add_argparse_args() in train.py, I get this error: error: argument --deterministic: invalid str_to_bool value: 'warn'. But I might miss something here or pass it the wrong way. Just plain python train.py --deterministic works.

Thanks @mareikethies, that error message is very helpful. I can confirm the invalid str_to_bool value: 'warn' error. Minimal working example:

1. Do pip install https://github.com/PyTorchLightning/pytorch-lightning/archive/6490996b3974019a254f2750f5c01e6d38ff6a6e.zip
2. Create a file called train_determinstic.py like so:

import argparse
import pytorch_lightning as pl
import torch


class BasicNet(pl.LightningModule):
    def __init__(self):
        super().__init__()
        self.layer = torch.nn.Linear(2, 2)

    def forward(self, x):
        return self.layer(x)

    def training_step(self, batch, batch_idx):
        output = self(batch)
        loss = torch.nn.functional.mse_loss(batch, output)
        return {"loss": loss}

    def configure_optimizers(self):
        return torch.optim.Adam(params=self.parameters(), lr=0.1, weight_decay=0.0005)


class RandomDataset(torch.utils.data.Dataset):
    def __init__(self):
        pass

    def __len__(self):
        return 1

    def __getitem__(self, idx: int):
        return torch.randn(2)


def cli_main():
    parser = argparse.ArgumentParser()
    parser = pl.Trainer.add_argparse_args(parent_parser=parser)
    args = parser.parse_args()
    trainer = pl.Trainer.from_argparse_args(args=args, max_epochs=1, accelerator="auto")

    model = BasicNet()
    dataloader = torch.utils.data.DataLoader(dataset=RandomDataset())

    trainer.fit(model=model, train_dataloaders=dataloader)


if __name__ == "__main__":
    cli_main()

3. Run python deterministic.py --deterministic=warn, which will produce deterministic.py: error: argument --deterministic: invalid str_to_bool value: 'warn'

So my guess is that the code here might need to change:
https://github.com/PyTorchLightning/pytorch-lightning/blob/0fe2ae74e919c8ef220a029cab2203572ca536a5/pytorch_lightning/utilities/argparse.py#L232-L251