How can I perform only validation without training

[xinfangliu]

❓ Questions and Help

It seems the metric 0.8737 in the checkpoint 'm10-f1_1=0.8737.ckpt' can not be found in progress_bar,
I want to load the .ckpt to perform validation without training,
How should I config the trainer?

[awaelchli]

The progress bar averages the values over time.
To test your model on the validation set you can do

model = ...
trainer = Trainer(...)
trainer.test(model, ckpt_path="path/to/m10-f1_1=0.8737.ckpt", test_dataloaders=model.val_dataloader())

[xinfangliu]

I'm afraid not. This code does not call validation_ step() and validation_ epoch_ end() and it can not calc the metric either

[awaelchli]

You need to implement test_step and test_epoch end. If testing does not differ from your validation, you can just call validstion_step from test_step.

[rohitgr7]

@awaelchli What if they(testing and validation procedures) are different or don't have any test_step or test_epoch_end or just want to use test_step and test_epoch_end to save the outputs of the model in a file. I suggest there should be something like .evaluate() or .validate() just for validation_step and validation_epoch_end with a same api as that of .test(). Thoughts??

[awaelchli]

I think we need to wait for #2107 for this to be possible. evaluate() would have to be different from the validation loop that runs during training, for example, it should not invoke callbacks like early stopping or checkpointing.

[ghost]

I thought its not allowed? I remember seeing it here

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[Code-Cornelius]

what about this:

"You can also run just the validation loop on your validation dataloaders by overriding validation_step() and calling validate()."

model = Model()
trainer = Trainer()
trainer.validate(model)

from https://pytorch-lightning.readthedocs.io/en/latest/common/lightning_module.html ; validation loop.

Also there:

"Validation

You can perform an evaluation epoch over the validation set, outside of the training loop, using pytorch_lightning.trainer.trainer.Trainer.validate(). This might be useful if you want to collect new metrics from a model right at its initialization or after it has already been trained."

trainer.validate(dataloaders=val_dataloaders)

https://pytorch-lightning.readthedocs.io/en/stable/common/trainer.html#validation

[awaelchli]

Yep! Nice find @Code-Cornelius. Since the original posting of this issue, we introduced Trainer.validate.