-
Notifications
You must be signed in to change notification settings - Fork 3.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Return the evaluation result of Trainer.test #1694
Comments
Hi! thanks for your contribution!, great first issue! |
Hi, I believe it is already possible to get the metrics. The trainer stores them after validation or test. I also think that returning the results in test would be a good feature to have. |
.test() already prints out the metrics no? But yeah, it would be nice to get: metrics = trainer.test() However, this is non-trivial in ddp or TPUs... but we can try it out. @awaelchli want to give this a shot? this is a hard one lol |
or we could do the same hack we did with weights on ddp and save the metrics to disk, exit process, load back up and return to user. |
@williamFalcon thanks for the work in #2029 as it helps fix the issue for the ddp case. However, it seems the part that returns the evaluation result is still missing. Maybe you want to re-open the issue? |
it is also related to #1989 |
🚀 Feature
This enhancement request is to let
Trainer.test
return the dictionary of test metrics.Motivation
Currently,
Trainer.test
returns nothing. The user would have to open Tensorboard to see the test result or write a custom logger. For beginners, this enhancement offers simplicity to beginners.In some scenarios for hparams search, the calling program needs to have the test result of each trial. Thus, it would be handy if
Trainer.test
returns a value.Pitch
If
test_epoch_end
already defines the return value, we can return theeval_results
fromrun_evaluation
and letTrainer.test
return that result.Alternatives
Pass a reference to a mutable collection of metrics summary to the Module and
on_test_end
, the user can choose to update the collection.Additional context
There could be some refactoring that needs to be done as currently,
Trainer.test
has different code paths depending on whethermodel
is passed or not or whetherddp
is used or not.The text was updated successfully, but these errors were encountered: