Skip to content
This repository has been archived by the owner on Nov 3, 2023. It is now read-only.

"parlai eval_model" doesn't produce tensorboard report when "--tensorboard-log" is enabled #4495

Closed
DrMatters opened this issue Apr 10, 2022 · 2 comments

Comments

@DrMatters
Copy link
Contributor

DrMatters commented Apr 10, 2022

Command to reproduce the bug:

parlai eval_model -t blended_skill_talk -mf zoo:blender/blender_90M/model --skip-generation True --tensorboard-log True --tensorboard-logdir ~/tb_report_for_issuue/

A user expects to see a folder called tb_report_for_issuue in the home directory containing the evaluation report data. The expected behavior is the same as when enabling tensorboard logging with parlai train_model.

P.S. It may be helpful to add other loggers (such as WandB) to the eval_model script.

@DrMatters DrMatters changed the title "parlai eval_model" doesn't produce TB report when "--tensorboard-log" is true and the TB directory set "parlai eval_model" doesn't produce TB report when "--tensorboard-log" is enabled Apr 10, 2022
@DrMatters DrMatters changed the title "parlai eval_model" doesn't produce TB report when "--tensorboard-log" is enabled "parlai eval_model" doesn't produce tensorboard report when "--tensorboard-log" is enabled Apr 10, 2022
@stephenroller
Copy link
Contributor

Considering TB expects a time series, not a static evaluation, what should be logged?

Happy to accept a patch for this if you've got an idea.

@DrMatters
Copy link
Contributor Author

PR: #4497
@stephenroller

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

2 participants