Skip to content
This repository has been archived by the owner on Nov 3, 2023. It is now read-only.

Add TB logging to parlai eval_model script #4497

Merged
merged 6 commits into from
Apr 29, 2022
Merged

Add TB logging to parlai eval_model script #4497

merged 6 commits into from
Apr 29, 2022

Conversation

DrMatters
Copy link
Contributor

Patch description
This patch adds the capability to report evaluation results to the tensorboard when using parlai train_model script.

Testing steps
Ran
parlai eval_model -t blended_skill_talk,empathetic_dialogues -mf zoo:blender/blender_90M/model --skip-generation True --tensorboard-log True --tensorboard-logdir ~/tb_reporting_test/ --bs 32 --datatype valid and with --datatype test

Other information
I was fine-tuning a blenderbot when WandB suddenly failed (probably due to the unintended launch of the second fine-tuning loop on the same model). After that, the tuning process worked fine and finished properly. BUT more important thing is that the final test-set evaluation was never done (some WandB exception caused the failure). So I decided to evaluate the model on the test set using the dedicated eval_model script and then use wandb sync on the generated Tensoflow report.

@DrMatters
Copy link
Contributor Author

#4495

@DrMatters
Copy link
Contributor Author

@stephenroller
I've made a patch as you asked here: #4495 (comment)

@stephenroller
Copy link
Contributor

How does this render? As a single data point in the time series?

@DrMatters
Copy link
Contributor Author

How does this render? As a single data point in the time series?

Yup!
This is still useful as it allows seamlessly wandb sync the tensorboard report into the existing run of wandb

image

@stephenroller stephenroller changed the title Add TB logging to parlai train_model script Add TB logging to parlai eval_model script Apr 28, 2022
parlai/scripts/eval_model.py Outdated Show resolved Hide resolved
@DrMatters
Copy link
Contributor Author

What about failed tests?

@stephenroller stephenroller merged commit a48c0db into facebookresearch:main Apr 29, 2022
@stephenroller
Copy link
Contributor

What about failed tests?

Pretty sure fixed in main.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants