This repository has been archived by the owner on Nov 3, 2023. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 2.1k
Add TB logging to parlai eval_model script #4497
Merged
stephenroller
merged 6 commits into
facebookresearch:main
from
DrMatters:ksemin/add_TB_logging_to_model_eval
Apr 29, 2022
Merged
Add TB logging to parlai eval_model script #4497
stephenroller
merged 6 commits into
facebookresearch:main
from
DrMatters:ksemin/add_TB_logging_to_model_eval
Apr 29, 2022
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
@stephenroller |
…com/DrMatters/ParlAI into ksemin/add_TB_logging_to_model_eval
How does this render? As a single data point in the time series? |
stephenroller
changed the title
Add TB logging to parlai train_model script
Add TB logging to parlai eval_model script
Apr 28, 2022
stephenroller
approved these changes
Apr 28, 2022
stephenroller
approved these changes
Apr 28, 2022
What about failed tests? |
Pretty sure fixed in main. |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Patch description
This patch adds the capability to report evaluation results to the tensorboard when using
parlai train_model
script.Testing steps
Ran
parlai eval_model -t blended_skill_talk,empathetic_dialogues -mf zoo:blender/blender_90M/model --skip-generation True --tensorboard-log True --tensorboard-logdir ~/tb_reporting_test/ --bs 32 --datatype valid
and with--datatype test
Other information
I was fine-tuning a blenderbot when WandB suddenly failed (probably due to the unintended launch of the second fine-tuning loop on the same model). After that, the tuning process worked fine and finished properly. BUT more important thing is that the final test-set evaluation was never done (some WandB exception caused the failure). So I decided to evaluate the model on the test set using the dedicated
eval_model
script and then usewandb sync
on the generated Tensoflow report.