Pearlli model chat analysis refactor (#4577)

* change analysis script to use databrowernstead of lading from disk * add model chat data for testing purposes * Revert "add model chat data for testing purposes" This reverts commit d2a0e7a. * get rid of re and json imports * remove start-date flag and fix lint error * add dummy class to mocks DataBrowser output * update test results * add results-folders flag * add results-folders flag * remove check for results-folders * remove results-folders flag * update test to run without results-folders * update readme * move --hit-block-list to subclass, add comments, remove unnecessary initialization * add protobuf to requirement for tensorboard 2.9.1 * change tensorboard to 2.9.0 Co-authored-by: Pearl Li <pearlli@devfair0420.h2.fair>
facebookresearch · Jun 15, 2022 · 2a60467 · 2a60467
1 parent 8fc3fec
commit 2a60467
Show file tree

Hide file tree

Showing 11 changed files with 394 additions and 385 deletions.
diff --git a/parlai/crowdsourcing/projects/multisession_chat/human_eval/compile_results.py b/parlai/crowdsourcing/projects/multisession_chat/human_eval/compile_results.py
@@ -29,6 +29,15 @@ class ModelChatResultsCompiler(BaseModelChatResultsCompiler):
     @classmethod
     def setup_args(cls):
         parser = super().setup_args()
+        parser.add_argument(
+            '--hit-block-list',
+            type=str,
+            default='',
+            help='Comma-separated list of all hits to block',
+        )
+        parser.add_argument(
+            '--results-folders', type=str, help='Comma-separated list of result folders'
+        )
         parser.add_argument(
             '--model-nickname', type=str, default='', help='name of the model'
         )
@@ -43,6 +52,10 @@ def setup_args(cls):
     def __init__(self, opt: Dict[str, Any]):
 
         AbstractTurnAnnotationResultsCompiler.__init__(self, opt)
+        if 'results_folders' in opt:
+            self.results_folders = opt['results_folders'].split(',')
+        else:
+            self.results_folders = None
 
         # Input args
         self.model_nickname = opt['model_nickname']

diff --git a/parlai/crowdsourcing/tasks/model_chat/README.md b/parlai/crowdsourcing/tasks/model_chat/README.md
@@ -59,4 +59,4 @@ Note that onboarding is not currently supported with human+model image chat: use
 
 ## Analysis
 
-Run `analysis/compile_results.py` to compile and save statistics about collected human+model chats. The `ModelChatResultsCompiler` in that script uses dummy annotation buckets by default; set `--problem-buckets` in order to define your own. Set `--results-folders` to the value of `mephisto.blueprint.chat_data_folder` used when running HITs.
+Run `analysis/compile_results.py` to compile and save statistics about collected human+model chats. The `ModelChatResultsCompiler` in that script uses dummy annotation buckets by default; set `--problem-buckets` in order to define your own. Set `--task-name` to specify a specific task to compile.
Original file line number	Diff line number	Diff line change
Expand Up		@@ -59,4 +59,4 @@ Note that onboarding is not currently supported with human+model image chat: use

		## Analysis

		Run `analysis/compile_results.py` to compile and save statistics about collected human+model chats. The `ModelChatResultsCompiler` in that script uses dummy annotation buckets by default; set `--problem-buckets` in order to define your own. Set `--results-folders` to the value of `mephisto.blueprint.chat_data_folder` used when running HITs.
		Run `analysis/compile_results.py` to compile and save statistics about collected human+model chats. The `ModelChatResultsCompiler` in that script uses dummy annotation buckets by default; set `--problem-buckets` in order to define your own. Set `--task-name` to specify a specific task to compile.