Move code for Mephisto ACUTE-Evals into ParlAI #3002
Conversation
| help="Number of pairs per model matchup, default 2", | ||
| type=int, | ||
| ) | ||
| parser.add_argument( |
There was a problem hiding this comment.
do we need those flags as they are already included in the blueprint?
There was a problem hiding this comment.
Having them here lets you set them from the command line. Hoping to make mephisto do this kind of thing automatically, but for now the transfer is manual.
| # Code Instructions | ||
| Once you have installed [ParlAI](https://github.com/facebookresearch/ParlAI/#installing-parlai) and [Mephisto](https://github.com/facebookresearch/mephisto/blob/master/docs/quickstart.md), follow the instructions below. | ||
|
|
||
| The `run.py` script is designed to allow you to run this entire task from command line with an invocation like |
There was a problem hiding this comment.
It may make sense to rename example_script.py, or update this comment.
There was a problem hiding this comment.
Whoops - yes, changing
| # group.add_argument( | ||
| # '--task-config', | ||
| # type=dict, | ||
| # default=DEFAULT_TASK_CONFIG, | ||
| # help='dict with keys "hit_title", "hit_description", "hit_keywords", ' | ||
| # 'determining how task is displayed on MTurk site', | ||
| # ) |
| # group.add_argument( | ||
| # '--softblock-list-path', | ||
| # dest="softblock_list_path", | ||
| # type=str, | ||
| # default=None, | ||
| # help='Path to list of workers to softblock, separated by line breaks', | ||
| # ) |
| # TODO(#98) this should be a util in a provider, not here | ||
| def softblock_workers(self): | ||
| """ | ||
| Softblock workers if necessary. | ||
| """ | ||
| if not self.opts["is_sandbox"] and self.opts["softblock_list_path"] is not None: | ||
| softblock_list = set() | ||
| with open(self.opts["softblock_list_path"]) as f: | ||
| for line in f: | ||
| softblock_list.add(line.strip()) | ||
| logger.info(f"Will softblock {len(softblock_list):d} workers.") | ||
| for w in softblock_list: | ||
| try: | ||
| logger.info(f"Soft Blocking {w}\n") | ||
| self.manager.soft_block_worker(w) | ||
| except Exception as e: | ||
| logger.exception( | ||
| f"Did not soft block worker {w}: {e}", exc_info=True | ||
| ) | ||
| time.sleep(0.1) |
There was a problem hiding this comment.
Nit: This is dead code, can be removed.
| help="Number of pairs per model matchup, default 2", | ||
| type=int, | ||
| ) | ||
| parser.add_argument( |
There was a problem hiding this comment.
Having them here lets you set them from the command line. Hoping to make mephisto do this kind of thing automatically, but for now the transfer is manual.
| @@ -0,0 +1,16 @@ | |||
| ## Description | |||
There was a problem hiding this comment.
Nit: remove this readme, it is outdated.
| @@ -0,0 +1,5 @@ | |||
| #!/usr/bin/env python3 | |||
There was a problem hiding this comment.
Nit: no need for init in webapp directory (especially as you have modified the test already).
Patch description
Moving the existing ACUTE-Eval Mephisto code into ParlAI
(Note: changed
tests/test_code.pyto excludewebappfolders, containing Javascript and no Python code)Testing steps