This repository has been archived by the owner on Nov 3, 2023. It is now read-only.
[Crowdsourcing] can call blueprints from command line #4254
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
There was a problem hiding this comment.
Oh this is fantastic, especially considering that you were able to remove the separate run script for ModelImageChat as well! But I wanted to check - looking at the CircleCI log for crowdsourcing_tests, I don't see an entry for test_model_image_chat anymore - do you know why that is?
| @@ -6,7 +6,7 @@ Code for crowdsourcing tasks that use Mephisto. See the [Mephisto quick start gu | |||
|
|
|||
| ## Running tasks | |||
|
|
|||
| Tasks are launched by calling the appropriate run script: for instance, an ACUTE-Eval run can be launched with `python parlai/crowdsourcing/tasks/acute_eval/run.py`, followed by any appropriate flags. All run parameters are set using [Hydra](https://github.com/facebookresearch/hydra): append the flag `-c job` to your run command to see a list of all available parameters, grouped by their package name (`mephisto.blueprint`, `mephisto.task`, etc.), which determines how they are called. Each run script has a YAML file of default parameters that will be loaded, found in the `hydra_configs/conf/` subfolder of each task. | |||
| Tasks are launched by calling the appropriate run script: for instance, an ACUTE-Eval run can be launched with `python parlai/crowdsourcing/tasks/acute_eval/run.py`, followed by any appropriate flags. All run parameters are set using [Hydra](https://github.com/facebookresearch/hydra): append the flag `-c job` to your run command to see a list of all available parameters, grouped by their package name (`mephisto.blueprint`, `mephisto.task`, etc.), which determines how they are called. Each run script points to a YAML file of default parameters (including the blueprints) that will be loaded, found in the `hydra_configs/conf/` subfolder of each task. You can specify a different blueprint in a YAML file and use that YAML file to run your task (see below). | |||
| @@ -6,7 +6,7 @@ This task will collect conversations between a human and a model. After each res | |||
|
|
|||
| ## Launching | |||
|
|
|||
| Call `run.py` to run this task with the default parameters, as set by `conf/example.yaml`. Some parameters that you can adjust include where to save data, lists of workers to soft-block, the maximum response time, etc. | |||
| Call `run.py` to run this task with the default parameters, as set by `hydra_configs/conf/example.yaml`. Some parameters that you can adjust include where to save data, lists of workers to soft-block, the maximum response time, etc. | |||
| @@ -26,7 +26,7 @@ In `worlds.py`, modify `ModelChatOnboardWorld.check_onboarding_answers()` to cha | |||
|
|
|||
| ## Human+model image chat | |||
|
|
|||
| `run_image_chat.py` can be run to chat with a model about an image: each conversation will begin with a selected image, and then the human and model will chat about it. | |||
| Call `run.py conf=example_image_chat` to chat with a model about an image: each conversation will begin with a selected image, and then the human and model will chat about it. This task is run with the parameters defined in `hydra_configs/conf/example_image_chat.yaml`. | |||
EricMichaelSmith
approved these changes
Dec 15, 2021
|
Big thanks for catching that!! 👏 |
mojtaba-komeili
added a commit
that referenced
this pull request
Dec 16, 2021
* ConversationTeacher parent class changed * reformat * [Crowdsourcing] can call blueprints from command line (#4254) * remove unnecessary run scripts * udpate readmes * BB2 handling increase in batch size during dynamic batching (#4238) * ConversationTeacher parent class changed * reformat * changed the teacher, tests are passing * removing 'episode_done' if it exists in the example Co-authored-by: Megan Ung <20617868+meganung@users.noreply.github.com>
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Patch description
After hydra upgrade, blueprints can now be specified in yaml files so we don’t need individual run scripts for each blueprint.
parlai/crowdsourcing/tasks/turn_annotations_static/run_in_flight_qa.pyandparlai/crowdsourcing/tasks/model_chat/run_image_chat.pyUpdated READMEs of those tasks with 2 blueprints (turn_annotations_static and model_chat) and also the general crowdsourcing README since we can specify a blueprint from the command line.
Testing steps
Ran the crowdsourcing jobs with the single run.py script with the config specified with the non default blueprint.