This repository has been archived by the owner on Nov 3, 2023. It is now read-only.
Add onboarding to crowdsourcing tests #4096
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
meganung
commented
Oct 15, 2021
| @@ -215,7 +273,10 @@ def _get_agent_state(self, task_data: Dict[str, Any]): | |||
| """ | |||
|
|
|||
| # Set up the mock human agent | |||
| agent_id = self._register_mock_agents(num_agents=1)[0] | |||
| if self.config.mephisto.blueprint.get("onboarding_qualification", None): | |||
There was a problem hiding this comment.
is there a better way to check if we should test onboarding? Would it be better to pass an onboarding bool variable?
There was a problem hiding this comment.
At the moment this is how we specify onboarding in Mephisto:
https://github.com/facebookresearch/Mephisto/blob/main/mephisto/abstractions/blueprints/mixins/onboarding_required.py#L103-L107
So I think this is really the only way to check that a config is trying to use onboarding.
| @@ -141,14 +141,13 @@ def _test_turn_annotations_static_task( | |||
| overrides += [ | |||
| '+mephisto.blueprint.annotation_last_only=False', | |||
| '+mephisto.blueprint.conversation_count=null', | |||
| 'mephisto.blueprint.onboarding_qualification=null', | |||
| 'mephisto.blueprint.onboarding_qualification=test_turn_annoataions', | |||
EricMichaelSmith
approved these changes
Oct 19, 2021
meganung
added a commit
that referenced
this pull request
Oct 25, 2021
* move defaults for fast acute * add _self_ and move defaults for acute eval * udpated config overrides in test * move defaults for model chat and update test * move defaults for model image chat and update test * add hydra_configs to config path and update test * move defaults and add self for qa data collection task * move defaults to yaml, add self, and update tests * onboarding qualification arg * Fixed Some Typos (#4100) * Add onboarding to crowdsourcing tests (#4096) * add support for registering and onboarding agent * add a value for onboarding qualification instead of null to test * add parameter assume_onboarding and fix spelling * move defaults for fast acute * add _self_ and move defaults for acute eval * udpated config overrides in test * move defaults for model chat and update test * move defaults for model image chat and update test * add hydra_configs to config path and update test * move defaults and add self for qa data collection task * move defaults to yaml, add self, and update tests * onboarding qualification arg * rebase * rebase more edits and cleanup * linting * placeholder to remove unused import error * move defaults for tasks in crowdsourcing projects * point to new mephisto release for circle ci Co-authored-by: Atharv jairath <54663702+atharvjairath@users.noreply.github.com>
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Added support for onboarding in crowdsourcing tests. Used this example in the Mephisto repo as a guide.
CI crowdsourcing tests.
(will rebase this PR after)