Move defaults to config yaml files for crowdsourcing tasks #4095
Conversation
* add support for registering and onboarding agent * add a value for onboarding qualification instead of null to test * add parameter assume_onboarding and fix spelling
…ch/ParlAI into move-defaults-acute-evals
|
Everything should be in order now, I think it's failing the one crowdsourcing TestChatDemo on CI because the CI config for ParlAI uses Mephisto v0.4.0. I merged the corresponding mephisto PR so locally when I'm on the latest Mephisto main commit and on this branch the crowdsourcing tests pass (including TestChatDemo). What's the proper or preferred procedure so we can merge this PR and also have the test pass? |
meganung
changed the title
|
I just released a |
There was a problem hiding this comment.
Oh yeah this is great, thanks for doing this :) Would you be able to add in similar changes in parlai/crowdsourcing/projects/wizard_of_internet/run.py and parlai/crowdsourcing/projects/multisession_chat/human_eval/run.py as well, so that the regression in Hydra 1.1 defaulting doesn't show up there either? Also, I have a few minor comments
Moved configs from run scripts to yaml config files for the crowdsourcing tasks.
Checked that the configs set are respected by running the run scripts with the flag --cfg job and inspecting the defaults list by running the run scripts with the flag --info defaults
crowdsourcing tests pass except for:test_chat_demo which will pass when the corresponding PR in Mephisto is mergeddone- pointed to new mephisto release tagtest_turn_annotations_static tests - failing because need to add onboarding testing (in a separate PR and then rebase this branch ontop of it). Config changes make the onboarding_qualification parameter to be non optional so cannot be null.this is done.