Skip to content
This repository has been archived by the owner on Nov 3, 2023. It is now read-only.

Bump Mephisto version for crowdsourcing checks #3652

Merged
merged 5 commits into from
May 24, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .circleci/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -167,7 +167,7 @@ commands:
command: |
cd ..
git clone git@github.com:facebookresearch/Mephisto.git Mephisto
cd Mephisto; git checkout v0.3.2 -b stable
cd Mephisto; git checkout v0.3.3 -b stable
pip install -r requirements.txt
python setup.py develop
# `echo` so that ENTER will be pressed at the prompt
Expand Down
2 changes: 1 addition & 1 deletion docs/source/tutorial_crowdsourcing.md
Original file line number Diff line number Diff line change
Expand Up @@ -210,7 +210,7 @@ Mephisto MTurk Tips and Tricks

### Preventing and Handling Crashes

- The `max_num_concurrent_units` argument when initializing [`TaskLauncher`](https://github.com/facebookresearch/Mephisto/blob/master/mephisto/operations/task_launcher.py) controls how many people can work on your task at any given time: set this sufficiently low for your task. Leaving this too high might cause your Heroku server to run into issues depending on how many messages per second it's trying to
- The `mephisto.task.max_num_concurrent_units` argument controls how many people can work on your task at any given time: set this sufficiently low for your task. Leaving this too high might cause your Heroku server to run into issues depending on how many messages per second it's trying to
process, and on how much data is being sent in those messages (such
as picture or video data).
- If you're running a model on your local machine, try to share the
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ mephisto:
task:
allowed_concurrent: 1
assignment_duration_in_seconds: 600
max_num_concurrent_units: 0 # 0 means infinite; set this to a positive integer to limit concurrent HITs and prevent crashes
maximum_units_per_worker: 1 # important for replicability and reliability
task_description: "Evaluate quality of conversations through comparison."
task_name: acute_eval_${current_time}
Expand Down
1 change: 1 addition & 0 deletions parlai/crowdsourcing/tasks/acute_eval/conf/example.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ mephisto:
task:
allowed_concurrent: 1
assignment_duration_in_seconds: 600
max_num_concurrent_units: 0 # 0 means infinite; set this to a positive integer to limit concurrent HITs and prevent crashes
maximum_units_per_worker: 1 # important for replicability and reliability
task_description: "Evaluate quality of conversations through comparison."
task_name: acute_eval_${current_time}
Expand Down
2 changes: 2 additions & 0 deletions parlai/crowdsourcing/tasks/model_chat/conf/example.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,8 @@ mephisto:
conversations_needed_string: "blender_90M:10"
task:
allowed_concurrent: 1
assignment_duration_in_seconds: 600
max_num_concurrent_units: 0 # 0 means infinite; set this to a positive integer to limit concurrent HITs and prevent crashes
maximum_units_per_worker: 3
task_description: |
<b><h4>Task Description</h4></b>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,8 @@ mephisto:
num_conversations: 10
task:
allowed_concurrent: 1
assignment_duration_in_seconds: 600
max_num_concurrent_units: 0 # 0 means infinite; set this to a positive integer to limit concurrent HITs and prevent crashes
maximum_units_per_worker: 3
task_description: |
<b><h4>Task Description</h4></b>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,10 @@ mephisto:
custom_source_bundle: ${task_dir}/webapp/build/bundle.js
num_conversations: 1
task:
allowed_concurrent: 1
assignment_duration_in_seconds: 600
max_num_concurrent_units: 0 # 0 means infinite; set this to a positive integer to limit concurrent HITs and prevent crashes
maximum_units_per_worker: 3
task_name: parlai-qa-example
task_title: "Test ParlAI QA Data Collection Task"
task_description: >
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,8 @@ mephisto:
response_field: False
task:
allowed_concurrent: 1
assignment_duration_in_seconds: 600
max_num_concurrent_units: 0 # 0 means infinite; set this to a positive integer to limit concurrent HITs and prevent crashes
maximum_units_per_worker: 5
task_description: |
<b><h4>Task Description</h4></b>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,8 @@ mephisto:
onboarding_qualification: turn_annotations_static
task:
allowed_concurrent: 1
assignment_duration_in_seconds: 600
max_num_concurrent_units: 0 # 0 means infinite; set this to a positive integer to limit concurrent HITs and prevent crashes
maximum_units_per_worker: 5
task_description: |
<b><h4>Task Description</h4></b>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ def test_base_task(self):
# Set up the config and database
overrides = [
'mephisto.blueprint.num_conversations=1',
'+mephisto.task.allowed_concurrent=0',
'mephisto.task.allowed_concurrent=0',
'+turn_timeout=300',
]
# TODO: remove all of these params once Hydra 1.1 is released with
Expand Down
3 changes: 3 additions & 0 deletions tests/crowdsourcing/tasks/test_chat_demo.py
Original file line number Diff line number Diff line change
Expand Up @@ -382,6 +382,9 @@ def test_base_task(self):
'+mephisto.blueprint.task_description_file=${task_dir}/task_description.html',
'+mephisto.blueprint.num_conversations=1',
'+mephisto.task.allowed_concurrent=0',
'+mephisto.task.assignment_duration_in_seconds=600',
'+mephisto.task.max_num_concurrent_units=0',
'+mephisto.task.maximum_units_per_worker=0',
'+num_turns=3',
'+turn_timeout=300',
]
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -142,7 +142,7 @@ def _test_turn_annotations_static_task(
'+mephisto.blueprint.conversation_count=null',
'mephisto.blueprint.onboarding_qualification=null',
'+mephisto.blueprint.random_seed=42',
'+mephisto.task.assignment_duration_in_seconds=1800',
'mephisto.task.assignment_duration_in_seconds=1800',
]
# TODO: remove all of these params once Hydra 1.1 is released with support
# for recursive defaults
Expand Down