Skip to content
This repository was archived by the owner on Nov 3, 2023. It is now read-only.

[Crowdsourcing] edits for OSS personal knowledge project #3945

Merged
merged 14 commits into from
Aug 18, 2021
Merged

Conversation

jxmsML
Copy link
Contributor

@jxmsML jxmsML commented Aug 16, 2021

Patch description

  • Add ContextGenerator based on the task_name = 'msc'
  • Flexibility to build ContextGenerator for any given task:
    • Modify conversation_start_mode to be either 'hi', or a 'task name' (e.g. 'blended_skill_talk', or 'msc'
    • get_context_generator in parlai/crowdsourcing/tasks/model_chat/utils.py will load ContextGenerator from the <task_name>/agents.py specified by conversation_start_mode
  • Add world_module as an additional parameter for run_task in parlai/crowdsourcing/tasks/model_chat/run.py
  • In parlai/crowdsourcing/tasks/model_chat/worlds.py, display fancy bot name with self.bot.agent_id rather than the default agent name from the 'id' field (mostly TransformerGenerator)
  • Move "validate problem buckets" from AbstractTurnAnnotationResultsCompiler to ModelChatResultsCompiler init method.
if 'none_all_good' not in self.problem_buckets:
        # The code relies on a catchall "none" category if the user selects no other
         # annotation bucket
        raise ValueError(
       'There must be a "none_all_good" category in self.problem_buckets!'
         )

Testing steps
CI

Other information

@@ -63,6 +63,13 @@ def setup_args(cls):
def __init__(self, opt: Dict[str, Any]):

super().__init__(opt)
# Validate problem buckets
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

move out of parlai/crowdsourcing/utils/analysis.py here as it could be that "none_all_good" isn't necessary in problem_buckets.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm, okay, on closer look, I'm fine with moving this check into the subclasses, because the logic that relies upon "none_all_good" being present is itself in the subclasses



def run_task(cfg: DictConfig, task_directory: str):
def run_task(cfg: DictConfig, task_directory: str, world_module=None):
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

adding world_module here so it doesn't have to call the model_chat.worlds by default but any world_module as long as it's consistent with cfg.mephisto.blueprint.world_file.

Copy link
Contributor

@EricMichaelSmith EricMichaelSmith left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, good to have - thanks for generalizing run_task, etc.! I have a few nitpicky questions, and in particular, before approving, it might be good to decide where the 'none_all_good' check should go

@jxmsML jxmsML merged commit 4aba382 into master Aug 18, 2021
@jxmsML jxmsML deleted the pk-eval-oss branch August 18, 2021 20:23
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants