Skip to content
This repository has been archived by the owner on Nov 3, 2023. It is now read-only.

Crowdsourcing data compiler class for using Mephisto abstractions. #4029

Merged
merged 7 commits into from
Sep 21, 2021

Conversation

mojtaba-komeili
Copy link
Contributor

Patch description

  • Cleaned up analysis.py module in ParlAI crowdsourcing: moved logic from AbstractDataBrowserResultsCompiler to AbstractResultsCompiler.
  • Added a readme entry that explains how to extend AbstractResultsCompiler for compiling a dataset from a Mephisto task.
  • Added new unit tests with mock versions of MephistoDB and DataBrowser.

Testing steps

  • The newly added unit tests.
  • Compiling a subset of Wizard of Internet data task with a new class extended from the AbstractResultsCompiler (see PS 2).

PS 1. kept AbstractTurnAnnotationResultsCompiler for now for not breaking any projects.
PS 2. I made a cleaned up version of wizard of internet data compiler that is based on this new AbstractResultsCompiler. The next step after landing this would be to add that.

Copy link
Contributor

@EricMichaelSmith EricMichaelSmith left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wow, really great to have this - thanks for pushing the PR out on this! The MockResultsCompiler in particular will be super valuable. I have a few nits + questions

parlai/crowdsourcing/utils/analysis.py Show resolved Hide resolved
parlai/crowdsourcing/utils/README.md Outdated Show resolved Hide resolved
parlai/crowdsourcing/utils/README.md Outdated Show resolved Hide resolved
parlai/crowdsourcing/utils/README.md Outdated Show resolved Hide resolved
parlai/crowdsourcing/utils/README.md Outdated Show resolved Hide resolved
parlai/crowdsourcing/utils/analysis.py Show resolved Hide resolved
parlai/crowdsourcing/utils/analysis.py Outdated Show resolved Hide resolved
parlai/crowdsourcing/utils/analysis.py Show resolved Hide resolved
tests/crowdsourcing/test_analysis.py Show resolved Hide resolved
tests/crowdsourcing/test_analysis.py Show resolved Hide resolved
@JackUrb
Copy link
Contributor

JackUrb commented Sep 20, 2021

The integrations with the Mephisto side make sense to me here, though Eric has a number of good questions and points still

parlai/crowdsourcing/utils/analysis.py Outdated Show resolved Hide resolved
parlai/crowdsourcing/utils/analysis.py Outdated Show resolved Hide resolved
tests/crowdsourcing/test_analysis.py Show resolved Hide resolved
@mojtaba-komeili mojtaba-komeili merged commit b2db317 into main Sep 21, 2021
@mojtaba-komeili mojtaba-komeili deleted the mephisto-data-browser branch September 21, 2021 22:23
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants