Mephisto QA data collection

[vaibhavad]

Description
Example of QA Data collection task using Mephisto

Testing steps

cd webapp
npm install; npm run dev
cd ..
python run.py

Discussion
There are two major points where this can be improved -

1. Dataloader - How to seamlessly integrate the QA data loader to re-use existing ParlAI tasks? The two requirements I can think of are - (1) User should be able to specify task from config, (2) There should be a wrapper on top of original ParlAI task to modify small things (SQuAD returns both Passage and Question, the task may require only passage)
2. Passage highlighting - Many QA data collection tasks require evidence from the passage. It can be made configurable. There should be a div on top of Worker input to show the span selected from the passage. This will require modifications in the bootstrap-chat package.

@EricMichaelSmith Please review it and suggest changes.

[EricMichaelSmith]

Wow, I love this - thanks @vaibhavad for adding this! It'll be a great addition to our set of Mephisto crowdsourcing tasks. I've added a bunch of comments, mostly relating to standard ParlAI and parlai.crowdsourcing conventions, but it looks like the major components are all here. To address your points below:

1. Yeah, there's a standard ParlAI way to load data - see my comments on this, but happy to explain further if you'd like.
2. I think showing evidence would be a great add-on here: I'll defer to @JackUrb and @pringshia for advice on configuring the front-end code for this

Great job!

[JackUrb]

Hi @vaibhavad, thanks for bringing this together! Definitely agreed that adding more context to a task such that it's not all inline in the chat would be great. In bootstrap-chat we support overriding with custom components within a chat application by providing a file containing those custom components. See how renderSidePane is updated in the custom_simple version of the ParlAI Chat demo here. We even have setups where the context is always updating, like what's present in the PR for multimodal chat here.

While it'd be a great enhancement for sure, I don't think it should stop this from getting merged as is, as this was the level of usability we had for the original task. Can always open a follow-up PR if you want to contribute more!

[vaibhavad]

Hi @JackUrb,

Thanks a lot for your comments. I'll explore overriding components in bootstrap-chat, and include evidence marking for QA tasks in a follow-up PR as you suggested.

[EricMichaelSmith]

Thanks so much for all of the refactors on this! Approving now. It looks like Jack is out of the office until next week, but it seems like he agrees that the remaining improvements can be done in a follow-up PR, so it's fine with me if you merge in :)

[vaibhavad]

Thanks @EricMichaelSmith for all the constructive feedback throughout. Feels good to have my first contribution.

I don't think I have permission to merge. You or @JackUrb can merge it if no more changes are required.

[EricMichaelSmith]

Thanks @EricMichaelSmith for all the constructive feedback throughout. Feels good to have my first contribution.

I don't think I have permission to merge. You or @JackUrb can merge it if no more changes are required.

Okay cool! It looks like some of the CI checks are failing (linting and stuff) - this is pretty minor, but when I have time today I'll fix those up before merging in (unless you want to take a crack at them first, totally up to you)