Add HuffPost

[KhalidAlt]

No description provided.

[jon-tow]

Hey, @KhalidAlt . Is there any chance you could add all the categories (listed here) to the answer_choices fields so that rank/accuracy scoring could easily be performed? Thanks!

[KhalidAlt]

I add choices_in_prompt as @jon-tow suggested. However, the categories are listed as string instead of labels and I can not use {{answer_choice[category]}} as suggested in the prompt guide. Therefore, I just wrote {{category}} as the target. Does it still work correctly with rank/accuracy regardless of this issue or should I edit the dataset to include categories as labels instead of just string.

[thinkzink]

I think the following could be reworded to make the description more separable from the surrounding text but I think the others look fine: 'Given the following article headline: "{{headline}}" and the following
short passage "{{short_description}}" What is the category of this news report?
||| {{category}}'

For consistency it would be important to edit the dataset for that {{answer_choice[category]}} ability in including categories as labels.

[cjlovering]

@thinkzink @KhalidAlt It looks like the changes are approved. Should this be merged?

[KhalidAlt]

@thinkzink @KhalidAlt It looks like the changes are approved. Should this be merged?

Yes, I think so.

[awebson]

Thanks for the PR! A few issues left:

1. None of your prompts have the answer choices in the input, so (a) models or even humans will never know to answer "impact" or "divorce" as a valid news category. (b) This may be mitigated by rank classification of the answer choices, but it's far from guaranteed. At the very lest, you need to set the choices_in_prompts accordingly.
2. Please lowercase or title case your answer choices. Most models are case-sensitive now and are a priori unlikely to generate all caps vocabulary.
3. The classes are imbalanced, so you probably don't really want to report naive accuracy. Fine for now, but please pay attention in you evaluation.
4. What is the file change of promptsource/templates/tydiqa/primary_task/.templates.yaml.swp?

[awebson]

This is ungrammatical. How about "Given these headline and description, what category can you infer this piece of news belongs to?"

[KhalidAlt]

Thanks for the PR! A few issues left:

1. None of your prompts have the answer choices in the input, so (a) models or even humans will never know to answer "impact" or "divorce" as a valid news category. (b) This may be mitigated by rank classification of the answer choices, but it's far from guaranteed. At the very lest, you need to set the `choices_in_prompts` accordingly.

2. Please lowercase or title case your answer choices. Most models are case-sensitive now and are a priori unlikely to generate all caps vocabulary.

3. The classes are imbalanced, so you probably don't really want to report naive accuracy. Fine for now, but please pay attention in you evaluation.

4. What is the file change of `promptsource/templates/tydiqa/primary_task/.templates.yaml.swp`?

1- I can add answer choices as list in the input as follow:

Given the following news headline and short description: {{headline}} {{short_description}} What is a correct category class from the following list {{answer_choices}} ? ||| {{answer_choices[label]}}

However, It may seem unnatural to use this style with humans. Also, I set up choices_in_prompt to false.

2- Fixed. I titled case all answer choices.

3- I will pay attention for this in the evaluation code. Thank you!

4- Fixed. It was a mistake and I removed the file.

[awebson]

Thank you!

I can add answer choices as list in the input as follow:

Given the following news headline and short description: {{headline}} {{short_description}} What is a correct category class from the following list {{answer_choices}} ? ||| {{answer_choices[label]}}

However, It may seem unnatural to use this style with humans.

I think it's pretty natural to ask people and models something like

choose from the list of options below:

• politics
• sports
• ...

But I will leave it up to you. In our experience, including the answer choices could either make a huge difference or nothing at all. You should definitely try it both ways in your evaluation. Let me know if you want me to merge this now or wait until you include some prompts with answer choices.

[KhalidAlt]

Now it is done! I added answer_choices in 3 of the prompts and kept two without any answer_choices in the prompts. Please review last commits and merge it if everything is OK.

[awebson]

Beautiful. Thanks so much!