Skip to content
This repository was archived by the owner on Nov 3, 2023. It is now read-only.

WoW Custom Metrics #3503

Merged
merged 1 commit into from
Mar 10, 2021
Merged

WoW Custom Metrics #3503

merged 1 commit into from
Mar 10, 2021

Conversation

klshuster
Copy link
Contributor

Patch description
Add a few custom evaluations for wizard of wikipedia:

  1. When doing the knowledge selection task, the teacher will compute additional title/checked sentence recall metrics; the latter measures whether the checked sentence appears in the text of any of the returned candidates
  2. When doing the response generation task, the teacher will compute f1 score of the generation w.r.t. the checked sentence

We skip the other test now because... it's not super necessary

Testing steps
Added tests to CI.

$ pytest tests/tasks/test_wizard_of_wikipedia.py
=====test session starts =====
platform linux -- Python 3.7.9, pytest-6.2.1, py-1.10.0, pluggy-1.0.0.dev0
rootdir: /private/home/kshuster/ParlAI, configfile: pytest.ini
plugins: hydra-core-1.0.0, requests-mock-1.8.0, regressions-2.1.1, datadir-1.3.1
collected 2 items

tests/tasks/test_wizard_of_wikipedia.py ..                                                                                                                                                                                                                                                                                                                           [100%]

=====slowest 10 durations =====
1.81s call     tests/tasks/test_wizard_of_wikipedia.py::TestWoW::test_custom_eval

(5 durations < 0.005s hidden.  Use -vv to show these durations.)
====2 passed in 3.65s =====

Copy link
Contributor

@stephenroller stephenroller left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Whooo love when people use custom_eval

@klshuster klshuster merged commit 38fce64 into master Mar 10, 2021
@klshuster klshuster deleted the wiz_metrics branch March 10, 2021 15:33
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants