-
Notifications
You must be signed in to change notification settings - Fork 2.1k
Added DST Teacher for multiwoz_v22 task #4656
Conversation
@@ -373,6 +379,212 @@ def get_id_task_prefix(self): | |||
return "MultiwozV22" | |||
|
|||
|
|||
class MultiWOZv22DSTTeacher(tod_agents.TodUserSimulatorTeacher): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
might be better to add a comment saying this teacher is needed for reproducing the Joint Goal Accuracy values reported in the simpleTOD & SOLOIST papers.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is it better now ? @chinnadhurai
Looks okay code-wise, though I haven't done a through review. Should this be in ParlAI-Internal rather than public ParlAI though? |
parlai/tasks/multiwoz_v22/agents.py
Outdated
@@ -393,5 +605,5 @@ class SingleApiSchemaAgent(MultiwozV22Parser, tod_agents.TodSingleApiSchemaAgent | |||
pass | |||
|
|||
|
|||
class DefaultTeacher(SystemTeacher): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
would be better to keep the default to SystemTeacher as is.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed this.
we need this teacher for the actual DST task in Multiwoz and reproduce JGA reported in the SOTA models. So, figured it might make sense to push to the public one. |
@stephenroller Can you please look into this ? It has been here for sometime now. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'll defer to @moyapchen ans @klshuster here. Don't see any issue though
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
deferring to @moyapchen for final say but looks OK to me
could we generate test files for it as well?
@moyapchen Can you please see this ? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
lgtm
The DST Teacher for
multiwoz_v22
is not present. Adding this teacher would allowmultiwoz_v22
to be usable for Dialog State Tracking task.cc @chinnadhurai