-
Notifications
You must be signed in to change notification settings - Fork 2.1k
Expose Precision and Recall to user via TeacherMetrics #4670
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is cool but I think it will be slow from recomputing these values?
The way this function is setup, extracting |
Or I can make "f1" default. If the user wants, they can print |
@stephenroller Can you please provide what you think based on the reply ? |
Sorry; compute and return all 3 metrics in a generalization of the existing utility method. The caller will be responsible for separating and accumulating them. |
@stephenroller Is it fine now ? We only compute |
@klshuster Can you please take a look ? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this looks great. could you please update the metrics master list (https://github.com/facebookresearch/ParlAI/blob/main/parlai/core/metrics.py#L53) to have the new metrics, and also please rename 'precision' and 'recall' to refer to the fact that they are computed based on unigrams (e.g., uni_prec
)
Patch description
Considering that other metrics such as
ctpb
,ctrunc
,ctrunclen
(less frequently used by user) are exposed, it makes a lot of sense to exposeprecision
andrecall
as they are internally being computed.