Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New metric module to improve flexibility and intuitiveness #475

Closed
wants to merge 16 commits into from

Conversation

sanchit97
Copy link
Contributor

@sanchit97 sanchit97 commented Jun 20, 2021

What does this PR do?

Summary

This PR adds a new metric module to Textattack, which provides an easier way to add new metrics or use existing metrics on adv. examples. Currently metrics are calculated in the attack_log_manager.py file, which is very unintuitive and non-flexible. Slides detailing this change are at: https://docs.google.com/presentation/d/1O77447GGWQF-xDnIdTr8d2MMPsQ9o9CG3pZZoo2bPrs/edit?usp=sharing

Additions

  • Added metrics module as textattack.attack_metrics.

Changes

  • attack_metrics is a new module, removing metrics from attack_log_manager.py

Deletions

  • **

Checklist

  • [ x ] The title of your pull request should be a summary of its contribution.
  • [ x ] Please write detailed description of what parts have been newly added and what parts have been modified. Please also explain why certain changes were made.
  • [ x ] If your pull request addresses an issue, please mention the issue number in the pull request description to make sure they are linked (and people consulting the issue know you are working on it)
  • [ x ] To indicate a work in progress please mark it as a draft on Github.
  • [ x ] Make sure existing tests pass.
  • [ x ] Add relevant tests. No quality testing = no merge.
  • [ ] All public methods must have informative docstrings that work nicely with sphinx. For new modules/files, please add/modify the appropriate .rst file in TextAttack/docs/apidoc.'

sanchit97 and others added 15 commits June 20, 2021 17:33

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
@sanchit97 sanchit97 changed the title New metrics module to ease usage of metrics on adversarial examples New metric module to improve flexibility and intuitiveness Jul 9, 2021
@sanchit97 sanchit97 marked this pull request as ready for review July 9, 2021 14:03
@sanchit97 sanchit97 requested a review from jinyongyoo July 9, 2021 14:04
Copy link
Collaborator

@jinyongyoo jinyongyoo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome start! Thank you for making this PR! I think it's good step towards cleaning up the messy calculations that happen within the log manager. Aside from the minor comments that I have made, the major question I have is about the way we/users interact withAttackMetric.

Currently, for each list of attack results, we need to instantiate a fresh AttackMetric object. Within each __init__ we call calculate that calculates the desirable metrics and then stores them as instance attributes (e.g. self.total_attacks). Then, we call various custom instance methods (e.g. successful_attacks_num, avg_num_queries_num) to retrieve the attributes.

I see two potential issues with this approach:

(1) The need to instantiate a different AttackMetric object for each different list of results.
Why do need to create a new AttackMetric object for any new list of attack results? Is AttackMetric's role to store the actual metric values for any arbitrary list of attack results, or is it to calculate the metric for a given list of results. If it is the latter, then I think there is no need to create a new instance. Instead, we can create one instance and then reuse it by passing a list of results to a method. This is similar to how our GoalFunction can be instantiated once and its get_results method can be used to produce GoalFunctionResults for list of AttackedText.

Side note: this is an interesting video that I found that is somewhat related to this. https://www.youtube.com/watch?v=o9pEzgHorH0

(2) Lack of common patterns between different AttackMetric classes.
We currently have three sub-classes of AttackMetric that share no common pattern. If I want to calculate the metric, I have to figure out which specific method I have to call (e.g. successful_attacks_num, avg_num_queries_num) to get the actual values. This will inevitably become more confusing as we add more metric classes.

Potential Solution (this is only a suggestion, so please feel free to come up with your own solution): Have calculate method of AttackMetric take in a list of AttackResult as an argument and make it the main method that people can use to calculate their metrics. If there are multiple metrics we want to return, we can simply return a dictionary of the values with appropriate keys.

"""A metric for evaluating Adversarial Attack candidates."""

@staticmethod
@abstractmethod
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think @staticmethod here is necessary since it's not really a static method.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sanchit97 please update regarding the comment..


@staticmethod
@abstractmethod
def calculate():
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same thing here. I don't think @staticmethod should be here (unless you intend to make it a static method). But in other sub-classes you define it as a regular instance method, so I'm guessing this should also be a regular method. Also in that case, you should have self as the first argument.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sanchit97 please update regarding the comment..

@@ -0,0 +1,9 @@
"""
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the difference between attack_metrics module and metrics module?

"""Calculates all metrics related to number of queries in an attack

Args:
results (:obj::`list`:class:`~textattack.goal_function_results.GoalFunctionResult`):
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I might be wrong, but can you check if this works a valid docstring for sphinx? I recall that references don't get nested. Ideally, we want something like list[GoalFunctionResult].

You can check by running sphinx-autobuild docs docs/_build/html --port 8765 --host 0.0.0.0 and then connecting to the host+port in your browser.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sanchit97 please update regarding the comment..

"""Calculates all metrics related to number of succesful, failed and skipped results in an attack

Args:
results (:obj::`list`:class:`~textattack.goal_function_results.GoalFunctionResult`):
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Similar thing here. Can you check if docstring works?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sanchit97 please update regarding the comment..

from textattack.attack_results import AttackResult


class AttackMetric(AttackResult, ABC):
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the reason why AttackMetric is a sub-class of AttackResult?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sanchit97 I have the same concerns as @jinyongyoo above

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this is an oversight. Fixed.

@sanchit97
Copy link
Contributor Author

sanchit97 commented Aug 12, 2021

My major objective to make different AttackMetric sub-classes was to group certain metrics together - e.g. words_perturbed only has metrics centered around word perturbations (like % of perturbed, max words changed, etc.).
I thought if we add a new metric, let's say Quality we can add a new class that deals with the fluency of adversaries (perplexity), grammar errors, etc. It's a general grouping of similar type of metrics.
However, your point is also true to simplify the process and add new functions and skip the class structure for new metrics as we go and not group metrics together.

sanchit97 pushed a commit that referenced this pull request Aug 20, 2021
@sanchit97 sanchit97 closed this Aug 20, 2021
qiyanjun added a commit that referenced this pull request Sep 29, 2021

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
New metric module to improve flexibility and  intuitiveness - moved from #475
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants