Do we need a timeout mechanism?

[wenh06]

Is your feature request related to a problem? Please describe.
Some attack recipes are extremely slow, so a timeout mechanism for a single attack on one sample is needed.

Describe the solution you'd like
Use a timeout context manager like

@contextmanager
def timeout(duration: float):
    """
    A context manager that raises a `TimeoutError` after a specified time.

    Parameters
    ----------
    duration: float,
        the time duration in seconds,
        should be non-negative,
        0 for no timeout

    References
    ----------
    https://stackoverflow.com/questions/492519/timeout-on-a-function-call

    """
    if np.isinf(duration):
        duration = 0
    elif duration < 0:
        raise ValueError("duration must be non-negative")
    elif duration > 0:  # granularity is 1 second, so round up
        duration = max(1, int(duration))

    def timeout_handler(signum, frame):
        raise TimeoutError(f"block timedout after {duration} seconds")

    signal.signal(signal.SIGALRM, timeout_handler)
    signal.alarm(duration)
    yield
    signal.alarm(0)

and replace this line

result = self._attack(goal_function_result)

with, for example,

try:
    with timeout(duration=time_out):
        result = self._attack(goal_function_result)
except TimeoutError:
    result = FailedAttackResult(goal_function_result, goal_function_result)
except Exception as e:
    if ignore_errors:
        # print(e)
        result = FailedAttackResult(
            goal_function_result, goal_function_result
        )
    else:
        raise e

One can also add a time-out argument in commands.textattack_cli, and pass this argument in related classes and functions.

Describe alternatives you've considered
NA

Additional context
NA

[jxmorris12]

I think this is a great idea @wenh06. The attack search process is exponential in nature so it's easy to get into a state where the attack will never finish in a reasonable amount of time. It could also be nice to try and predict how long the attack would take, and warn the user if they're running something that is very very slow.

[jxmorris12]

Just thinking about this a little more:

• There are lots of ways to make functions time out in Python. Maybe asyncio is nicest, but it might cause problems – like weird error messages when things break – if all the important code is running asynchronously.
• We should probably add a new type of attack result, like TimedOutAttackResult or ExpiredAttackResult or something like that, and show it in the results (X skipped, Y succeeded, Z failed, and M timed out) if users specify a timeout.
  • Alternatively maybe we could just consider a timeout a failure (this is what query budget does) but that could be confusing, since it wouldn't be clear how something failed. (Though maybe it's confusing for query budget too..)

[kenneth-ge]

Bump on this! I really like this, and have had to implement a custom timeout feature in my own production code, so this would definitely be really useful. The issue is that most attacks finish in just a couple minutes, but some small proportion of attacks take more than 2 days to finish, so having a timeout mechanism allows us to finish most of our attacks in a reasonable amount of time