Unexpected delay and retry behavior

[ku1ik]

When you enqueue 2 delayed jobs like this:

HelloJob.enqueue(%{a: 1}, delay: 10_000)
HelloJob.enqueue(%{a: 2}, delay: 1_000)

then both jobs are executed 10 seconds later.

This seems to come from the RabbitMQ message expiration behavior, where only the message in the front of the queue is considered for expiry.

Given that most people don't look at implementation details (of both Task Bunny and RabbitMQ), I believe this shortcomming should be explicitly mentioned in the README, and docstring of enqueue/2 function.

Related thing is, the same RMQ behavior causes similar unexpected behavior for job retries when retry_interval/1 callback is implemented to return non static interval (like with backoff example in the README) - when there's retry with long backoff (1 hour) at the front of retry queue and TB enques 1st retry of another job with short backoff (1 minute), then both jobs will be retried after 1 hour - seems like a major issue with retry, breaking all sorts of workflows.

The only way to mitigate this right now is to use the same, single delay value for jobs enqueued to a given queue, never use dynamic backoff, and use the same backoff value for all (even different) jobs that are handled by the same queue.

I don't see a simple solution to this though. If we can't easily fix dynamic backoff issue then I'd suggest to change retry_interval/1 to retry_interval/0, in order to not confuse users.