Processing difference between Celery and Dramatiq

[amureki]

Greetings @codingjoe ,

while using the package I realized that failing processing tasks (in my case, it was the deleted file, so nothing to process) are being retried several times.
This is because in Dramatiq (the queue I am using) the default is leading to retrying the failing task 20 times:
https://dramatiq.io/guide.html#message-retries (with the max backoff of 7 days, which leads to task being executed for quite a long time).

Celery has a different strategy and does not do autoretries unless you explicitly specify it:
https://docs.celeryq.dev/en/stable/userguide/tasks.html#automatic-retry-for-known-exceptions

My main concern here is not the inconsistency in the behaviour (which might be difficult to maintain between different tools), but rather the suboptimal process when using dramatiq here.

I am expecting the dramatiq not to retry such errors at all (or not more than 3 times, for example).

Do you see this as a valid concern? How'd you suggest solving it?

Cheers,
Rust

[codingjoe]

Interesting thought. I was looking from in from a task system angle. From this angle, the defaults are consistent (except for the queue name, which confuses people sometimes).
I think which path to choose gravely depends on the question, if image processing and the way this package does it, has implication on the likelihood of a task to fail. Or the resilience of the image pipeline, regardless of your queue system.
Currently, I am not aware of such implications. We only process pictures once the DB row is committed and the files should be available.
The only race condition I know if a user would upload a new image, before the previous task has succeeded. This can create a bit of white noise, which certainly could be reduced.

[amureki]

Thanks, Johannes,

I agree that from the perspective of task systems this would be consistent, so probably it is better to keep the defaults intact.
But nevertheless, I'd love to find a solution to a technically possible error.
In my latest case (there were more, I remember, but would need to dig the history), user uploaded the image and then changed it to another one, while the queue did not manage to process the first one (for example, it was busy with other processing tasks). So exactly the case you mentioned.

And this event creates a scattered noise over the time period of days (maybe months) according to Dramatiq defaults.
The only solution I see now - to get to the redis/rabbitmq and manually dig and remove the pending scheduled tasks, which is a tedious operation.

[codingjoe]

Hm… since the files are there, before we submit the task, we could fail if the file is not present and NOT retry. Regarding the warning on sentry, I would suggest simply ignoring it up to a certain threshold.

Please don't hesitate to provide a patch, that doesn't retry on missing files.