Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

v0.6.0 job failures go into a continuous loop #372

Closed
kdesjard opened this issue Jul 15, 2024 · 8 comments
Closed

v0.6.0 job failures go into a continuous loop #372

kdesjard opened this issue Jul 15, 2024 · 8 comments
Assignees
Labels
bug Something isn't working

Comments

@kdesjard
Copy link
Contributor

kdesjard commented Jul 15, 2024

Job failures go into a continuous loop regardless of # of retries.

Using sqlite and for the given job, the attempts column is zero.

@geofmureithi geofmureithi self-assigned this Jul 15, 2024
@geofmureithi geofmureithi added the bug Something isn't working label Jul 15, 2024
@geofmureithi
Copy link
Owner

Will look into it and do a patch

@geofmureithi
Copy link
Owner

@kdesjard does the current 0.6 branch work for you?

@kdesjard
Copy link
Contributor Author

kdesjard commented Jul 17, 2024

No, it still continuously loops and the db row has 0 attempts:

sqlite> select attempts,max_attempts from jobs;
0|25

Does not work with or without a retry policy. With a retry policy of zero or one, I still get "attempt to subtract with overflow"

Am I supposed to call record_attempt ?

@geofmureithi
Copy link
Owner

Are you using the RetryLayer? Could you share a minimal example?

@geofmureithi
Copy link
Owner

I think I found the bug.
Ack does not try to increase the number of attempts.
Will fix this.

@geofmureithi
Copy link
Owner

Should be resolved in #374
The problem was ack did not update the retries.

@kdesjard
Copy link
Contributor Author

With a retry policy of 0:

RetryLayer::new(RetryPolicy::retries(0));

A failed job runs many times (not sure if 25 or 26):

sqlite> select attempts,max_attempts from jobs;
26|25

A successful job runs once, but is the DB has 2:

select attempts,max_attempts from jobs;
2|25

@geofmureithi
Copy link
Owner

With a retry policy of 0:

RetryLayer::new(RetryPolicy::retries(0));

A failed job runs many times (not sure if 25 or 26):

sqlite> select attempts,max_attempts from jobs;
26|25

A successful job runs once, but is the DB has 2:

select attempts,max_attempts from jobs;
2|25

I think there is confusion here between RetryLayer and Storage retries. RetryLayer just defines the number of in memory retries but it's still based on attempts and max_attempts. You may want to either set max_atempts or return Error::Abort to prevent retries.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants