Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Can't set 0 retries #61

Open
nvonpentz opened this issue Apr 27, 2019 · 5 comments · May be fixed by #62
Open

Can't set 0 retries #61

nvonpentz opened this issue Apr 27, 2019 · 5 comments · May be fixed by #62

Comments

@nvonpentz
Copy link

nvonpentz commented Apr 27, 2019

I have a job that makes a database query that takes around an hour to complete. For some reason qless thinks the process has stalled when it hasn't and attempts to retry the job. This happens 5 times so the process takes 5 hours instead of 1 hour.

I try to set retries=0 as a workaround, but it doesn't work (I think) because of this line:

'retries', retries or 5,

since 0 or 5 evaluates to 5, I can't set 0 retries.

@moteus
Copy link

moteus commented Apr 27, 2019

I think you have to periodically call heartbeat function for this job.

@dlecocq
Copy link

dlecocq commented Apr 27, 2019

@nvonpentz - @moteus is right that the best thing to do is to periodically call job.heartbeat() to retain the lock on the job. Alternatively, you can configure the heartbeat time (this is the amount of time a job can go without checking before it's automatically retried) for the queue with myQueue.heartbeat = <some time in seconds>

@nvonpentz
Copy link
Author

Thank you @moteus and @dlecocq.

I don't think I can call job.heartbeat() since I'm working with a single long blocking database call. The first opportunity to call job.heartbeat() would be after an hour when the query returned, and at that point I'd like to call job.complete() instead.

Setting a long heartbeat ( > than 1 hour) solved the problem of the job retrying @dlecocq.

We should be able to set 0 retries though, no?

@moteus
Copy link

moteus commented Apr 28, 2019

You can run query in a separate thread or process
Also, many libraries supports async calls (e.g. some odbc drivers do)

@dlecocq
Copy link

dlecocq commented Apr 29, 2019

@nvonpentz - I just looked through qless-core because I couldn't off hand remember whether retries meant attempts, or re-attempts. However, it does look like setting it to 0 should be valid, and this is just an issue with the bindings. A CR to only use the default when the retries argument is None should do the trick (and this is the canonical way to implement default arguments in python). I don't have time at the moment, but we'd certainly welcome a CR like that. In the mean time, setting heartbeat appropriately and retries to 1 should help to mitigate the problem.

@nvonpentz nvonpentz linked a pull request Sep 23, 2019 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants