Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bug: stop retrying to embed stuff #9842

Closed
Dschoordsch opened this issue Jun 11, 2024 · 2 comments · Fixed by #9877
Closed

Bug: stop retrying to embed stuff #9842

Dschoordsch opened this issue Jun 11, 2024 · 2 comments · Fixed by #9877
Assignees
@Dschoordsch
Labels
bug p2 Needs to be addressed on a reasonable timescale

Comments

@Dschoordsch
Copy link
Contributor

Dschoordsch commented Jun 11, 2024 •
edited
Loading

When checking the embedding queue, there are entries with retries in the thousands. We should stop after a handful of retries.

There are also entries with "no similar discussions found" which is then retried. That shouldn't be an error state.
@Dschoordsch Dschoordsch added bug p2 Needs to be addressed on a reasonable timescale labels Jun 11, 2024
@Dschoordsch
Copy link
Contributor Author

Classifying as p2 because it keeps our infrastructure busy

@Dschoordsch Dschoordsch changed the title Bug: stop retrying embedding stuff Bug: stop retrying to embed stuff Jun 11, 2024
@Dschoordsch Dschoordsch self-assigned this Jun 26, 2024
@Dschoordsch
Copy link
Contributor Author

It happens because we reset failed jobs to stalled. There are 2 issues:

  • resetting stalled jobs should only consider running jobs
  • failing a job should reset the start time

@Dschoordsch Dschoordsch mentioned this issue Jun 26, 2024
Merged
10 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Dschoordsch Dschoordsch

Labels
bug p2 Needs to be addressed on a reasonable timescale
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix: Don't reset failed embedding jobs ParabolInc/parabol
1 participant
@Dschoordsch