Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[server][dvc] Increase graceful shutdown time for Da Vinci to reduce timeout exception #1331

Merged
merged 6 commits into from
Dec 6, 2024

Conversation

sixpluszero
Copy link
Contributor

[server][dvc] Increase graceful shutdown time for Da Vinci to reduce timeout exception

For Da Vinci, when we close / retire a version, it will call shutdownIngestionTask API, which will try to unsubscribe and sync offset for graceful shutdown. Today the timeout is 10 seconds and it could timeout for topic with many partition assignment.
This will result in two kinds of exceptions:

  1. When it is doing graceful shutdown and syncing offset, it may fail after the wait timeout as it will also remove storage engine
  2. It may also failed to persist certain lingering data in drainer after timeout.

This PR tries to do minimal changes to reduce the noises in application that runs DVC by increasing the total timeout for per SIT shutdown wait.

How was this PR tested?

N/A

Does this PR introduce any user-facing changes?

  • No. You can skip the rest of this section.
  • Yes. Make sure to explain your proposed changes and call out the behavior change.

@sixpluszero sixpluszero merged commit 6bbf149 into linkedin:main Dec 6, 2024
51 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants