Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(v2): drain inflight requests #3639

Merged
merged 3 commits into from
Oct 21, 2024
Merged

Conversation

kolesnikovae
Copy link
Collaborator

Currently, because the HTTP/gRPC server is a dependency of services and is shut down last, there is a possibility that it may receive requests after de-initialization. In most cases, if the deployment is managed properly, the issue is barely noticeable: the load balancer's grace period allows clients to remove the peer from the "pool."

However, when dealing with client-side discovery (e.g., via memberlist/ring, Kubernetes, DNS, etc.), nothing prevents clients from sending requests to a target that is already de-initialized but not yet excluded from the list of available/discoverable instances. The fix ensures that a request will either be handled or rejected appropriately.

Note that heartbeats/health checks are not helpful in this case (and in many other scenarios).

The problem is not specific to the experimental modules; however, the fix is only enabled for them.

@kolesnikovae kolesnikovae marked this pull request as ready for review October 21, 2024 07:13
@kolesnikovae kolesnikovae requested a review from a team as a code owner October 21, 2024 07:13
@kolesnikovae kolesnikovae merged commit 09b5ad1 into main Oct 21, 2024
18 checks passed
@kolesnikovae kolesnikovae deleted the fix/drain-inflight-requests branch October 21, 2024 08:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants