Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Node/CCQ/Server: Clean restart #3598

Merged
merged 3 commits into from
Dec 11, 2023
Merged

Conversation

bruce-riley
Copy link
Contributor

@bruce-riley bruce-riley commented Dec 11, 2023

This PR changes the proxy server to handle restarts more cleanly, hopefully without dropping query requests.

It adds two new config parameters (with what should be reasonable default values): shutdownDelay1 and shutdownDelay2. When a sigterm is received, it will do the following:

  • Set a flag to stop responding to health checks
  • Sleep for shutdownDelay1 seconds to allow the load balancer to detect that the server is down and stop routing requests to it.
  • Sleep for up to shutdownDelay2 seconds, but checking every second to see if there are no more pending requests.
  • When there are no more pending requests (or after shutdownDelay2 seconds), the server exits.

The default values for the two config parameters are based on the default health check parameter values. We may want to tweak those parameters, in which case these parameters can be reduced.

The length of time that Linux will wait between a sigterm and a sigkill is controlled by the TimeoutStopSec parameter in the systemd config file. The default is 90s (seconds).

In the tilt environment, shutdownDelay1 is set to zero by default, meaning just shutdown immediately.

@bruce-riley bruce-riley changed the base branch from main to node/ccq/server_add_permissions_file_watcher December 11, 2023 20:34
Base automatically changed from node/ccq/server_add_permissions_file_watcher to main December 11, 2023 20:44
@bruce-riley bruce-riley marked this pull request as ready for review December 11, 2023 20:57
Copy link
Contributor

@evan-gray evan-gray left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

generally in favor, just a couple small comments

node/cmd/ccq/status.go Show resolved Hide resolved
node/cmd/ccq/query_server.go Outdated Show resolved Hide resolved
evan-gray
evan-gray previously approved these changes Dec 11, 2023
Copy link
Contributor

@evan-gray evan-gray left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking forward to this

panoel
panoel previously approved these changes Dec 11, 2023
Copy link
Contributor

@evan-gray evan-gray left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking forward to this

@bruce-riley bruce-riley merged commit 7a2a19c into main Dec 11, 2023
22 checks passed
@bruce-riley bruce-riley deleted the node/ccq/server_clean_restart branch December 11, 2023 22:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants