Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore(server): graceful shutdown #3125

Merged
merged 20 commits into from
Nov 19, 2024
Merged

chore(server): graceful shutdown #3125

merged 20 commits into from
Nov 19, 2024

Conversation

iainsproat
Copy link
Contributor

@iainsproat iainsproat commented Sep 25, 2024

Description & motivation

When a deployment rolls out (causing a shutdown of an existing server), we see error messages from the Apollo server.

Screenshot 2024-09-25 at 15 49 05

New connections were being made while the server was being shut down.

Apollo server was automatically shutting down when a SIGTERM or SIGINT signal was received; this created a race condition where it may close prior to the http server being drained of connections.

Changes:

  • apollo server should not automatically close on receiving SIGTERM/SIGINT
  • the apollo server should be shutdown after the server is drained of connections, this can be sequenced by the terminus package.
  • the readiness endpoint should be managed by terminus, which ensures they respond unready if the server is being shutdown.
  • the shutdown timeout (grace period) can be configured via environment variable

To-do before merge:

Screenshots:

Validation of changes:

Screenshot 2024-09-26 at 19 13 08 Screenshot 2024-09-26 at 19 13 13

Checklist:

  • My pull request follows the guidelines in the Contributing guide?
  • My pull request does not duplicate any other open Pull Requests for the same update/change?
  • My commits are related to the pull request and do not amend unrelated code or documentation.
  • My code follows a similar style to existing code.
  • I have added appropriate tests.
  • I have updated or added relevant documentation.

References

@gjedlicska
Copy link
Contributor

this doesn't feel right to have as a module.

There is only application (express and apollo server) logic in here and no business logic.
I would put the code somewhere next to the app.ts file

Copy link
Contributor

@gjedlicska gjedlicska left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this logic should not be a module, its application logic

- terminus can only handle readiness check, not liveness
- app needs to return readiness handler, so that server terminus can use it
@iainsproat iainsproat marked this pull request as ready for review September 27, 2024 13:17
fabis94
fabis94 previously approved these changes Nov 12, 2024
packages/server/modules/stats/tests/stats.spec.ts Outdated Show resolved Hide resolved
@gjedlicska gjedlicska merged commit a1ee8a8 into main Nov 19, 2024
26 of 28 checks passed
@gjedlicska gjedlicska deleted the iain/graceful-shutdown branch November 19, 2024 10:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants