Use infinite restarts for xsnippet-api.service #61
Conversation
systemd defaults are fairly strict (<= 5 restarts within 10s) before it gives up on the service. And we would prefer the service to recover automatically after transient failures even if takes longer than that. E.g. yesterday, I found that xsnippet-api had been down after the last machine reboot because it could not fetch the Auth0 key on start, but the same thing could happen with database connection as well.
| @@ -2,6 +2,7 @@ | |||
| Description = XSnippet API | |||
| After = network.target network-online.target | |||
There was a problem hiding this comment.
but the same thing could happen with database connection as well.
I'd suggest to add PostgreSQL to both Wants= and After=. So at least after the reboot we can preserve a proper ordering. Or maybe even Requires= if we want xsnippet-api not be started if PostgreSQL could not start.
There was a problem hiding this comment.
Yeah, that sounds good to me! I'm just not sure whether PostgreSQL unit actually waits until the process is ready to accept connections. So we might need infinite (or, at the very least, extra restarts) as defense in depth.
There was a problem hiding this comment.
If it's linked with systemd, I'd expect it to report status "ready" once it's ready to accept connections.
There was a problem hiding this comment.
Ah, interesting! Didn't expect that! https://github.com/postgres/postgres/blob/578a7fe7b6f8484f6d7caa2fda288abb3fe87aa0/src/backend/postmaster/postmaster.c#L2317-L2325 confirms there is integration with systemd, indeed.
Let me add an explicit dependency on postgresql.service then.
systemd defaults are fairly strict (<= 5 restarts within 10s) before it gives up on the service. And we would prefer the service to recover automatically after transient failures even if takes longer than that.
E.g. yesterday, I found that xsnippet-api had been down after the last machine reboot because it could not fetch the Auth0 key on start, but the same thing could happen with database connection as well.