Feature request: Notify systemd when consul-template has completed its first run

[sfnelson]

On systems that uses systemd for managing processes, it would be helpful if consul-template could inform systemd when it had finished processing templates so that systemd can delay initialising other services until their configurations have been generated.

Specifically, I want to use vault's PKI infrastructure to issue a certificate to identify a machine. I can configure consul-template to create the certificate file and notify the services that use it when the certificate changes (via exec), but I can't delay the services from starting on boot until the certificate has been created. My workaround is to run consul-template -once with ExecStartPre, then run it as a daemon with ExecStart, but this causes the certificate to be issued twice because consul-template doesn't retain information about leases between invocations. This is racy enough that sometimes services will fail to initialise because consul-template is concurrently writing the second certificate/key pair while they are initialising.

If consul-template supported sd_notify (consul does) then I could create a Type=notify service for consul-template and systemd would correctly delay services that depend on consul-template until after it has completed its first run.

Related: hashicorp/consul#2121

[sethvargo]

Hi @sfnelson

Is there a reason you can't signal systemd using the arbitrary command that runs after all templates have been rendered, something like:

$ consul-template -template:foo:bar:/my-script.sh

# my-script.sh
do_stuff
sudo systemctl start foo

[sfnelson]

Hi @sethvargo. Thanks for your reply.

I don't understand what you mean. Perhaps you could explain further?

Here is a simplified version of my problem:

consul-template.service
 - nginx.hcl: /etc/nginx/nginx.conf.tpl => /etc/nginx/nginx.conf
 - forwarder.hcl: /etc/forwarder.conf.tpl => /etc/forwarder.conf
 - certificates.hcl: PKI => /etc/ssl/private/localhost.{cert,key}
nginx.service
 - needs /etc/nginx/nginx.conf (nginx.hcl)
 - needs /etc/ssl/private/localhost.{cert,key} (certificate.hcl)
log-forwarder.service
 - needs /etc/forwarder.conf (forwarder.hcl)
 - needs /etc/ssl/private/localhost.{cert,key} (certificate.hcl)

I could use command in one of the templates to start/restart the services (or a systemd target), but which one should I add it to? In both cases, there are two templates that need to be processed before the service can start.

I could also put all the templates into a single file, but that adds coupling between the services, and in practice I have ~4 services that need the certificate. I want to be able to keep these services separate so that I can use them independently on different systems.

[sethvargo]

Hi @sfnelson

Consul Template isn't really designed to be managed this way, so I'm very hesitant to add support for this. I understand you have a real problem to solve, but I think you're trying to shove a round peg in a square hole. Pushing the boundaries of any tool is going to give you unexpected (and unwanted) results.

Consul Template is two run in two different modes:

• Template mode - in this mode, each template has a command. The command is any valid shell command/script and it fires each time the template contents on disk are changed. The command must exit within a set period of time.

• Exec mode - in this mode, all templates are rendered and then Consul Template spawns and manages a child process. This child process is then sent a configurable signal when any one of the template contents change on disk.

In your example, I think exec mode is the correct choice, since you need all templates to be rendered at least once before you spawn the process. You could use systemd to launch the initial Consul Template process, but Consul Template would manage nginx (in the example above) as a child command. All signals to Consul Template are proxied to the child, so it behaves exactly as if it were the nginx pid.

[sfnelson]

Thanks for your thoughtful reply. We've come to similar conclusions regarding template/exec mode, and have considered running our processes as managed processes within individual consul-template instances. I think that you're right that this is an approach that would work, though it would not be very elegant as we would end up duplicating a bunch of secrets between consul-template instances (e.g. four machine identity certificates, instead of one).

I sympathise with your argument about people using consul-template in a way that you did not intend, but using consul-template in exec mode to manage processes seems to violate security best practices to me, as promoting consul-template to the level of process manager requires elevating its privileges to at least the level of the process(es) it is managing.

Is there some compromise possible here? My pull request that notifies systemd when all templates have been rendered seems like a pretty minor change in behaviour, and it's based on a similar change that was accepted by the consul team recently.

[sfnelson]

As an aside, this thread has some good reasons why running nginx as a child process of consul-template would not be a good idea: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=728015 – essentially, it would prevent nginx from being able to reload without dropping connections. I'd be very interested in your thoughts on how integrate a forking process like nginx with certificates provided by vault's PKI backend.

[sethvargo]

Hi @sfnelson

I'm really quite opposed to adding any kind of OS-level coupling like this. First it's systemd, then it's upstart, then it's whatever the next latest init system happens to be at the time. I don't think tools should be directly integrating or signaling the init system. That's what shell scripts and wrappers are for. We (the tool) will never be able to provide the level of customization every user will want.

I'll give you a corollary. We added child process reaping to Consul Template a few releases ago. It sounded good in theory, but in practice it was a disaster. We ended up completely removing the functionality.

I understand the Consul team already accepted a change like this. Had I been involved in that conversation, I would have advocated against it there too 😄 . We will not be adding this functionality to Consul Template. Thank you for understanding and have a great day!

[vanaf]

Is there any way to exec one-shot command after all templates are rendered?
For single template execution we can just execute systemd-notify --ready as template command.
But what to do for multiple templates? exec mode according to documentation is only for long-running processes

The child process must remain in the foreground. This is a requirement for Consul Template to manage the process and send signals.

[vanaf]

Another issue with using systemd-notify as template command is that it is executed only if template was actually rendered. If file already exists and kept untouched systemd-notify will not be executed and control process will never know that we are good to go

[juliantaylor]

What you can do is add a consul-template -once command as an ExecStartPre= in your systemd file so the templates are all rendered when systemd considers the service active.

But that does not work well when you are issuing certificates from the vault pki as consul-template will fetch new certificates everytime it starts and you end up with two certifcates issued and two service restarts on consul-template service start.