Standalone Certificate

[ghost]

Is it possible to request a standalone certificate for further use (in my case a simple Mail server), that is autorenewed by the nginx-letsencrypt container without spinning up an container, that is basically useless?
I was thinking of something like
docker exec nginx-letsencrypt /app/request....
Parallel to this "standalone cert" i would like to use the complete functionality of this nice container including the proxy for other services (in my case for a simple wiki and a nextcloud instance).

[cphamlet]

I would also like to know this.

[cphamlet]

This docker container makes use of the simp_le client, a lightweight letsencrypt client. https://github.com/zenhack/simp_le

You can try running that command on start, and stick it in the auto renew code.

[buchdag]

An experimental version of this feature inspired by @ymettier PR #220 has been added to dev.

This instructions are here. Any feedback is welcome.

[ladrua]

Just tested this feature, works like a charm. Personally really need this in production yesterday... 😺
How are the changes to the letsencrypt_user_data file picked up/triggered though?

[buchdag]

@ladrua for now they're not picked up at all, meaning you'll either have to

• wait for the next service loop execution (every hour)
• manually trigger it with docker exec your-le-container /app/signal_le_service

[ladrua]

Thanks. Thats fine. I had some issues all of a sudden on a new build of dev, had to RUN chmod +x /app/signal_le_service && chmod +x /app/letsencrypt_service because I got Permission Error. Also after, the chalange configs are left behind in the conf.d(example.com-standalone.conf) folder, and we get errors due to double definition of hostnames(we have other configs for the actuall services).

[ladrua]

Oh, And in /app/functions.sh:45 I got a complaint that the access_log line was missing a suffix ;

[buchdag]

@ladrua I pushed two more commits to dev. The missing ; has been corrected and the challenge config should be correctly removed too. No idea where you permission issue came from, I'll have to take a look again.

[ladrua]

@buchdag Thanks! I'll have a go later. Also, does it create the challenge configs even though renewal is not necessary? Must it? Just realized that this recreation happend every time(hour) the service was triggered. So if I manually deleted the standalone confs, they came back an hour later.

[ladrua]

And another thought, how are we sure the challange configs are used in favour of the manual configs you have for the underlying services during renewal/generation? Do we need some sort of naming convention for the manual configs? I have added a subfolder to the conf.d directory called manuals that I have included in the nginx.conf as a import after the include /etc/nginx/conf.d/*.conf line, just to be sure they are loaded last. But that requires changes in the nginx-proxy containers config file.

[buchdag]

does it create the challenge configs even though renewal is not necessary? Must it?

Yes, because simp_le is determining by itself if the certificate needs to be renewed or not, there is no "pre-renewal" hook we can use to create / change the challenge config only when a certificate is about to be created or renewed.

The situation is the same for the certificate that are created from containers environment variables, add_location_configuration is executed every hour for every domain, wether a certificate is going to be created/renewed or not.

how are we sure the challange configs are used in favour of the manual configs you have for the underlying services during renewal/generation?

I can't answer with certainty yet, that's why this feature is considered experimental for now.

[ladrua]

I see, I guess there must be a naming convention for the standalone configs(maybe matching the LETSENCRYPT_STANDALONE_CERTS array defeintions?), and the service disable/re-enable them somehow?

[ladrua]

Just did a rebuild, I now get this error from docker logs unknown log format "vhost" in /etc/nginx/conf.d/example.com-standalone-cert.conf:4 .. Hm the domain starts with da so it tries to load the config before the default.conf and thats where the vhost logging is defined? I also get

/etc/nginx/certs/example.com /app
Creating/renewal example.com certificates... (example.com)
2018-07-06 09:48:46,879:INFO:simp_le:1564: Certificates already exist and renewal is not necessary, exiting with status code 1.
/app
/etc/nginx/certs/example2.com /app
Creating/renewal example2.com certificates... (example2.com)
2018-07-06 09:48:48,757:INFO:simp_le:1479: Generating new certificate private key
2018-07-06 09:48:50,552:ERROR:simp_le:1446: CA marked some of the authorizations as invalid, which likely means it could not access http://example.com/.well-known/acme-challenge/X. Did you set correct path in -d example.com:path or --default_root? Are all your domains accessible from the internet? Please check your domains' DNS entries, your host's network/firewall setup and your webserver config. If a domain's DNS entry has both A and AAAA fields set up, some CAs such as Let's Encrypt will perform the challenge validation over IPv6. If your DNS provider does not answer correctly to CAA records request, Let's Encrypt won't issue a certificate for your domain (see https://letsencrypt.org/docs/caa/). Failing authorizations: https://acme-v01.api.letsencrypt.org/acme/authz/5sRAWYh8XDt0i_Zc7HTh0u61SFcMTvJFF65NUD0c9dE
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/simp_le.py", line 1584, in main
    return main_with_exceptions(cli_args)
  File "/usr/lib/python2.7/site-packages/simp_le.py", line 1567, in main_with_exceptions
    persist_new_data(args, existing_data)
  File "/usr/lib/python2.7/site-packages/simp_le.py", line 1489, in persist_new_data
    account_key=client.key, key=None, cert=None, chain=None))
  File "/usr/lib/python2.7/site-packages/simp_le.py", line 1195, in persist_data
    plugin.save(new_data)
  File "/usr/lib/python2.7/site-packages/simp_le.py", line 493, in save
    key = self.dump_key(data.key)
  File "/usr/lib/python2.7/site-packages/simp_le.py", line 436, in dump_key
    return OpenSSL.crypto.dump_privatekey(self.typ, data.wrapped).strip()
AttributeError: 'NoneType' object has no attribute 'wrapped'

Unhandled error has happened, traceback is above

Debugging tips: -v improves output verbosity. Help is available under --help.
/app
Reloading nginx proxy (f6a4d720ab48f26ccb1f6ff5ae2b388579098f1547f7b98198123616f6fd3c89)...
2Custom dhparam.pem file found, generation skipped
h2018/07/06 09:48:50 Contents of /etc/nginx/conf.d/default.conf did not change. Skipping notification ''
�2018/07/06 09:48:50 [emerg] 3852#3852: unknown log format "vhost" in /etc/nginx/conf.d/example.com-standalone-cert.conf:4
nginx: [emerg] unknown log format "vhost" in /etc/nginx/conf.d/example.com-standalone-cert.conf:4

The full output of the log, there are two domains(example.com and example2.com), the first Create/renewal succeedes, but it seems that it leaves the config behind, and then the second Create/renew fails.

[buchdag]

I've pushed a new commit with the vhost logging format defined directly in the challenge config file, that should fix that logging error.

I see, I guess there must be a naming convention for the standalone configs(maybe matching the LETSENCRYPT_STANDALONE_CERTS array defeintions?), and the service disable/re-enable them somehow?

The expected behavior is (no matter what happen to the certificate):

• create required standalone config files
• reload nginx
• let simp_le create / renew / do nothing
• remove standalone config files
• reload nginx

No idea why simp_le crashed like that, I did not touch anything that effect the simp_le process. The only relevant issue I found is this one on another repo : evertramos/nginx-proxy-automation#62

[ladrua]

Sorry, I did update my comment a bit with a bit more detailed information.

[ladrua]

I think you need to remove the fix for the vhost log_format, as I now get duplicate "log_format" name "vhost" in /etc/nginx/conf.d/default.conf:31 and I am pretty sure it happened because of my domain name starting with da and hence the config is loaded before the default.conf Not sure how to solve that though.

[ladrua]

It still leaves my example.com-standalone-cert.conf it might be because the second domain example2.com fails?

/etc/nginx/certs/example.com /app
Creating/renewal example.com certificates... (example.com)
2018-07-06 10:39:58,610:INFO:simp_le:1564: Certificates already exist and renewal is not necessary, exiting with status code 1.
/app
/etc/nginx/certs/example2.com /app
Creating/renewal example2.com certificates... (example2.com)
2018-07-06 10:40:00,421:INFO:simp_le:1479: Generating new certificate private key
2018-07-06 10:40:02,160:ERROR:simp_le:1446: CA marked some of the authorizations as invalid, which likely means it could not access http://example.com/.well-known/acme-challenge/X. Did you set correct path in -d example.com:path or --default_root? Are all your domains accessible from the internet? Please check your domains' DNS entries, your host's network/firewall setup and your webserver config. If a domain's DNS entry has both A and AAAA fields set up, some CAs such as Let's Encrypt will perform the challenge validation over IPv6. If your DNS provider does not answer correctly to CAA records request, Let's Encrypt won't issue a certificate for your domain (see https://letsencrypt.org/docs/caa/). Failing authorizations: https://acme-v01.api.letsencrypt.org/acme/authz/BG_wfHlxgf2zf3W0SKzcCBeapDFGszBU56tGoH7Hdp0
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/simp_le.py", line 1584, in main
    return main_with_exceptions(cli_args)
  File "/usr/lib/python2.7/site-packages/simp_le.py", line 1567, in main_with_exceptions
    persist_new_data(args, existing_data)
  File "/usr/lib/python2.7/site-packages/simp_le.py", line 1489, in persist_new_data
    account_key=client.key, key=None, cert=None, chain=None))
  File "/usr/lib/python2.7/site-packages/simp_le.py", line 1195, in persist_data
    plugin.save(new_data)
  File "/usr/lib/python2.7/site-packages/simp_le.py", line 493, in save
    key = self.dump_key(data.key)
  File "/usr/lib/python2.7/site-packages/simp_le.py", line 436, in dump_key
    return OpenSSL.crypto.dump_privatekey(self.typ, data.wrapped).strip()
AttributeError: 'NoneType' object has no attribute 'wrapped'

Unhandled error has happened, traceback is above

Debugging tips: -v improves output verbosity. Help is available under --help.
/app
Reloading nginx proxy (f6a4d720ab48f26ccb1f6ff5ae2b388579098f1547f7b98198123616f6fd3c89)...
2Custom dhparam.pem file found, generation skipped
h2018/07/06 10:40:02 Contents of /etc/nginx/conf.d/default.conf did not change. Skipping notification ''
�2018/07/06 10:40:02 [emerg] 3929#3929: duplicate "log_format" name "vhost" in /etc/nginx/conf.d/default.conf:31
nginx: [emerg] duplicate "log_format" name "vhost" in /etc/nginx/conf.d/default.conf:31

[buchdag]

I think you need to remove the fix for the vhost log_format

Done. I also changed the test unit slightly to cover making change to letsencrypt_user_data and manually triggering of the service loop to pick up those changes.

it happened because of my domain name starting with da and hence the config is loaded before the default.conf Not sure how to solve that though.

The most straightforward and simple solution is probably just to name the config file standalone-cert-$domain.conf instead of $domain-standalone-cert.conf. This change has been committed to dev.

It still leaves my example.com-standalone-cert.conf it might be because the second domain example2.com fails?

Could you try to remove the certificate covering the domain whose challenge is failing from letsencrypt_user_data ?

[ladrua]

Looks good to me with the latest changes, but the change of name convention to standalone-cert-$domain.conf would not work for say.. xxx.com.

[buchdag]

but the change of name convention to standalone-cert-$domain.conf would not work for say.. xxx.com

I don't understand why it would not work, "working" here being the domain specific, self generated conf file loading after default.conf no matter what the domain is.

Unless I'm mistaken, a conf file starting with an s will always load after a conf file starting with a d.

[ladrua]

Sorry, I was thinking about it all wrong. Yes, it should be fine 👍
Hope this makes it to master as this is super usefull, well done!

[buchdag]

Hope this makes it to master as this is super usefull, well done!

Thanks, I really appreciate that you took the time to test this and give feedback.

I'd like a bit more user testing and feedback (even just "yup it works") from people with different setups before merging it into master.

I'll keep dev in sync with master anyway.

[buchdag]

@ymettier @mossholderm @ryneeverett @ForsakenHarmony @augusteo @Pimmetje @hamon-e @Panderine @CWempe @flocomkoko @cphamlet @hadrien-toma @kosli @VeeeneX @felixsteghofer @curtiszimmerman

You manifested interest at some point for the ability to generate "standalone" certificates with letsencrypt-nginx-proxy-companion (ie certificate not created from a Docker container environment variables). A beta version of this feature, inspired by @ymettier #220 has been merged to the dev branch and is available through Docker hub with the dev tagged version of this container.

Instructions are here : https://github.com/JrCs/docker-letsencrypt-nginx-proxy-companion/wiki/Standalone-certificates-(beta)

If you are still using letsencrypt-nginx-proxy-companion and are still interested by this feature, test and feedback would be super helpful.

Sorry in advance if the mention for an old topic bothered you.

[curtiszimmerman]

Cool! I have use cases for this, and some of them are weird edge cases. Thanks for the tag! I'll check it out.

[ladrua]

@buchdag I just updated with the docker image, and I am still having issue that the standalone-cert*.conf files are beeing created at every check, and then not removed in conf.d folder. Casuing problems with overriding some of my configs that are imported after the conf.d import. Other then that, it is working great!

[buchdag]

That's weird, I've tried adding special checks for that in the test unit, and the resulting build pass without issues. Are you sure you pulled or build an up to date version and those aren't leftovers files from a previous dev version ?

[buchdag]

I updated the dev image to track changes in latest + the following changes:

• /etc/nginx/conf.d presence and writeability is now checked once during container init with the existing check_writable_directory function
• any remaining standalone configuration files are cleaned up on container exit
• added the previously mentioned check to the certs_standalone test unit
• added an additional debug message when standalone configuration files are deleted during the LE service loop

[ladrua]

@buchdag I cant actually be sure that they werent left there from previous build. Will try and do some more tests this evening. Thank you!

[buchdag]

@ladrua the for loop that handle standalone config removal was at the wrong level, that should finally be fixed in the latest dev version.

[ladrua]

I can confirm that its working as expected now. Have it running on two different live environments, and its looking good so far.

[buchdag]

Anything else to report from people who tested this feature (or are currently using it) ?

I'm considering it for merging into the master branch in the coming weeks.

[ladrua]

All I can say is that I've been running them since I last mentioned it, in two production environments, without any issues.

[Gaibhne]

The patch for this in the dev branch does not work anymore, because the current nginx images don't run as root anymore, making it necessary to run on a 1024+ port. Unfortunately, port 80 is hard coded in functions.sh#L109. Of course it is relatively trivial to change it yourself to a different port and recompile the image, but I would suggest a config parameter to allow for a more convenient usage.

[buchdag]

The patch for this in the dev branch does not work anymore.

Yet the test still passes using the latest nginx:alpine.

As far as I can tell the current nginx images are still listening on port 80.

Could you provide more info about the issue you are experiencing ?

[buchdag]

Sorry it took soooooo long to merge but this feature is now in latest