Can't get cert from letsencrypt for my domain name

[christeasdale]

Confirmed working changing to v02 at least for me. Changing my prod machine and making sure it doesn't break, but my test worked fine and can now access securely.

Originally posted by @bdschuster in #104 (comment)

I seem to be have certificate issues. Gooby can't seem to get a cert for my domain (MYDOMAIN).net, or any domain. It doesn't even create a folder for the my domain name. I have tried managing my domain and a few rcleans, Plex works but I think this is the reason I have been having clients that run AndroidTV with the new player. This I guess checks certificates and fails with a connection error.
Common Name: letsencrypt-nginx-proxy-companion is an self signed default cert. when I attempt to navigate to plex.plexmovies.net

I even attempted with a completely new clean Ubuntu server and fresh install of Gooby and still couldn't get it working to get a certificate. I am pulling my hair out over here. Mine is not the same error it is using V02 but seems somewhat similar. I could really use some help. Thanks

[Adoruta]

It might be a couple of things.
Try docker logs letsencrypt -f to see what that's saying about your certificate. It's quite possible you've hit limits on how many can be requested now.

Secondly, there was a fairly recent change on the letsencrypt docker image, so possibly you've been hit by that. I added the line - ${CONFIGS}/Docker/acme.sh:/etc/acme.sh to the volumes section for letsencrypt-nginx-proxy-companion so the acme folder was missing outside of the docker image. Watchtower and docker-compose would periodically replace the letsencrypt docker image and therefore reset the acme folder contents within the docker within the docker image.

Contents of file \opt\Gooby\scripts\nginx\03-proxy.yaml (I've uploaded as a txt file).
03-proxy.txt

[christeasdale]

I made the change to the 03-proxy.yaml you gave me in the txt file. Then I did "Manage Domain Name" in Gooby.
then got "docker logs letsencrypt -f" with the following:

Info: running letsencrypt-nginx-proxy-companion version v2.0.0
Info: Custom Diffie-Hellman group found, generation skipped.
Reloading nginx proxy (82a2f1d7d197ba8191513f947a0f412ac742a3b1b113b937529a328c09073531)...
2020/12/03 10:15:20 Contents of /etc/nginx/conf.d/default.conf did not change. Skipping notification ''
2020/12/03 10:15:20 [notice] 43#43: signal process started
2020/12/03 10:15:20 Generated '/app/letsencrypt_service_data' from 5 containers
2020/12/03 10:15:20 Running '/app/signal_le_service'
2020/12/03 10:15:20 Watching docker events
2020/12/03 10:15:20 Contents of /app/letsencrypt_service_data did not change. Skipping notification '/app/signal_le_service'
[Thu Dec 3 10:15:20 UTC 2020] Create account key ok.
[Thu Dec 3 10:15:21 UTC 2020] Registering account: https://acme-v02.api.letsencrypt.org/directory
[Thu Dec 3 10:15:22 UTC 2020] Registered
[Thu Dec 3 10:15:22 UTC 2020] ACCOUNT_THUMBPRINT='iT49AbWMl43B6PpQTVMIMbLFNCmMddtzVIEnyyMWMxk'
Reloading nginx proxy (82a2f1d7d197ba8191513f947a0f412ac742a3b1b113b937529a328c09073531)...
2020/12/03 10:15:22 Contents of /etc/nginx/conf.d/default.conf did not change. Skipping notification ''
2020/12/03 10:15:22 [notice] 68#68: signal process started
Creating/renewal plex.(MYDOMAIN).net certificates... (plex.(MYDOMAIN).net)
[Thu Dec 3 10:15:23 UTC 2020] Using CA: https://acme-v02.api.letsencrypt.org/directory
[Thu Dec 3 10:15:23 UTC 2020] Creating domain key
[Thu Dec 3 10:15:23 UTC 2020] The domain key is here: /etc/acme.sh/(MY_EMAIL)@gmail.com/plex.(MYDOMAIN).net/plex.(MYDOMAIN).net.key
[Thu Dec 3 10:15:23 UTC 2020] Single domain='plex.(MYDOMAIN).net'
[Thu Dec 3 10:15:23 UTC 2020] Getting domain auth token for each domain
[Thu Dec 3 10:15:24 UTC 2020] Create new order error. Le_OrderFinalize not found. {
"type": "urn:ietf:params:acme:error:rateLimited",
"detail": "Error creating new order :: too many certificates already issued for exact set of domains: plex.(MYDOMAIN).net: see https://letsencrypt.org/docs/rate-limits/",
"status": 429}
[Thu Dec 3 10:15:24 UTC 2020] Please check log file for more details: /dev/null
Sleep for 3600s

Clearly I am over the rate limits. I checked crt.sh and it seems when I did a reboot after over a month since last reboot on Nov 30th and on reboot it attempted to make 10 certificates. How should I proceed. Is the rate limit lift after a week?

[christeasdale]

The change you had me make to 03-proxy.yaml, won't that revert on next reboot or will it take an rclean? Will that be needed after my rate limit is lifted? Oh thanks much, reverse proxy configs isn't my best area of skills by a mile.

[Adoruta]

You may find certificates start to renew from tomorrow. To go faster, you'd need to change domain unfortunately. The watchtower service checks for updates to docker images in the background and will pull down new docker images, replacing the existing docker images to replace your existing containers. That'd trigger new certificate requests because the old ones were cleared out when your existing container for letsencrypt was replaced by watchtower.

[Adoruta]

I'm not sure when the changes to 03-proxy.yaml will be reverted. I guess as you've exceeded your certificate requests you could find out now. I've forked Gooby so I can keep my configuration, but warning - it's also a work in progress, and optimised for Dropbox rather than Google Drive.

[christeasdale]

Well plex.plexmovies.net still resolves to the server it just isn't a secure connection. You understand why it tried to request 10 certs on Nov 30th reboot? Can I leave it running for 24hours and then how should I proceed, reboot, rclean, other?

[Adoruta]

I hypothesise on the 30th November reboot, the container was replaced (that contained your acme generated certs), so they all needed recreating. Yes, you can leave it running for 24 hours. If you keep checking docker logs letsencrypt -f then you should see them getting issued as the rate limits expire.

[christeasdale]

Much appreciated ... All this advanced reverse proxy config is way out of my comfort zone. I might need to not let the server run so long without a reboot going forward.

[TechPerplexed]

That makes two of us, @christeasdale 😄
To answer at least one of your questions: the 03 proxy file (or any of the component files really) won't be overwritten with a regular rclean, in fact unless something significant needs to be updated, none of them will change at all.
If it does need to be updated, it would be during a new release and be listed in the changelog here on Github :)

[Ninazuu]

Also having this issue, I have no idea what I am doing after reading the above, all was working fine until I ran a system cleanup last week and now it is saying I have an invalid certificate.
I deleted and reinstalled the origin certificate but that has not fixed it.
I also had the issue with docker where I had to create an account and then log in on the server and these 2 things happened at the same time, are they related?

[christeasdale]

I also added a Docker account since I was getting the "Pull" error and was hoping that might solve this issue. I thought maybe updated containers weren't being installed. I hate to say it but I am glad I am not the only one having this issue, Means I might not be crazy.

[christeasdale]

Also having this issue, I have no idea what I am doing after reading the above, all was working fine until I ran a system cleanup last week and now it is saying I have an invalid certificate.
I deleted and reinstalled the origin certificate but that has not fixed it.
I also had the issue with docker where I had to create an account and then log in on the server and these 2 things happened at the same time, are they related?

Check crt.sh with your domain name. For whatever reason Gooby on my server requested 10 certificates during a reboot on 11/30. Apparently it must have kept trying when none of them worked and still aren't working. Asking for that many certs got me restricted on letsencypt so I am waiting it out until my limit is lifted. I think it will be Monday. But I am worried it will again make too many requests if whatever the issue isn't resolved.

[christeasdale]

That makes two of us, @christeasdale 😄
To answer at least one of your questions: the 03 proxy file (or any of the component files really) won't be overwritten with a regular rclean, in fact unless something significant needs to be updated, none of them will change at all.
If it does need to be updated, it would be during a new release and be listed in the changelog here on Github :)

So my letscrypt ban was lifted and Gooby install pulled two new certs. But the file "\opt\Gooby\scripts\nginx\03-proxy.yaml" was reverted back to the original file. I am not sure how important Adoruta's changes were to this file. But there still seems to be an issue?

When I did reboot not only was that script changed back to the one on Github but my server again pulled two new certs from letscrypt. Is this expected behavior? Because if it is will I get another ban if I restart my server more than 5 times in one week? Something doesn't seem correct here?

[Coxeroni]

I got hit by the same issue and reached the rate limit. After changing that line in 03-proxy.yaml how long do I need to wait and does the refresh of the certificates happen automatically?

[Adoruta]

I've created a new pull request to fix the proxy problem. I'll need @TechPerplexed to approve before it becomes part of Gooby. As for when the certificates will refresh, https://letsencrypt.org/docs/rate-limits/ has the best answer. It'll happen automatically - eventually. Problem at the moment is the proxy container may be recreated, losing any certificates stored within it. What my script does is expose another folder that is then preserved when the server is rebooted / rclean executed.

[christeasdale]

I got hit by the same issue and reached the rate limit. After changing that line in 03-proxy.yaml how long do I need to wait and does the refresh of the certificates happen automatically?

Sorry to say but it seems likely your LetsCrypt ban won't lift until early next Monday. Hopefully they will have this issue worked out with Gooby by then.

Adoruta thanks again for your work to solve this.

[TechPerplexed]

Thanks guys for your troubleshooting skills and coming up with solutions. The patch has now been applied.... however it would (should) not automatically overwrite your current file except in the case of upgrading an older version of Gooby. @christeasdale I'm really puzzled why yours reverted, did you restore a backup perhaps?

@Adoruta thanks for the patch, let's hope this solves things... fingers crossed!

[christeasdale]

Thanks guys for your troubleshooting skills and coming up with solutions. The patch has now been applied.... however it would (should) not automatically overwrite your current file except in the case of upgrading an older version of Gooby. @christeasdale I'm really puzzled why yours reverted, did you restore a backup perhaps?

@Adoruta thanks for the patch, let's hope this solves things... fingers crossed!

Nope all I did was "sudo reboot" and I just checked in WinSCP and it seems all files in my "/opt/Gooby" location get reverted back to the Github version on each reboot. I can see it by the changed date. Is this odd behavior? Something I need to look into?

[TechPerplexed]

Yes that shouldn't happen. Can you check if you have a file /var/local/Gooby/Docker/.config/version with the content 2.2.2?
That file confirms you have the latest Gooby version which will then prevent running the upgrade patch that overwrites your existing yaml files...

[Adoruta]

Might be worthwhile raising as a new issue... If I do a clean install of Gooby then the version is not set, so I typically force through an update through the gooby menu straight after install to get it set. Then it settles down. Without it, I've witnessed erratic behaviour.

[TechPerplexed]

You are correct @Adoruta - the version file is set after the first rclean (either from the menu, from running the command or after a reboot). I would agree that is a bit of a sloppy solution so I'm open for any suggestions in that regard 👍

However once the version is set, it shouldn't overwrite those initial files any longer. I have a feeling that Chris' timelapse between reboots has been SO long that it missed a vital update in between...

[christeasdale]

I just did a "rclean" to get the change and it shows this message "Your system has already been upgraded to v2.2.2... skipping upgrade" I checked the version file "/var/local/Gooby/Docker/.config/version" and it shows 2.2.2. I ran another update Gooby inside the Gooby menus. Problem now is it now yet again pulled more certs from LetsCrypt ... I am up to 8 since Monday. If this patch doesn't work I will soon be screwed until next Monday.

[TechPerplexed]

Just to verify: you checked the date on the yaml files INSIDE the components folder, right? Those are the ones that feed the composer file and should not be overwritten any longer.

[Adoruta]

Also, please check your /var/local/Gooby/Docker/docker-compose.yaml file (e.g. sudo nano /var/local/Gooby/Docker/docker-compose.yaml) on line 41 to confirm the letsencrypt-nginx-proxy-companion section has a volumes block that includes - ${CONFIGS}/Docker/acme.sh:/etc/acme.sh. If it doesn't something is up and you'll certainly get the certificates getting regenerated on a reboot. It should look like this:

Certificate handling

letsencrypt-nginx-proxy-companion:
restart: always
image: jrcs/letsencrypt-nginx-proxy-companion
container_name: letsencrypt
hostname: letsencrypt
environment:
- ACME_CA_URI=https://acme-v02.api.letsencrypt.org/directory
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
- ${CONFIGS}/Docker/acme.sh:/etc/acme.sh
volumes_from:
- nginx-proxy
healthcheck:
test: ["CMD-SHELL", "pidof letsencrypt_service"]
interval: 10s
timeout: 2s
retries: 3

[christeasdale]

\opt\Gooby\scripts\nginx or a different location?

[TechPerplexed]

The file /var/local/Gooby/Docker/docker-compose.yaml is created from the files inside /var/local/Gooby/Docker/components. So you'll need to edit /var/local/Gooby/Docker/components/03-proxy.yaml for it to "stick" after a reboot/rclean.

[christeasdale]

Neither file had this line "- ${CONFIGS}/Docker/acme.sh:/etc/acme.sh" and I added them to both. They held for restart but now I have pulled all 10 - certs for the week. I hope this works.

But this brings up the whole reason I looked into this issue. The Plex "New Player" on AndroidTV checks for a valid cert or playback fails. I have several FireTV and me with a new ShieldTV having this problem.

So my server I was having this issue. Gooby changes Plex setting "Custom server access URLs" to "https://plex.MYDOMAIN.net:8443" It isn't a secure it and why it isn't is because it is using a cert that belongs to Plex Inc and not my domain name. But if I change i"Custom server access URLs" to "https://plex.MYDOMAIN.net" that is secure with my LetsCrypt cert and AndroidTV Plex app "New Player" now finds a good cert and works without anymore errors.

Also note that I have remote access turned off in Plex settings, Plex is just publishing the "https://plex.MYDOMAIN.net" domain name to Plex.tv.

I use Google Domains and DNS with an A record for "plex" to my server IP.

It seems really weird that "https://plex.MYDOMAIN.net:8443" uses Plex Inc cert? You guys are really helping me here. Did I break the internet?!

[christeasdale]

Just an update on the most recent restart everything in "/opt/Gooby" location got downloaded once again from Github fresh.

[TechPerplexed]

@christeasdale to answer your last reply first: yes, this is the default behaviour and nothing alarming about it :) The /opt/Gooby folder is refreshed with every rclean/reboot. However, the files in /var/local/Gooby/Docker/components are "yours" and will not be overwritten unless there is compelling reason to do so, such as when there is an upgrade that makes updating one of those files mandatory. I'm on the fence of doing so for this patch in fact...

As for the Plex cert using 8443... yikes, I'm not sure I'm afraid. I noticed that both plex.mydomain and plex.mydomain:8443 seem to behave a bit odd. It first gives a 401 error and then displays the Plex logo without going any further. Unfortunately I stopped using Plex a few years ago in favor of Emby, so I wouldn't really notice it.
When I access my Plex server through https://app.plex.tv/ it seems to behave normally. Perhaps the all knowing @Adoruta has an answer to that? 😸

[christeasdale]

Okay I feel better about at least one thing. I am not sure this solution has fixed my issue yet. But I kind of want to sit tight until next Monday as to not violate LetsCrypt limit again. Maybe there is an easy solution of something I could change to stop Gooby from adding :8443 after my domain in Plex settings "Custom server access URLs"? But hope @Adoruta or someone else chimes in that might know more. These are all things I have never touched in Gooby. I just basically run the setup and follow directions.

[TechPerplexed]

I just ran rclean (with the new settings) and I'm now getting the "526 certificate invalid" on my sites. I'm afraid it's not solved yet... not sure what can be done!

[Adoruta]

Spooky. Using Cloudflare DNS with proxy enabled for these sites?

[TechPerplexed]

Yes I am

Edit: I'm getting a list of errors like these:

[Wed Dec 9 21:51:33 UTC 2020] Getting domain auth token for each domain
[Wed Dec 9 21:51:34 UTC 2020] Create new order error. Le_OrderFinalize not found. {
"type": "urn:ietf:params:acme:error:rateLimited",
"detail": "Error creating new order :: too many certificates already issued for exact set of domains: emby.(mydomain).com: see https://letsencrypt.org/docs/rate-limits/",

[Adoruta]

Can you turn the proxy off temporarily whilst you renew your certificates please and see if it bursts into life. If so, you can turn back on the proxy, but of course it'll likely fail again in 90 days.

[TechPerplexed]

You mean the CloudFlare proxy? I just did that... unfortunately it seems this didn't solve the SSL error - I'm still getting the 526 error plus the notion that too many certificates have been issued for my domain. I wish I knew what suddenly changed with the certificates!

[Adoruta]

If I look at https://github.com/nginx-proxy/docker-letsencrypt-nginx-proxy-companion then it mentions "The v2.0.0 release of this project mark the switch of the ACME client used by the Docker image from simp.le to acme.sh. This switch result in some backward incompatible changes, so please read this issue for more details before updating your image. The last tagged version that uses simp_le is v1.13.1." which is what was released very recently, so my gut feeling is it's that. I guess the other option is to pull in the tagged 1.13.1 version instead of latest and remove the acme.sh volume from docker-compose. Thoughts?

[christeasdale]

Mine seems to be holding with my plex.MYDOMAIN.net secure using the LetsCrypt cert. but plex.MYDOMAIN.net:8443 returns
Error code: SSL_ERROR_BAD_CERT_DOMAIN because it is using cert that belongs to Plex INC. But I haven't touched anything no reboot or rclean.

docker logs letsencrypt -f returns
Creating/renewal plex.MYDOMAIN.net certificates... (plex.MYDOMAIN.net)
[Wed Dec 9 21:34:32 UTC 2020] Domains not changed.
[Wed Dec 9 21:34:32 UTC 2020] Skip, Next renewal time is: Sat Feb 6 16:59:22 UTC 2021
[Wed Dec 9 21:34:32 UTC 2020] Add '--force' to force to renew.
Sleep for 3600s

If any of that helps.

[TechPerplexed]

We might also try looking into why it pulls so many certificates in with each rclean. Obviously this was not an issue in the past, now it is.
The relevant lines (I believe) in this regard are

/usr/local/bin/docker-compose down
(...)
/usr/local/bin/docker-compose pull
/usr/local/bin/docker-compose up --remove-orphans --build -d
docker system prune -a -f --volumes

Alternatively, we could definitely try to "force" v1.13.1 but not sure if that would solve the issue. Since I am well into the ban, I'm willing to test... I tried to change the image from wilder/nginx-proxy:alpine to wilder/nginx-proxy:1.13.1 but I guess it's not as easy as all that... it didn't pull anything 😄

[bdschuster]

We might also try looking into why it pulls so many certificates in with each rclean. Obviously this was not an issue in the past, now it is.
The relevant lines (I believe) in this regard are

/usr/local/bin/docker-compose down
(...)
/usr/local/bin/docker-compose pull
/usr/local/bin/docker-compose up --remove-orphans --build -d
docker system prune -a -f --volumes

Alternatively, we could definitely try to "force" v1.13.1 but not sure if that would solve the issue. Since I am well into the ban, I'm willing to test... I tried to change the image from wilder/nginx-proxy:alpine to wilder/nginx-proxy:1.13.1 but I guess it's not as easy as all that... it didn't pull anything 😄

@TechPerplexed , you would change the jrcs/letsencrypt-nginx-proxy-companion to jrcs/letsencrypt-nginx-proxy-companion:v1.13.1, it the companion that is the issue. I did this, and it fixed the issue and my certs were re-issued with no error.

[christeasdale]

We might also try looking into why it pulls so many certificates in with each rclean. Obviously this was not an issue in the past, now it is.
The relevant lines (I believe) in this regard are

/usr/local/bin/docker-compose down
(...)
/usr/local/bin/docker-compose pull
/usr/local/bin/docker-compose up --remove-orphans --build -d
docker system prune -a -f --volumes

Alternatively, we could definitely try to "force" v1.13.1 but not sure if that would solve the issue. Since I am well into the ban, I'm willing to test... I tried to change the image from wilder/nginx-proxy:alpine to wilder/nginx-proxy:1.13.1 but I guess it's not as easy as all that... it didn't pull anything 😄

@TechPerplexed , you would change the jrcs/letsencrypt-nginx-proxy-companion to jrcs/letsencrypt-nginx-proxy-companion:v1.13.1, it the companion that is the issue. I did this, and it fixed the issue and my certs were re-issued with no error.

You guys are talking way above my skill level but I will say the changes that @TechPerplexed and @Adoruta had my apply have worked so far. I had to reboot last night and so far my certs have not attempted to update, which is good since I am currently at the 10 certs, weekly limit. Still getting this below, which I believe is correct. When Monday comes I am willing to test some more things when I have cert requests from LetsCrypt to spare.

Creating/renewal plex.MYDOMAIN.net certificates... (plex.MYDOMAIN.net)
[Sun Dec 13 00:16:20 UTC 2020] Domains not changed.
[Sun Dec 13 00:16:20 UTC 2020] Skip, Next renewal time is: Sat Feb 6 16:59:22TC 2021
[Sun Dec 13 00:16:20 UTC 2020] Add '--force' to force to renew.
Sleep for 3600s

[mr-handerson]

nginx-proxy/acme-companion#510

If I understand correctly Gooby is crafted around the now legacy simp_le version whereas the new v2.0+ is using acme.sh which involves some significant changes.

So there's going to need to be some rewrites to conform to the new acme.sh method. Including I would assume actually installing acme https://github.com/acmesh-official/acme.sh

[Coxeroni]

I wanted to roll back to v1.13.1 and I now get the error that I have too many docker pull requests... How should I have made more than 100 pull requests in the last 6 hours?? Haven't done myself anything lately?

[bdschuster]

I wanted to roll back to v1.13.1 and I now get the error that I have too many docker pull requests... How should I have made more than 100 pull requests in the last 6 hours?? Haven't done myself anything lately?

I'm assuming you have watchtower? If so, without any interval parameters, it checks like every min I believe for a new image.

[Coxeroni]

No watchtower, just portainer and nothing else. Have to dig down on what caused this...
Edit: well yes there is watchtower, never used it. Does it come with gooby or docker as a standard?
Edit2:
Successfully rolled back now and everything is back to normal. I will keep it this way until this is sorted out here :)
If you need testers, give me a shout

[TechPerplexed]

Sorry, I hadn't been around to test things these past few days. Which is actually just as well, because today I found that my ban has been lifted and everything is working just fine with the new settings (that is, the added line of @Adoruta - not the older certificate as per @bdschuster to be specific).

So here is my new theory now: probably when switching to the new location, it's possible to run into a ban because Let's Encrypt has to pull the certificates YET AGAIN which gets us that dreaded ban. I'm hoping (thinking) that from now on it will behave and that yours will be working fine too soon, @Coxeroni

Oh and to answer your last question: yes, Watchtower is installed by default because up to now, this never caused any issues, and it's convenient to always have the latest version of containers. As with everything, the yaml file can easily be deleted and it'll never(ish) come back again :)

[Coxeroni]

Well docker introduced rate limits recently and it seems I somehow hit that. But since it worked now this is probably nothing to worry about.

I didn't have the patience to wait for the new settings to work, but I might also upgrade at one point again iwhen the dust has settled.

[TechPerplexed]

Right, we'll know more in a few weeks. But the fact that @christeasdale hasn't experienced these issues any longer, and mine is behaving too now is giving me some hope... Tentatively haha!

[christeasdale]

Right, we'll know more in a few weeks. But the fact that @christeasdale hasn't experienced these issues any longer, and mine is behaving too now is giving me some hope... Tentatively haha!

So I did an rclean earlier today and with all the changes applied from this thread I am still good. It is checking the local folder and not pulling new certs from LetsCrypt. I guess the real question is will it still automatically update the certs when they are close to expiring. I won't know until early February. @TechPerplexed I will leave it up to you if you want to leave this open.

[TechPerplexed]

Fantastic! I haven't had any problems since either. I will close this issue for now. Just a heads up (@Coxeroni & others): I am seriously considering a minor release version to pull the new yaml files, probably early in the new year. So if you don't want those changes, make sure you revert back to your current version afterwards :)

Once again, many thanks for your troubleshooting skills and code contribution, @Adoruta - it's appreciated!

[Coxeroni]

If the fix works, I will gladly have it updated. I was just getting nervous that none of my subpages were working anymore ;)