-
Notifications
You must be signed in to change notification settings - Fork 2.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
proxy_protocol mode breaks HTTP01 challenge Check stage #466
Comments
If the purpose of the check process is to check external availability of the acme challenge, perhaps making the request to the external ingress IP would be the best idea? |
I'm unsure what the bug is here - according the ACME spec, cert-manager should always communicate with a challenge endpoint using HTTP. HTTPS is specifically not supported and should not be used here. cert-manager already attempts to make a request to the external address for the ingress controller (i.e. by making a request to the same IP that Letsencrypt will use when performing a challenge validation). Can you describe your DNS/ingress further? It sounds like you have some kind of split horizon DNS set up that may be tripping up cert-manager? |
|
The challenge is being requested over HTTP, but the nginx-ingress controller is expecting requests to be made using the proxy protocol - which my load balancer is configured to do. When the go-http client makes a request directly to the nginx-ingress controller (i.e. not using the load balancers external IP) the proxy_protocol isn't used, causing it to fail, If I manually run I am currently using the |
@munnerz - There is no split horizon DNS, the only issue is that the nginx-ingress controller is expecting requests using the proxy_protocol method, and the cert-manager controller is making requests to it using plain HTTP. I would suggest adding a flag that toggles cert-manager to make all requests to the nginx-ingress controller via the load balancer instead of hitting it directly. |
More troubleshooting indicates there is some strange behaviour in my cluster, using a similar setup in minikube works fine. The error message I'm seeing when I set logging to verbose is: From within the cert-manager instance, the DNS host resolves correctly:
But if I install curl within the cert-manager instance I get odd behaviour:
Whereas if I run the same command from any other host I get this:
|
@HeWhoWas i ran into the same issue today, did you find a solution? from within cert-manager pod, DNS lookups resolve the public loadbalancer IP. but it still seems that cert-manager will contact the node-ip directly to solve the acme challenge, hence does not go through the public loadbalancer |
@metallhopf Our solution was to use DNS-01 validation, bypassing these issues altogether. I believe at the time this ticket was opened, that feature was only just being introduced and wasn't an option in a release version. Sorry I can't be more help, but for what is worth we've found DNS validation to be simpler and more robust than HTTP. |
That what CM always tries to do. It hits the domain name of the certificate using HTTP. It expects that domain name will resolve to the external IP. And that's the point, to test the challenge as e.g. Let's Encrypt would, from the outside. There is no value or point in CM making extra steps to find out and use the internal node IP instead, that wouldn't be checking anything useful. If the traffic from the CM or CM challenge Pods are not going external, probably you have some special cluster DNS, network DNS, or perhaps hairpin routing getting in the way. |
you are correct, i was able to confirm this. the public loadbalancer ip is resolved correctly for the http request. but traffic is routed internally directly from the cert-manager pod to nginx-ingress-controller (does not go through the loadbalancer). cluster is hosted on digitalocean, will edit this post if i find a solution. |
got the same issue with proxy protocol at digitalocean. my only workaround is to temporarily disable proxy protocol on the load balancer (and nginx ingress config map) allowing the certificate to be issued. i'm hoping for a better solution to avoid interruption in service during certificate renewal. |
I get the same on a cluster at Brightbox, which also uses hairpinning by default and an external load balancer that sends PROXY protocol. The hairpinning causes other issues too admittedly, but it's obviously a common setting for Kubernetes clusters.
Since it's fairly common (or at least, so I believe), perhaps an option just to disable testing the challenge? The challenge will work externally, from Let's Encrypt, just not internally. |
Just hit this with a DO managed Kube cluster and a DO LB in proxy-protocol mode. Seems like the the nginx-ingress LB is broken with DO's proxy protocol: From the Ingress
Unfortunately the DNS01 challenge is broken for DigitalOcean in 0.7.0 (and based on my testing in 0.6.0 as well) so HTTP01 is a must for DO. |
I faced this one as well, why is this one closed? Cert manager http01 challenge is not working with DO load balancer with proxy protocol at the moment, which is the only load balancer that makes sense (as it forwards request IP). I guess I will try to use DNS01 until resolved, but how HTTP01 should be working, can we open this one? |
Unfortunately, DigitalOcean doesn't support DNS01. Maybe unless you're using DigitalOcean's DNS service - TBD. |
@altoning Not sure what are trying to say, I managed to get digitalocean dns01 challenge working with cert manager 0.7.0 and proxy protocol load balancer. That being said, this one still needs to get attention, as this is just a temp solution, I don't wanna have digital ocean API key in k8s, it is tied to specific digital ocean user and it stops working when the user is removed from the project (that is by design and should be so, I just don't want to tie cert manager validation to that). |
@HeWhoWas Please can you reopen this issue as it hasn't been resolved. |
I believe this is due to a design flaw in Kubernetes. When using LoadBalancer services that use an external service (like Brightbox or DO mentioned here), kube-proxy intercepts the outgoing requests to the load balancer external IP at the network level to keep them within the Kubernetes cluster but doesn't understand that some LoadBalancers can do more than just standard TCP balancing. So this will break internal connections to external load balancers that do more, such as proxy-support or even SSL offloading. We've now fixed this at Brightbox by not telling kube-proxy about the external IP addresses of the LoadBalancers, so it doesn't intercept them. I think DO are going to fix it the same way, and AWS have done this all along. See kubernetes/kubernetes#66607 for more details. So this isn't a cert-manager problem. |
Reopening due to all the conversation and at @dottodot request. If this turns out to not be a cert manager issue, happy to have it closed. |
Not sure this will help anyone, but I managed to fix this :) |
I think @johnl's reference to kubernetes/kubernetes#66607 is the cause. By luck or design, it works on AWS because the AWS k8s cloud provider code only adds the external host name not the external IP address. The good news is a patch to Kubernetes in kubernetes/kubernetes#77523 should eventually fix this for everyone. I don't think it is a |
@sorenmat is it possible you could share how you configure it on a per ingress basis? |
sure @Philio annotations:
kubernetes.io/ingress.class: "nginx"
use-proxy-protocol: "true" |
@sorenmat Thanks for the quick reply, unfortunately though it didn't work, no proxy protocol configuration at all was added to |
Same issue here. I need proxy-protocol because of client IPs so it is not a solution to disable it. If it is on, cert-manager is not working because of that pre-check. There are two solutions:
I would prefer solution No.1 because there is a reason we are checking that endpoint before asking LetsEncrypt to do the same. This check is valuable as it prevents quota issues. This is important to be resolved as people will need proxy protocol and it is bad if I see that pre-check code is at: With http_proxy=http://my.website.com:80 curl -v http://my.website.com/.well-known/acme-challenge/mwbIQcwaB9LL6wwIGRjQuRfL8cl5lFfGXocuQ3Y_fqs --haproxy-protocol I see a recent commit 099abed and PR #1850 by @kinolaev that can have connection to what we are talking about. Is this change adding PROXY header to checks or just using proxy? |
Hello @MichaelOrtho, |
Hmm, my apologies, I appear to have posted in the wrong repository. I see now that I am NOT in the official certbot repo, for some reason I thought I was. |
so need reopen this issue? |
looks like kubernetes/enhancements#1392 is merged for 1.19 |
Hi all, I ran into the same issue. I've just published It uses CoreDNS rewriting to intercept traffic that would be heading toward the external load balancer. It then adds a PROXY line to requests originating from within the cluster. This allows cert-manager's self-check to pass. It's able to do this all through DNS rewriting and spinning up a tiny HAProxy, so there's no need to wait for either kubernetes or cert-manager to fix this issue in their packages. |
Thank you very much @compumike! It works great :) |
In my setup nginx-controller is the public facing loadbalancer, there isn't any other (why would I need it) and when it is configured with "use-proxy-protocol: true", then acme self-checks in cert-manager fail. Necessity to disable use-proxy-protocol every time I expect cert-manager to create my certs or re-new them is quite inconvenient and destroyed the purpose of the tool. Looks like in this setup (I also don't use kube-proxy, why would I, if I can configure all proxying in nginx) there is no recommended solution that would make it work? |
Hi all. Same issue, in DigitalOcean k8s and proxy-protocol. |
@compumike This was the final step on 72 hour debugging journey to get my docker-compose app https accessible from Kubernetes ! THANK YOU !!! |
thak you Very Much... its really is only solved the problem since ours searching solutions |
Just wanted to let everyone know that this is still something that does pop up with DigtalOcean loadbalancers with the proxy protocol. It looks like there is some upstream activity to hopefully fix the root cause for this: kubernetes/kubernetes#66607 |
Note: Questions were resolved, please see edits below, thank you! @Jyrno42 Thanks for providing the patched cert-manager images! The fix for the proxy protocol works well with my ingress-nginx in proxy protocol mode 👍 However, I found an issue you (or someone else :D) maybe can help me with. The pod hosting the HTTP01 challenge does not start since the docker image tag is not properly populated:
Manually editing the tag to Do you know how to fix this? Maybe the auto-build needs some updates? I'm using v1.3.0 of your patched images (e.g. Thank you so much for your help and the effort you already put into fixing the problem at hand! 🎉 EDIT: So the problem with the {STABLE_DOCKER_TAG} somehow resolved by re-deploying cert-manager 🤷 |
Follow up to my post above: Looks like my issue with the acme http challenge pod having an invalid docker tag of However, there is an easy workaround by specifying the acme http challenge image via command line argument: If you use helm, you can supply this via helm values like this: image:
repository: jyrno42/cert-manager-controller
tag: v1.3.1
extraArgs:
- "--acme-http01-solver-image=jyrno42/cert-manager-acmesolver:v1.3.1" |
This works and fix the issue on scaleway too |
HTTP01 challenge for wild card certificate works, but is it recommended to use HTTP01 challenge for validating the challenges? |
@Jyrno42 Hello, your latest image v1.5.0-beta.0 has some problem when container start standard_init_linux.go:211: exec user process caused "no such file or directory" |
it works for me also, thanks |
I am yet another "me too" for this issue on Digital Ocean... even with the latest 1.22 release: but maybe this will be solved in 1.25 ❯ kubectl version
<...>
Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.8", GitCommit:"7061dbbf75f9f82e8ab21f9be7e8ffcaae8e0d44", GitTreeState:"clean", BuildDate:"2022-03-16T14:04:34Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"} My symptoms are the same as many of you:
But when fetching the URL externally, things work: ❯ curl -vvv http:///my-domain.tld/.well-known/acme-challenge/BxoPaSKOqSedr3HvG47jQGKNOxFJBTNDTUfoeNnzjEg
<...>
> GET /.well-known/acme-challenge/BxoPaSKOqSedr3HvG47jQGKNOxFJBTNDTUfoeNnzjEg HTTP/1.1
> Host: /my-domain.tld
> User-Agent: curl/7.82.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Wed, 27 Apr 2022 15:45:27 GMT
< Content-Type: text/plain; charset=utf-8
< Content-Length: 87
< Connection: keep-alive
< Cache-Control: no-cache, no-store, must-revalidate
<
* Connection #0 to host /my-domain.tld left intact
BxoPaSKOqSedr3HvG47jQGKNOxFJBTNDTUfoeNnzjEg.sw-Xjc-9d_IujG1ZEJT7Ol--ZOFyfKlHemo2LYZhDn0% I was able to move past this by:
|
Just for people coming here from Google (like me): @KeksBeskvitovich's solution also works for Hetzner Cloud by setting these annotations : load-balancer.hetzner.cloud/uses-proxyprotocol: "true"
load-balancer.hetzner.cloud/hostname: lb-subdomain.example.com and this value: controller:
config:
use-proxy-protocol: true for ingress-nginx. Afterwards, you will have both the source IP and a working cert-manager! |
what if i have multiple domains pointing to the same LB address? |
Any updates on it? |
Thanks, also facing this issue on hetzner. Fix seems to work but I have multiple domains pointing to that LB. It is also not a scalabe solution since the ingress needs to know the domain (all domains) before routes are applied and it is reasanoble to assume that these resources are not handled by the same person. Ingress routes may even be applied by CI. Is there any other way to stop the kube-proxy to alter the DNS resolution so the external IP is used? |
https://github.com/compumike/hairpin-proxy this worked for me. |
This fixed it for me, I did need to change the target server to nginx-ingress-ingress-nginx-controller.default.svc.cluster.local but it works like a charm, @compumike |
Is this a BUG REPORT or FEATURE REQUEST?:
/kind bug
What happened:
When running ingress-nginx with
use-proxy-protocol: true
, the check stage of cert-manager fails as it (appears to) communicate with the ingress controller using plain HTTP requests.What you expected to happen:
How to reproduce it (as minimally and precisely as possible):
use-proxy-protocol: true
, andproxy-real-ip-cidr: x.x.x.x
(Use the real load balancer IP) for the nginx controllerAnything else we need to know?:
Environment:
Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.0", GitCommit:"fc32d2f3698e36b93322a3465f63a14e9f0eaead", GitTreeState:"clean", BuildDate:"2018-03-26T16:55:54Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.0", GitCommit:"fc32d2f3698e36b93322a3465f63a14e9f0eaead", GitTreeState:"clean", BuildDate:"2018-03-26T16:44:10Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
nginx-ingress-controller:
cert-manager:
E0413 12:25:57.580259 1 controller.go:196] certificates controller: Re-queuing item "kube-system/therealhost.example.com" due to error processing: error waiting for key to be available for domain "therealhost.example.com": context deadline exceeded
The text was updated successfully, but these errors were encountered: