How to generate gRPC certificates for hosts that are not localhost?

[kiwiidb]

Issue and Steps to Reproduce

I'm running CLN on a Kubernetes cluster, with host cln1.regtest.getalby.com and other internal hostnames. I have the certificates but when I try to connect to the host I get the following error:

certificate is valid for ingress.local, not cln1.regtest.getalby.com.

[kiwiidb]

@cdecker , do you know where I could find some documentation on how to do this?

[cdecker]

That is very strange, ingress.local sounds a lot like it's going through some kind of service mesh or load balancer. Sadly I don't have much experience running cln-grpc on kubernetes, so I'm not sure how we could debug this issue.

Maybe openssl s_client can be used to see a) what IP the host resolves to, b) if that matches any IP of the kube host or pod in the service overlay, c) what the server claims it's name is and if it matches our expectation.

Fwiw cln-grpc doesn't really use real hostnames when the cert is autogenerated.

[kiwiidb]

I am trying to connect through a Load Balancer / Ingress controller ingress-nginx. I figured I should connect on port 443 instead of 80, and I've changed some annotations:

      kubernetes.io/ingress.class: nginx
      nginx.ingress.kubernetes.io/backend-protocol: GRPCS
      nginx.ingress.kubernetes.io/ssl-passthrough: 'true'

and I'm getting (slightly different than last time):

certificate is valid for cln, localhost, not cln1.regtest.getalby.com

Curiously, cln and localhost are exactly the hostnames that are past in here.
In LND, you also have to set the hostnames for the certificates, so shouldn't this be done dynamically here as well?

[cdecker]

That is indeed much more reasonable, since as you noted cln and localhost are the hostnames we configure when generating the certificates.

It also seems like the change in configuration is now using SNI to pass through the encrypted stream instead of acting as a cleartext proxy (which'd open its own TLS connection to the plugin, thus presenting its own cert to the client, and potentially failing due to the domain name verification anyway). So this makes a lot of sense, and I wasn't aware the nginx ingress even allowed this HTTPS level proxying, I thought it'd mostly do TCP proxying which'd be transparent.

I like your idea of adding tlsextradomains or even the main domain as a CLI parameter. What do you think should the semantics for it be? I can think of the following variants:

• If a certificate is present on disk, don't try to generate one
• If a certificate is present on disk, check that the domain matches with the passed in parameter, and generate a new one if it doesn't match
• Always generate a new certificate, independently from whether there is one

I personally prefer the first, as it is less likely to accidentally overwrite a manually created certificate with one that may not have all the bells and whistles attached (extra domains, key parameters, etc), while the other two are a bit more self-contained.

The fact that we don't generate certificates if they are already present allows you already to generate one using openssl or cfssl (which has a much simpler API), which can include your domain.

[vincenzopalazzo]

Do you think that the first one make the overriding certificate a manual process? I think that the second point has a lot of good reason to have implemented in.

BTW, also the last one sounds good if the certificate generate with the same domain is always equal.

[cdecker]

Do you think that the first one make the overriding certificate a manual process?

Yes, you'd delete the certs and then restart the node (no worries the plugin caches the certs on load, so you can just delete them while it's running).

BTW, also the last one sounds good if the certificate generate with the same domain is always equal.

I don't like this one too much since it'd prevent operators from using custom certificates that are maintained externally. Think adding swapping out the CA to be a real TLD, or configuring the TLS parameters such as hashing algos and encryption schemes, or any other of the thousands of openssl options that we definitely don't want to shoehorn into the cln-grpc and lightningd command line flags.

[vincenzopalazzo]

I don't like this one too much since it'd prevent operators from using custom certificates that are maintained externally. Think adding swapping out the CA to be a real TLD, or configuring the TLS parameters such as hashing algos and encryption schemes, or any other of the thousands of openssl options that we definitely don't want to shoehorn into the cln-grpc and lightningd command line flags.

Ah right, so now the option sounds less good

[kiwiidb]

First option sounds good to me.

BTW, got it to work with cfssl using this example on how to generate mtls certs.

[rvullriede]

I just ran unto this issue when I've tried to connect to my cln node via GRPC the first time.
When I checked the issued certificate it noticed that it still only includes "cln" and "localhost" as "Subject Alternative Name"

I'd second the idea to align with LND and allow to set (at least one) tlsextraip and/or tlsextradomain via config (file or CLI). In combination with only (re-)generating the certificates if they are missing that should cover most cases.

(When I first encountered the same issue with LND a while ago I didn't understand why they didn't simply include the hosts private network IP per default but that might go to far and would only cover very simple cases, mostly with home-grade setups with only one IP etc).