https backends' certificates are not verified #4503
Comments
sebastianblunt
changed the title
sebastianblunt
changed the title
|
@sebastianblunt please update to 0.26.1 and make sure you enable the setting
This was added in 0.26.0 - #4327 |
|
Thanks, I hadn't seen those new annotations. Setting For this use case, I don't need mutual authentication, however I can put dummy values into I presume the |
That means there is an issue with the format of the certificate. Please check two things:
|
|
@aledbf the error message @sebastianblunt included makes it pretty clear that the controller is looking for a private key in that file, not an x509 certificate. Is there no way for the ingress controller to verify the backend cert without also enabling client auth? Also, if the |
|
And it should be removed from https://github.com/kubernetes/ingress-nginx/blob/master/docs/user-guide/nginx-configuration/annotations.md... |
|
If |
# from https://kubernetes.github.io/ingress-nginx/examples/PREREQUISITES/
openssl req -x509 -sha256 -newkey rsa:4096 -keyout ca.key -out ca.crt -days 356 -nodes -subj '/CN=Fake CA'
openssl req -new -newkey rsa:4096 -keyout server.key -out server.csr -nodes -subj '/CN=valid'
openssl x509 -req -sha256 -days 365 -in server.csr -CA ca.crt -CAkey ca.key -set_serial 01 -out server.crt
kubectl create secret generic tls-secret --from-file=tls.crt=server.crt --from-file=tls.key=server.key
kubectl create secret generic ca-secret --from-file=tls.crt=server.crt --from-file=tls.key=server.key --from-file=ca.crt=ca.crt
echo "
apiVersion: apps/v1
kind: Deployment
metadata:
name: test-deployment
labels:
app: test-deployment
spec:
replicas: 1
selector:
matchLabels:
app: test-deployment
template:
metadata:
labels:
app: test-deployment
spec:
containers:
- name: main
image: nicolaka/netshoot
command: ['bash', '-c']
args:
- 'while true; do openssl s_server -key key -cert /data/tls.crt -key /data/tls.key -accept 443 -www; done'
ports:
- containerPort: 443
volumeMounts:
- mountPath: /data
name: tls-secret
readOnly: true
volumes:
- name: tls-secret
secret:
secretName: tls-secret
---
apiVersion: v1
kind: Service
metadata:
name: test-service
spec:
selector:
app: test-deployment
ports:
- protocol: TCP
port: 443
targetPort: 443
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: test-ingress
annotations:
nginx.ingress.kubernetes.io/backend-protocol: HTTPS
nginx.ingress.kubernetes.io/configuration-snippet: |
# this is required because the CN of the generated certificate.
# or nginx sends the content of $proxy_host https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_ssl_name
proxy_ssl_name valid;
nginx.ingress.kubernetes.io/proxy-ssl-secret: default/ca-secret
nginx.ingress.kubernetes.io/proxy-ssl-verify: 'on'
#nginx.ingress.kubernetes.io/proxy-ssl-protocols: TLSv1.1
spec:
rules:
- host: test-deployment
http:
paths:
- path: /valid
backend:
serviceName: test-service
servicePort: 443
- path: /invalid
backend:
serviceName: test-invalid-service
servicePort: 443
" | kubectl apply -f -Get ingress controller pod name: export POD=$(kubectl get pods -n ingress-nginx -l app.kubernetes.io/name=ingress-nginx -o jsonpath="{.items[0].metadata.name}")Run a test: kubectl exec -n ingress-nginx $POD -- curl localhost/valid -H 'Host: test-deployment'
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0<HTML><BODY BGCOLOR="#ffffff">
<pre>
s_server -key key -cert /data/tls.crt -key /data/tls.key -accept 443 -www
Secure Renegotiation IS supported
Ciphers supported in s_server binary
TLSv1.3 :TLS_AES_256_GCM_SHA384 TLSv1.3 :TLS_CHACHA20_POLY1305_SHA256
TLSv1.3 :TLS_AES_128_GCM_SHA256 TLSv1.2 :ECDHE-ECDSA-AES256-GCM-SHA384
TLSv1.2 :ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 :DHE-RSA-AES256-GCM-SHA384
TLSv1.2 :ECDHE-ECDSA-CHACHA20-POLY1305 TLSv1.2 :ECDHE-RSA-CHACHA20-POLY1305
TLSv1.2 :DHE-RSA-CHACHA20-POLY1305 TLSv1.2 :ECDHE-ECDSA-AES128-GCM-SHA256
TLSv1.2 :ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 :DHE-RSA-AES128-GCM-SHA256
TLSv1.2 :ECDHE-ECDSA-AES256-SHA384 TLSv1.2 :ECDHE-RSA-AES256-SHA384
TLSv1.2 :DHE-RSA-AES256-SHA256 TLSv1.2 :ECDHE-ECDSA-AES128-SHA256
TLSv1.2 :ECDHE-RSA-AES128-SHA256 TLSv1.2 :DHE-RSA-AES128-SHA256
TLSv1.0 :ECDHE-ECDSA-AES256-SHA TLSv1.0 :ECDHE-RSA-AES256-SHA
SSLv3 :DHE-RSA-AES256-SHA TLSv1.0 :ECDHE-ECDSA-AES128-SHA
TLSv1.0 :ECDHE-RSA-AES128-SHA SSLv3 :DHE-RSA-AES128-SHA
TLSv1.2 :RSA-PSK-AES256-GCM-SHA384 TLSv1.2 :DHE-PSK-AES256-GCM-SHA384
TLSv1.2 :RSA-PSK-CHACHA20-POLY1305 TLSv1.2 :DHE-PSK-CHACHA20-POLY1305
TLSv1.2 :ECDHE-PSK-CHACHA20-POLY1305 TLSv1.2 :AES256-GCM-SHA384
TLSv1.2 :PSK-AES256-GCM-SHA384 TLSv1.2 :PSK-CHACHA20-POLY1305
TLSv1.2 :RSA-PSK-AES128-GCM-SHA256 TLSv1.2 :DHE-PSK-AES128-GCM-SHA256
TLSv1.2 :AES128-GCM-SHA256 TLSv1.2 :PSK-AES128-GCM-SHA256
TLSv1.2 :AES256-SHA256 TLSv1.2 :AES128-SHA256
TLSv1.0 :ECDHE-PSK-AES256-CBC-SHA384 TLSv1.0 :ECDHE-PSK-AES256-CBC-SHA
SSLv3 :SRP-RSA-AES-256-CBC-SHA SSLv3 :SRP-AES-256-CBC-SHA
TLSv1.0 :RSA-PSK-AES256-CBC-SHA384 TLSv1.0 :DHE-PSK-AES256-CBC-SHA384
SSLv3 :RSA-PSK-AES256-CBC-SHA SSLv3 :DHE-PSK-AES256-CBC-SHA
SSLv3 :AES256-SHA TLSv1.0 :PSK-AES256-CBC-SHA384
SSLv3 :PSK-AES256-CBC-SHA TLSv1.0 :ECDHE-PSK-AES128-CBC-SHA256
TLSv1.0 :ECDHE-PSK-AES128-CBC-SHA SSLv3 :SRP-RSA-AES-128-CBC-SHA
SSLv3 :SRP-AES-128-CBC-SHA TLSv1.0 :RSA-PSK-AES128-CBC-SHA256
TLSv1.0 :DHE-PSK-AES128-CBC-SHA256 SSLv3 :RSA-PSK-AES128-CBC-SHA
SSLv3 :DHE-PSK-AES128-CBC-SHA SSLv3 :AES128-SHA
TLSv1.0 :PSK-AES128-CBC-SHA256 SSLv3 :PSK-AES128-CBC-SHA
---
Ciphers common between both SSL end points:
ECDHE-ECDSA-AES256-GCM-SHA384 ECDHE-RSA-AES256-GCM-SHA384 DHE-RSA-AES256-GCM-SHA384
ECDHE-ECDSA-CHACHA20-POLY1305 ECDHE-RSA-CHACHA20-POLY1305 DHE-RSA-CHACHA20-POLY1305
ECDHE-ECDSA-AES128-GCM-SHA256 ECDHE-RSA-AES128-GCM-SHA256 DHE-RSA-AES128-GCM-SHA256
ECDHE-ECDSA-AES256-SHA384 ECDHE-RSA-AES256-SHA384 DHE-RSA-AES256-SHA256
ECDHE-ECDSA-AES128-SHA256 ECDHE-RSA-AES128-SHA256 DHE-RSA-AES128-SHA256
ECDHE-ECDSA-AES256-SHA ECDHE-RSA-AES256-SHA DHE-RSA-AES256-SHA
ECDHE-ECDSA-AES128-SHA ECDHE-RSA-AES128-SHA DHE-RSA-AES128-SHA
AES256-GCM-SHA384 AES128-GCM-SHA256 AES256-SHA256
AES128-SHA256 AES256-SHA AES128-SHA
Signature Algorithms: ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:Ed25519:Ed448:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA224:RSA+SHA224:DSA+SHA224:DSA+SHA256:DSA+SHA384:DSA+SHA512
Shared Signature Algorithms: ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:Ed25519:Ed448:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA224:RSA+SHA224:DSA+SHA224:DSA+SHA256:DSA+SHA384:DSA+SHA512
Supported Elliptic Groups: X25519:P-256:X448:P-521:P-384
Shared Elliptic groups: X25519:P-256:X448:P-521:P-384
---
No server certificate CA names sent
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES256-GCM-SHA384
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES256-GCM-SHA384
Session-ID:
Session-ID-ctx: 01000000
Master-Key: 24E0267B752D3515353D3A7A1B3E73110A07EBE083BC506E31A724CBC988299BE2B772898F662A11058F74F1A7DBB3C4
PSK identity: None
PSK identity hint: None
SRP username: None
Start Time: 1569897752
Timeout : 7200 (sec)
Verify return code: 0 (ok)
Extended master secret: yes
---
0 items in the session cache
0 client connects (SSL_connect())
0 client renegotiates (SSL_connect())
0 client connects that finished
1 server accepts (SSL_accept())
0 server renegotiates (SSL_accept())
1 server accepts that finished
0 session cache hits
0 session cache misses
0 session cache timeouts
0 callback cache hits
0 cache full overflows (128 allowed)
---
no client certificate available
</pre></BODY></HTML>
100 4932 0 4932 0 0 535k 0 --:--:-- --:--:-- --:--:-- 535k
echo "
apiVersion: apps/v1
kind: Deployment
metadata:
name: test-deployment2
labels:
app: test-deployment2
spec:
replicas: 1
selector:
matchLabels:
app: test-deployment2
template:
metadata:
labels:
app: test-deployment2
spec:
containers:
- name: main
image: nicolaka/netshoot
command: ['bash', '-c']
args:
- 'openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout key -out cert -subj /CN=test-deployment && while true; do openssl s_server -key key -cert cert -accept 443 -www; done'
ports:
- containerPort: 443
---
apiVersion: v1
kind: Service
metadata:
name: test-invalid-service
spec:
selector:
app: test-deployment2
ports:
- protocol: TCP
port: 443
targetPort: 443
" | kubectl apply -f -
4. check the certificate verification fails
```console
kubectl exec -n ingress-nginx $POD -- curl localhost/invalid -H 'Host: test-deployment'
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
10<html>0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>openresty/1.15.8.2</center>
</body>
</html>
0 163 100 163 0 0 10866 0 --:--:-- --:--:-- --:--:-- 11642From the ingress controller logs: |
Pull request are welcome |
Right now the cert and key are not optional https://github.com/kubernetes/ingress-nginx/blob/master/rootfs/etc/nginx/template/nginx.tmpl#L820 |
|
Hi @aledbf, Thanks for confirming the expected behavior. I think the biggest issue here is that the old option, Additionally, I think a security advisory is in order here to alert users that this configuration option is now silently ignored, along with a patch release that will switch the behavior to failing hard. |
|
#4688 is to fix the one-way authentication issue when only the ca.crt is in the secret referred by the annotation proxy-ssl-secret. |
|
Closing. Fixed in #4689 |
Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/.):
What keywords did you search in NGINX Ingress controller issues before filing this one? (If you have found any duplicates, you should instead reply there.):
(found #2680 which is closed)
Is this a BUG REPORT or FEATURE REQUEST? (choose one):
BUG REPORT
NGINX Ingress controller version: Tested on 0.24.1 and 0.26.1
Kubernetes version (use
kubectl version):Environment:
uname -a):Darwin SebsMacbook.local 18.7.0 Darwin Kernel Version 18.7.0: Thu Jun 20 18:42:21 PDT 2019; root:xnu-4903.270.47~4/RELEASE_X86_64 x86_64What happened:
I am unable to get nginx-ingress-controller to reject a backend certificate. Specifying one certificate in the
nginx.ingress.kubernetes.io/secure-verify-ca-secretannotation for an ingress, and having the corresponding backend serve up a completely different (and self-signed) certificate does not lead to any error.If setting the annotation to a non-existent secret, you get the following logs:
If the secret does exist, I see:
What you expected to happen:
Backend certificates that do not exactly match the certificate specified in
secure-verify-ca-secretto be rejected.How to reproduce it (as minimally and precisely as possible):
Apply the following:
and run
Anything else we need to know:
The now-deleted documentation for secure-verify-ca-secret said
Currently there is no documentation for
secure-verify-ca-secretexcept that the annotation exists.Even if it is possible to get it to reject certificates, I would still expect the behavior of secure-verify-ca-secret to be fail by default (if the secret is somehow misconfigured or missing), or at the very least give an error in that case.
The text was updated successfully, but these errors were encountered: