Router: Websocket support over a TLS terminating reverse proxy

[marvkis]

Hi,

I've been working on helm charts openziti/helm-charts#234 to get a 'kubernetes browzer' support. Since there is (imho) no need for mTLS on the WSS endpoint, it should work through a reverse proxy doing the TLS termination.
This would make things easier in the kubernetes world, as the certificate renewal and handling works very well with the existing ingress controllers / reverse proxies. So I tried to set it up like that way for the WSS endpoint, but the communication between the NGINX reverse proxy and the WSS endpoint failed.

On the router I see this logged when I try to access the WSS endpoint:

[74385.266]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:3023]: {remote=[10.42.0.186:38284] error=[not handler for requested protocols ]]} handshake failed
[74385.266]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:3023]: {remote=[10.42.0.186:38300] error=[not handler for requested protocols ]]} handshake failed
[74385.267]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:3023]: {remote=[10.42.0.186:38302] error=[not handler for requested protocols ]]} handshake failed

These are the logs on the NGINX side:

2024/07/02 20:39:45 [error] 1384#1384: *29019299 SSL_do_handshake() failed (SSL: error:0A000438:SSL routines::tlsv1 alert internal error:SSL alert number 80) while SSL handshaking to upstream, client: 127.0.0.1, server: wss.external.freaks.de, request: "GET / HTTP/2.0", upstream: "https://10.42.0.178:3023/", host: "wss.external.freaks.de"
2024/07/02 20:39:45 [error] 1384#1384: *29019299 SSL_do_handshake() failed (SSL: error:0A000438:SSL routines::tlsv1 alert internal error:SSL alert number 80) while SSL handshaking to upstream, client: 127.0.0.1, server: wss.external.freaks.de, request: "GET / HTTP/2.0", upstream: "https://10.42.0.178:3023/", host: "wss.external.freaks.de"
2024/07/02 20:39:45 [error] 1384#1384: *29019299 SSL_do_handshake() failed (SSL: error:0A000438:SSL routines::tlsv1 alert internal error:SSL alert number 80) while SSL handshaking to upstream, client: 127.0.0.1, server: wss.external.freaks.de, request: "GET / HTTP/2.0", upstream: "https://10.42.0.178:3023/", host: "wss.external.freaks.de"
127.0.0.1 - - [02/Jul/2024:20:39:45 +0000] "GET / HTTP/2.0" 502 150 "-" "curl/8.6.0" 36 0.003 [openziti-core-router-listener-edge-wss-443] [] 10.42.0.178:3023, 10.42.0.178:3023, 10.42.0.178:3023 0, 0, 0 0.001, 0.000, 0.001502, 502, 502 3939e56ebe6d53a8734621aabba1919d

I started digging into this and playing around with openssl gave me an idea:

# doesn't work
openssl s_client -connect 10.42.0.246:3023
# works
openssl s_client -connect 10.42.0.246:3023  -alpn http/1.1

Looks like the router needs ALPN headers in the handshake. But it looks like NGINX doesn't have ALPN support on the backend side:
https://serverfault.com/questions/765258/use-http-2-0-between-nginx-reverse-proxy-and-backend-webserver

The error message we see in the logs is fired here: https://github.com/openziti/transport/blob/0666f1970ea9ab620fdffd5d902719941cb34c7d/tls/listener.go#L344

During my research I also came across traces indicating that there is a 'ws' variant, so I tried it with address: ws:0.0.0.0:3023 - but this only produces this error:

[49200.245]    INFO ziti/router/xgress_edge.(*listener).Listen: {address=[ws:0.0.0.0:3023]} starting channel listener
[49200.245]   FATAL ziti/router.(*Router).startXgressListeners: error listening [edge] (transport.ws not supported. use transport.wss)

Is there any way to make this work? Is the ALPN support a hard requirement or could it be optional? Or could the websocket port be made available over HTTP so that the termination is done at the reverse proxy and the internal traffic is unencrypted? (As I understand it, the webassembly creates an mTLS connection to the router using websocket as the transport layer).

Thanks & Bye,
Chris

[qrkourier]

As I understand the Ziti security model, edge clients must present a trusted client certificate to the router edge, WebSocket listener in this case, when they are creating a channel for a Ziti service they're authorized to dial or terminator for a service they're authorized to bind.

If that's accurate, then it's essential for the WebSocket listener to terminate TLS so that mTLS negotiation can succeed.

BrowZer (ZBR) clients obtain an ephemeral client certificate from the Ziti controller after "bootstrapping" with OpenID Connect. That's the cert they present to the WebSocket listener.

The controller's client API and the router's WebSocket listener must both present a publicly-trusted server certificate because the ZBR client is running inside a normal web browser that can not be configured to trust Ziti's root CA.

Only the router's WebSocket listener must terminate TLS, however, because ZBR clients never present the client certificate to the controller's client API.

EDIT: I stand corrected! Now I think the router's WebSocket TLS listener will work normally behind a reverse proxy that also provides server TLS. This is because no client cert is required to negotiate TLS with the WebSocket listener, only server TLS. The ZBR running inside the normal web browser will then negotiate mTLS for the edge transport protocol, presenting the ephemeral client certificate obtained from the Ziti controller, inside that WebSocket (server) TLS tunnel.

[qrkourier]

I'm working on proving this out in preparation for a development round on BrowZer deployments and docs.