Spike on HTTPS reverse proxy for pd

[conorsch]

We want to expose pd's gRPC service over HTTPS on 443/TCP. For now, it'd be enough to use a reverse proxy that speaks h2/grpc. The goal is to use Buf Studio to query the protos. Here's an example URL: https://studio.buf.build/penumbra-zone/penumbra/penumbra.view.v1alpha1.ViewProtocolService/ChainParameters?target=https%3A%2F%2Fgrpc.testnet-preview.penumbra.zone&selectedProtocol=grpc-web (n.b. change the target URL param as necessary)

Attempt to use caddy to expose a fullnode's pd service over HTTPS. We think the h2c:// scheme would enough; see this HN comment and the official Caddy docs. Additionally, see the Buf Studio docs on CORS for necessary headers.

This work is towards longer-term efforts like #1406 & #1556. For now, we simply want to unblock external partners who are consuming the protobufs in browsers.

[conorsch]

❯ curl -I https://shadow.petrichor.guru
HTTP/2 200 
access-control-allow-origin: https://studio.buf.build
alt-svc: h3=":443"; ma=2592000
content-type: application/grpc
date: Thu, 15 Dec 2022 01:08:59 GMT
grpc-status: 12
server: Caddy
content-length: 0

That shows working TLS termination with an HTTP 200 response. That's progress! But it isn't enough to make Buf Studio happy. Why not?

Caddyfile for serving the above

shadow.petrichor.guru

# proxy pd, as grpc service
reverse_proxy h2c://127.0.0.1:8080

# CORS support for Buf Studio
header / {
  # Access-Control-Allow-Origin *
  Access-Control-Allow-Origin  https://studio.buf.build
}

[conorsch]

Progress, but still short of the mark. Here's what's running:

• ✔️ full node running at buf-studio.penumbra.zone
• ✔️ pd exposed directly on 8080/TCP
• ✔️ pd exposed via h2c reverse proxy at 443/TCP, with TLS termination handled by caddy

If I use this URL in Buf Studio, I see an error:

penumbra.view.v1alpha1.ViewProtocolService
ConnectError - 

However, in the web console for my browser, I see a 200 OK on the OPTIONS request. (Previously, a CORS error was displayed. Massaging the associated HTTP headers via the Caddyfile resolved that problem.) If you run this curl command exported from the web console, you'll see a 200 yourself locally:

curl 'https://buf-studio.penumbra.zone/penumbra.view.v1alpha1.ViewProtocolService/ChainParameters' -X OPTIONS -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:107.0) Gecko/20100101 Firefox/107.0' -H 'Accept: */*' -H 'Accept-Language: en-US,en;q=0.5' -H 'Accept-Encoding: gzip, deflate, br' -H 'Access-Control-Request-Method: POST' -H 'Access-Control-Request-Headers: content-type,x-grpc-web,x-user-agent' -H 'Referer: https://studio.buf.build/' -H 'Origin: https://studio.buf.build' -H 'DNT: 1' -H 'Connection: keep-alive' -H 'Sec-Fetch-Dest: empty' -H 'Sec-Fetch-Mode: cors' -H 'Sec-Fetch-Site: cross-site' -H 'TE: trailers' -I

That's not good enough for Buf Studio. What does work on Buf Studio is this URL, which requires that you:

1. Run buf beta studio-agent locally on CLI
2. Choose Set Agent URL and enter http://localhost:8080
3. For Target, enter http://buf-studio.penumbra.zone:8080.

For what it's worth, you can also set the Target to http://fullnode.testnet.penumbra.zone:8080 and it'll work just as well. So it looks like we can use Buf Studio, even to talk to a non-TLS'd endpoint, as long as we leverage the studio-agent. While the HTTPS setup doesn't work with Buf Studio, it does appear to serve well.

[hdevalence]

The studio-agent is running a local proxy, but the point of having solid TLS/grpc-web support for pd is so that it's possible to use without any additional infrastructure. So using studio-agent doesn't really solve the problem, because the goal isn't to make Buf Studio specifically work, it's to enable web content to make RPC requests to Penumbra nodes.

I'm a little confused about what's going on with the CORS headers there. Previously, I got Buf Studio to work against pd without any proxying, using the experimental work in #1666, and didn't have to do anything special with headers.

So it seems to me like the problem is likely somehow related to the way Caddy is being used as a proxy -- which reinforces the goal of having this functionality built into pd natively, rather than forcing people to debug Caddy issues to be able to make requests.

[hdevalence]

I rebased the experimental support I did in #1666 onto the 038-kalyke tag here: https://github.com/penumbra-zone/penumbra/tree/acme-rustls-kalyke

Maybe we could try running that on a publicly accessible machine with a DNS name, and see if it still works?

[conorsch]

Aye, giving that a shot now...

[conorsch]

New server up at pd-tls.penumbra.zone, running the code from the acme-rustls-kalyke branch. Try this URL for Buf Studio. Does that work for you, @hdevalence? Works for me:

[hdevalence]

Yep, works for me. It's a bit weird and uncomfortable that the experimental auto-https code works without extra setup, but Caddy doesn't. It might be useful to compare headers or something between them to see if we get any clues about why?

[conorsch]

Let me take a look. Downgrading caddy to HTTP1.1 may help.

[conorsch]

Downgrading caddy to HTTP1.1 may help.

That was it. Caddy defaults to HTTP2, and we explicitly set HTTP1.1 for tonic-web here:

penumbra/pd/src/main.rs

Lines 212 to 213 in 349f66f

	// Allow HTTP/1, which will be used by grpc-web connections.
	.accept_http1(true)

Try buf-studio.penumbra.zone in Buf Studio. Working well for me over here. Full caddyfile:

# Downgrade to HTTP1.1 from the default of HTTP2.
{
    servers :443 {
        protocols h1
    }
}

buf-studio.penumbra.zone

# proxy pd, as grpc service
reverse_proxy h2c://127.0.0.1:8080

# Make runtime logs more verbose
log {
        level DEBUG
}

So we can easily bolt on a temporary HTTPS solution on any testnet with this method. Will leave these two machines running for 038-kalyke, and update "pd-tls.penumbra.zone" for 039, when the time comes.

Haven't done this yet, but omitting the http1 downgrade should be possible if we're using https on the frontend. Will try that soon and note results.

[hdevalence]

Sick, nice find!

Does that config force HTTP/1 only, or allow either HTTP/1 or HTTP/2? We'll need HTTP/2 to support native grpc; in pd we serve both simultaneously.

[conorsch]

As written it's h1 only, but I just updated it to h1 h2 and that appears to be working fine still. Once we land #1666, we can do full h2 across the board, we won't need HTTP1.1 support anywhere, AFAICT.

[hdevalence]

I think we might need/want h1 support in pd to make grpc-web work, although I'm not clear on the details -- it's recommended by the tower-web docs, though.

[conorsch]

Made a small change to the cluster config, which provides automatic HTTPS for pd: https://grpc.testnet-preview.penumbra.zone. That URL works just fine in Buf Studio, and will remain working. The same setup isn't yet live for the weekly testnet, given delays in #1659, but plan of record is to cut over on the deploy for #1757. I'll handle that over there.

Leaving the pd-tls machine up the interim, since it's nice to have an HTTPS target that's running the stable testnet. As soon as 039-praxidike goes out, that machine can be cleaned up.

[conorsch]

As of 040-themisto (#1814), we now have a stable HTTPS for talking to pd: https://grpc.testnet.penumbra.zone. That URL can be used in Buf Studio with the gRPC-web connection option:

That URL will remain active as we release subsequent testnets. Similarly, https://grpc.testnet-preview.penumbra.zone exists, for the very latest work merged into main.

Cleaned up the temporary "pd-tls" and "buf-studio" hosts, since they're no longer necessary.