Request forwarding in CCF

[achamayou]

Historically, CCF has implemented transparent request forwarding of from replicas to primary, and made this forwarding sticking to preserve session consistency (ie. a read in a session should always observe all previous writes in the same session).

CCF endpoints have also on occasion (after the transition to HTTP) implemented HTTP-level redirect (/node/network/self, /node/network/primary and more recently /node/join), where transparent forwarding wasn't suitable. This discussion is hopefully a collective answer to the question:

Should CCF always use HTTP-level redirect as a forwarding mechanism?

I think we would like to gather the following facts before discussion:

1. What's the redirect behaviour for the most popular HTTP client libraries in the most popular client languages (JS, Java, C#, Python)? Is it handled by the library? If so, it is on by default?
2. Is redirect-only compatible with typical load-balancer deployments in the cloud?

[achamayou]

Python

• requests: redirect on by default for GET/OPTIONS/POST/PUT/PATCH/DELETE/HEAD, can be turned off manually.
• httpx: only redirect when user specifies follow_redirects=True since 0.20 (behaviour identical to requests prior to 0.20), otherwise off by default.
• msrest (used by the ACL SDK): redirect by default

JavaScript

• browser: fetch() redirects by default, in compliance with the spec. Motivation is not just usability but also security
• node-fetch: Same as browser, follows redirect by default.
• axios: Defaults to following redirects 5 levels down.
• request: Deprecated, but still widely used it seems, and follows redirects by default.
• got: Follows by default
• azure-sdk-for-js (used by the ACL SDK): redirect by default

C#

• System.Net.HttpWebRequest (used by the ACL SDK) redirects by default

Java

• OkHttp (which is what seems to be used by the ACL SDK, but with so much indirection that I'm not totally sure) also redirects by default.

[letmaik]

• curl needs -L
• wget follows by default, but only within the same host

[eddyashton]

A further difficulty we've discovered is that many clients, when following redirects, will remove the Authorization header if the redirect goes to a different origin.

See for example step 13 of the spec for Fetch, or MDN's Browser compatibility table for the Authorization header:

This means JWT tokens passed in the Authorization header will be lost when we redirect, either to a node address or a different load balancer name. It's possible that some implementations may only use the top-level domain to determine origin, so ccf.com -> write.ccf.com would not be considered a "different origin", but this doesn't appear to be well-specified or broadly true.

Investigating some client libraries to establish what they do.

Python

• requests: Removes Authorization header if hostname doesn't match exactly.
• httpx: Removes Authorization header if hostname doesn't match exactly.
• msrest: Unclear (now deprecated?), but uses requests under-the-hood.
• azure-sdk-for-python: Removes Authorization header via a SensitiveHeaderCleanupPolicy, inserted whenever redirects are enabled, but with a flag to disable this behaviour. PR adding this was here, which includes tests showing how you'd create a pipeline with the opt-out flag.

Javascript

• Browser Fetch: Removes Authorization header "If request’s current URL’s origin is not same origin with locationURL’s origin"
• azure-sdk-for-js: Removed, since this PR.

C#

• Authorization header is removed by design. Was briefly not removed, and this was considered a CVE.

Workarounds

There are 2 potential workarounds for this, so that we can still provide redirects in the framework. Both require client modifications:

• Use an alternative header. If we pass a JWT token in x-ccf-auth, no client will strip it on redirect. However, clients will need to construct/insert this manually, and won't benefit from default JWT authentication plugins.
• Advise clients to intercept redirects manually. All of these clients offer an option to not follow redirects automatically, and usually a way to programmatically intercept a redirect to modify the request on-demand. Using this, we can build CCF clients which follow redirects but don't strip the Authorization header when doing so.

[eddyashton]

2. Is redirect-only compatible with typical load-balancer deployments in the cloud?

I think it's likely that if redirect-only isn't acceptable, that implies that any redirects aren't acceptable, because individual nodes aren't exposed. So the built-in endpoints that we've implemented with redirects won't work in those deployments. There are ways to work around this (set all node public addresses to the load balancer's address), but in that scenario the redirects aren't doing what we expect. We also discussed a scheme where we have a load balancer for backups and a load balancer for primaries, and some way to declare the logic so that a node forwards to the primary load balancer if it believes it should forward to a primary, but that seems like a lot of work in both the framework and each deployment to support.

I think our current assumption is that redirects must be possible (nodes must be publicly accessible), and the framework can be written with that assumption, and deployments must ensure that is true. But I agree we need to confirm that's acceptable, and if so then make it an explicit requirement going forward.

[achamayou]

Yes, absolutely. It's not necessarily either/or, we could still support both mechanisms, and make the benefits and drawbacks clear: forwarding makes fewer assumption about node accessibility, but doesn't offer the same QoS/address benefits as redirect does.

The potential simplifications of the node to node protocol if it no longer is used for forwarding are quite attractive though.