-
Notifications
You must be signed in to change notification settings - Fork 4.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
TcpProxy improperly sets :protocol=bytestream
for HTTP2
#20378
Comments
cc @lambdai |
cc @alyssawilk |
Thank @stevenctl for drafting this. @alyssawilk Also, if any protocol override is needed we can relax This relax aligns with other requirements of setting host headers in tunnel. I have some code implementing the relaxation without impacting the HCM router |
Additional issue: For HTTP 2 CONNECT rfc7540 8.3:
AFAICT I can't use xDS to make Envoy initiate a proper H2 CONNECT. |
I don't believe that your assesment is correct. The It looks that Go doesn't support the "extended" version, and since it's an extension of the HTTP/2 protocol, it should be negotiated using However, you cannot really proxy TCP over HTTP/2 without it (or without reinventing a custom framing or another method for that purpose), so it's unclear what should be the fallback. |
I believe there is an option to use POST ( for 'legacy' servers ) - and it may be safer to default to it, unless it is explicitly known that the other side supports extended connect. POST will work with almost all http/2 servers, and has equivalent behavior (connect_config: allow_post: true) |
Right, bytes can be sent using |
The upstream headers are prepared after the connection is established. Theoretically we can relies on the negotiated settings. I am wondering if peeking the negotiated result is overkill. |
I don't know if 'POST requires prior knowledge' - it is the most boring POST request, all servers and clients are supposed to support this without any other special handshake. Using extended CONNECT does require knowing that the other side support the extension - POST doesn't since it's universally supported. No doubt extended CONNECT is more fancy and likely better matching the use case - but why take the extra complexity when a much simpler option exists that works with all existing infrastructure ? |
This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions. |
This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as "help wanted" or "no stalebot". Thank you for your contributions. |
It's not resolved: I think we have two approach to connect to a server that is not catching up with the extended CONNECT (https://datatracker.ietf.org/doc/html/rfc8441#section-3)
|
We did some investigation of the RFCs and I believe this is not the expected protocol to use in this case (or, at least how we want to use CONNECT). There are a few relevant RFCs here:
In both of these protocols, negotiation is supposed to be done via SETTINGS_ENABLE_CONNECT_PROTOCOL. We are hoping to proxy application traffic over a CONNECT tunnel (eg app -> envoy proxy, connect encap -> envoy proxy, connect decap --> app). This currently works fine because we have Envoy on each end, but we have experimented with substituting either end with a non-Envoy proxy and see issues there. |
cc @wrowe @DavidSchinazi for thoughts |
Also was the first link supposed to be https://datatracker.ietf.org/doc/html/rfc8441 ? |
Setting |
The regular CONNECT method cannot be used for proxying TCP over HTTP/2, since it requires server to open a TCP connection to the destination, and not to layer it on top of HTTP/2.
This is missing quite a bit of context. RFC8441 supports only WebSockets because that was the use case that needed immediate solution at the time. Before it was finalized, I approached Patrick about adding As such, I believe that using the extended CONNECT method with |
I am not sure I understand this. Its in the HTTP/2 spec https://datatracker.ietf.org/doc/html/rfc7540#section-8.3. I have written multiple HTTP2 CONNECT servers/clients in multiple different languages - all seems fine and well supported. Go and rust's hyper library both have code explicitly to handle HTTP/2 CONNECT. AFAIK Chrome also supports this.
As far as I know Envoy is the only software that uses this protocol, and there is no standard for this. Is CONNECT support in Envoy intended to be an Envoy specific protocol not interoperable with other softwares? |
Yes, regular CONNECT is well supported, but it requires server to establish a TCP connection to the destination, i.e.
whereas we want to proxy TCP over HTTP/2 across multiple hops, i.e.
That is, regular CONNECT is terminating HTTP/2, whereas extended CONNECT is not.
It is meant to be interoperable, but the |
@DavidSchinazi ^ Piotr has a valid point for a HTTP2 tunneling proxy. Regular connect implies termination so we can't chain them without violating RFC. |
Thanks, I think I finally get why the That being said, it still seems like regular CONNECT is useful. First, isn't this case perfectly valid for Envoy: Second, what we want is TCP over HTTP/2 across multiple hops, but not all hops. It seems like the concern is the RFC says
And we are not establishing a raw TCP connection but rather another TCP-over-HTTP/2 connection? But in practice, what we want is
Which you could interpret still as violating that part of the RFC. However, I think you could also view it as perfectly fine. The fact we have multiple hops is an implementation detail; the RFC doesn't say that a "proxy" must be a single process or cannot other than establish a raw TCP connection (most do a lot more than this, authz, etc). So really our diagram could look like this:
Which seems perfectly fine. Even if this were to not be 110% RFC compliant (not convinced it isn't, but if), in practice this is supported by a ton of systems. The |
What is important is that any non-istio proxy can understand the protocol in question. So regular connect is fine at the edges of the mesh for termination, but within the mesh, the expectation is to anticipate another hop. Any other proxy would behave correctly at the edge, but would not do the right thing within the mesh, like Piotr described. |
At any hop we can make any decision, so I don't think we always know the next hop is HTTP/2 or raw TCP. Also I don't even agree the spec says the next hop cannot be
All it says is the payload must be transmitted over a TCP connection, it doesn't say how. TCP over HTTP/2 is "transmitting the payload over a TCP connection". Also, my understand of Envoy philosophy is that downstream and upstream decoupled. It seems the concern is about receiving CONNECT downstream and then sending CONNECT upstream as well -- isn't that not really Envoy's choice to make? If we chose to configure that and it violates some RFC, that doesn't seem like Envoy's concern. |
It's pretty clear what the intent is, though.
So the plan is to move away from the extended CONNECT with |
I am not sure I am following how regular CONNECT can be not interoperable? I can definitely see arguments that it doesn't follow the RFC - but the issue here is we are doing HTTP2 on downstream and upstream, right? From a downstream client perspective, they send the CONNECT and their TCP payload eventually makes it to where they requests; whether the intermediate steps are regular CONNECT or extended CONNECT or plain TCP or FooBarProtocol is invisible to them. From the upstream server perspective, they just receive a standard CONNECT request - they don't know/care that its the first hop or Nth hop this TCP payload has traversed. |
So why exactly do you want to replace extended CONNECT with |
A few reasons...
Our requirements are to use CONNECT for each hop and to interoperate with non-Envoy clients/servers (Java, Go, etc). These other clients do support |
It's not. Can you point me to a single widely used software that does something else than raw TCP for regular CONNECT?
I can guarantee you that it would not be easier, and considering that the core HTTP RFCs just went through a multi-year rewrite, I don't see amending regular CONNECT as a valid option. Also, adding
Agreed. But adding regular CONNECT in addition to extended CONNECT is not what this bug or @alyssawilk's PR (#21440) is doing. Both are about replacing extended CONNECT with regular CONNECT.
If we don't care about true interoperability, and only about what's visible at the edges, then again, why is it a problem that Envoy uses extended CONNECT with
If we're only talking about client/servers, then why are you even using CONNECT and not raw TCP? |
I am perfectly fine with supporting both with some flag.
Say we have C -> P1 -> P2 -> P3 -> P4 -> S. At each hop, proxy can chose to send CONNECT or raw TCP. Our Envoy hops would send CONNECT in most cases, but non-envoy hops are not required to if they refuse to send CONNECT requests in response to downstream CONNECT requests. Each proxy is 100% independent - they can recieve TCP or CONNECT and send TCP or CONNECT - they don't care what other hops do.
We have more than just Envoy's in our mesh, we want to support arbitrary non-Envoy components of the mesh which can use our protocol. |
I think we're going in circles, but how exactly is invalid usage of regular CONNECT better than proper usage of the extended CONNECT with not-standardized |
Yep, I think so. Probably best to let @DavidSchinazi weigh in - I think we have both expressed our opinions here.
HaProxy config. I then have another proxy on 127.0.0.1:15008 (Go)
So HAProxy is accepting a CONNECT request and sending a CONNECT request in response; this is what we want Envoy to do. |
any workaround to apply? the websocket Request Envoy send to a golang upstream http server always be rejected now with below: "2022/05/31 21:18:40 http2: invalid pseudo headers: invalid pseudo-header ":protocol"" |
Sorry for the delay, I was on vacation last week. My apologies, but @PiotrSikora your understanding of CONNECT is incorrect. Here is the relevant text from the latest spec:
The only correct and standardized way to tunnel TCP in HTTP is regular CONNECT. The is no Extended CONNECT |
Or from a different perspective: "extended CONNECT" extends the normal
CONNECT, which is intended for proxying HTTPS/TCP
with extra functionality, like WS or UDP. It is not deprecating or
replacing the original CONNECT, just extends it.
…On Tue, May 31, 2022 at 8:21 AM David Schinazi ***@***.***> wrote:
Sorry for the delay, I was on vacation last week. My apologies, but
@PiotrSikora <https://github.com/PiotrSikora> your understanding of
CONNECT is incorrect. Here is the relevant text from the latest spec
<https://datatracker.ietf.org/doc/html/draft-ietf-httpbis-semantics-19#section-9.3.6>
:
CONNECT is intended for use in requests to a proxy. The recipient
can establish a tunnel either by directly connecting to the server
identified by the request target or, if configured to use another
proxy, by forwarding the CONNECT request to the next inbound proxy.
An origin server MAY accept a CONNECT request, but most origin
servers do not implement CONNECT.
The only correct and standardized way to tunnel TCP in HTTP is regular
CONNECT. The is no Extended CONNECT :protocol that is defined for
proxying TCP.
—
Reply to this email directly, view it on GitHub
<#20378 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAAUR2VOSQPLLUKH2MKAEODVMYU5ZANCNFSM5Q42HXRA>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
This is specifically about non-websocket CONNECT usage. For websockets, Envoy is correct - |
Thanks Howard! very clear now. |
RFC 7540 obseleted by 9113. CONNECT now covered in Section 8.5 https://datatracker.ietf.org/doc/html/rfc9113#section-8.5 |
@moderation RFC9113 still refers to "TCP streams", so it doesn't clarify things. The draft RFC that David mentioned is an updated HTTP semantics which is decoupled from the underlying transport. I think the concern is that this new HTTP semantics may not be universally supported, but same can be said about Extended CONNECT. |
Ah yes. Latest HTTP Semantics RFC 9110 - https://datatracker.ietf.org/doc/html/rfc9110 |
#21613 thanks! |
The latest documents (in particular RFC 9110 - HTTP Semantics and RFC 9113 - HTTP/2) do not actually change the meaning of what was there before, they mainly explain things better. In particular, the meaning of
|
Risk Level: Medium (data plane changes) Testing: updated tests Docs Changes: n/a Release Notes: inline Runtime guard: envoy.reloadable_features.use_rfc_connect Fixes #20378 Signed-off-by: Alyssa Wilk <alyssar@chromium.org>
Risk Level: Medium (data plane changes) Testing: updated tests Docs Changes: n/a Release Notes: inline Runtime guard: envoy.reloadable_features.use_rfc_connect Fixes envoyproxy#20378 Signed-off-by: Alyssa Wilk <alyssar@chromium.org> Signed-off-by: Tianyu Xia <tyxia@google.com>
Risk Level: Medium (data plane changes) Testing: updated tests Docs Changes: n/a Release Notes: inline Runtime guard: envoy.reloadable_features.use_rfc_connect Fixes envoyproxy#20378 Signed-off-by: Alyssa Wilk <alyssar@chromium.org> Signed-off-by: Amila Senadheera <amila.15@cse.mrt.ac.lk>
TcpProxy improperly sets
:protocol=bytestream
for HTTP2: When usingTunnelingConfig
for an HTTP/2 tunnel, it send an out-of-spec psuedo header.Description:
https://github.com/envoyproxy/envoy/blob/main/source/common/tcp_proxy/upstream.cc#L282-L285
Envoy will always set
:protocol
, but it's not part of the HTTP/2 spec:An extended CONNECT spec does mention it however:
This becomes a problem for picky implementations, like Go/Golang's net/http:
Repro steps:
GODEBUG="http2debug=2"
TunnelingConfig
(UsePost=false)Http2ProtocolOptions
AllowConnect
and astatic
endpoint which is a Golang HTTP2 server.http2: invalid pseudo headers: invalid pseudo-header ":protocol"
The text was updated successfully, but these errors were encountered: