Tunnel TCP over HTTP/2 POST

[wlhee]

Today, Envoy supports tunneling TCP over HTTP/2 CONNECT, which is awesome!

I understand that HTTP/2 CONNECT method is designed to handle such a use case. However, edge proxy might not accept CONNECT due to security concern.

Therefore, I wonder if we can extend the existing TCP proxy to support HTTP/2 POST, so that any edge proxies would be happy to accept the tunneling request.

[wlhee]

cc @htuch @alyssawilk

[alyssawilk]

While wrapping TCP in CONNECT is industry standard (and included in various RFCs) I haven't heard of any standards around POST encapsulation. Unless I'm missing one, I suspect we wouldn't want that upstream, but I can think of some fairly straight-forward changes to upstream Envoy we could make to make it fairly easy to implement a custom plugin to allow this locally.

As a side note, the security concern with CONNECT is mainly that when you decapsulate payload you're letting potentially unsanitized content into your network. I think switching to encapsulating and decapsulating via POST would just be hiding the security problem so probably more dangerous than CONNECT. You'd risk whatever proxy you're proxying with not supporting fully bidirectional data with POST (where often when the response is started, the request data is stopped)

[wlhee]

Thanks Alyssa, a custom plugin sounds awesome! How would we move forward to adding a plugin? I could help contributing if you could share some pointers with me.

[alyssawilk]

Unfortunately it's not quite as simple as adding a plug-in, I think first we'd need to make tcp pluggable.
@lambdai had started looking into that but is now focused on the in-memory proxying, but might be able to catch you up. Basically we'd want to refactor source/common/tcp_proxy/upstream.* (in place) to mimic the HTTP GenericConnectionPool and GenericUpstream, then add config to make it pluggable. Happy to explain in more detail if you want to ping me on hangouts.

[alyssawilk]

Ok, refactoring done, so I'll close this off as fixed once #13548 lands

Functionally you should be able to create your own custom upstream like the generic upstream which instead of creating TCP or HTTP, creates TCP or Your-custom-subclass which inherits from HttpUpstream and overrides setRequestEncoder to do the exact same code, only swapping POST instead of CONNECT. If you need help with this feel free to give me a ping.

[wlhee]

Thanks Alyssa, I will check it!

[wlhee]

@alyssawilk based on my experiment, it looks like TcpProxy or some cluster magic for upstream Envoy also need to adjust accordingly?

I have the following setup:

client -> (TCP) -> client Envoy TcpProxy (tunnel enabled) -> (HTTP/2 POST) -> upstream Envoy HTTP listener -> loopback cluster -> (HTTP/1?) -> upstream Envoy TcpProxy listener -> (TCP) -> server

It looks like if client Envoy TcpProxy uses H2 POST in the HttpUpstream, it stops working.
It seems like there is some logic in upstream Envoy, which will only be triggered for HTTP/2 CONNECT and bytestream to unwrap the TCP stream from the H2 stream?

Example config can be found here: #13001 (comment)