GRPC explicit client proxy returns 404

[zanes2016]

Envoy Returns 404 for all GRPC requests when http_proxy is Set

Title: When the http_proxy environment variable is set, Envoy returns HTTP 404. However, the connection succeeds when we use dnsmasq and transparently proxy the grpc requests to envoy.

Is this an issue due to the :authority and :path being set equal when then http_proxy port is set?

Envoy Explicit Proxy Decoded Header:

':authority', 'rpc-video-abr-agent.service.consul'
':path', 'rpc-video-abr-agent.service.consul'
':method', 'CONNECT'
'user-agent', 'grpc-go/1.11.1'

Envoy Transparent Proxy Decoded Header:

':method', 'POST'
':scheme', 'http'
':path', '/message.Abr/GetBitrate'
':authority', 'rpc-video-abr-agent.service.consul:80'
'content-type', 'application/grpc'
'user-agent', 'grpc-go/1.11.1'
'te', 'trailers'
'grpc-timeout', '29972342n'
'x-forwarded-proto', 'http'
'x-request-id', '0426547f-7998-44dc-bd72-59e74fc4e859'
'x-envoy-expected-rq-timeout-ms', '15000'

Description:

Envoy Config:

...
"filters": [
         {
          "name": "envoy.http_connection_manager",
          "config": {
           "idle_timeout": "60.000s",
           "http_filters": [
            {
             "name": "envoy.router",
             "config": {}
            }
           ],
           "http2_protocol_options": {},
           "http_protocol_options": {
            "allow_absolute_url": true
           },
           "rds": {
            "config_source": {
             "ads": {}
            },
            "route_config_name": "master"
           },
           "stat_prefix": "main_http_proxy"
          }
         }
        ]
       }
      ]
     },
...
        "name": "http-rpc-video-abr-agent",
        "domains": [
         "rpc-video-abr-agent:80",
         "rpc-video-abr-agent.service.consul:80",
         "rpc-video-abr-agent.service.consul.westeros-master.uhana.io:80",
         "rpc-video-abr-agent.westeros-master.uhana.io:80",
         "rpc-video-abr-agent",
         "rpc-video-abr-agent.service.consul",
         "rpc-video-abr-agent.service.consul.westeros-master.uhana.io",
         "rpc-video-abr-agent.westeros-master.uhana.io"
        ],
        "routes": [
         {
          "match": {
           "prefix": "/",
           "headers": [
            {
             "name": "content-type",
             "exact_match": "application/grpc"
            }
           ]
          },
          "route": {
           "cluster": "rpc-video-abr-agent-master",
           "idle_timeout": "60s"
          }
         },
         {
          "match": {
           "regex": ".*"
          },
          "route": {
           "cluster": "rpc-video-abr-agent-master",
           "idle_timeout": "60s"
          }
         },
         {
          "match": {
           "prefix": "/"
          },
          "route": {
           "cluster": "rpc-video-abr-agent-master",
           "idle_timeout": "60s"
          }
         }
        ]
       },
...

Envoy Logs (with http_proxy set) works without error:

[2018-08-20 22:33:27.353][43][debug][main] source/server/connection_handler_impl.cc:218] [C366] new connection
[2018-08-20 22:33:27.353][43][debug][http] source/common/http/conn_manager_impl.cc:190] [C366] new stream
[2018-08-20 22:33:27.353][43][debug][http] source/common/http/conn_manager_impl.cc:889] [C366][S16374404059013911312] request end stream
[2018-08-20 22:33:27.353][43][debug][http] source/common/http/conn_manager_impl.cc:490] [C366][S16374404059013911312] request headers complete (end_stream=true):
':authority', 'rpc-video-abr-agent.service.consul'
':path', 'rpc-video-abr-agent.service.consul'
':method', 'CONNECT'
'user-agent', 'grpc-go/1.11.1'

[2018-08-20 22:33:27.353][43][debug][http] source/common/http/conn_manager_impl.cc:1083] [C366][S16374404059013911312] encoding headers via codec (end_stream=true):
':status', '404'
'date', 'Mon, 20 Aug 2018 22:33:26 GMT'
'server', 'envoy'

[2018-08-20 22:33:27.353][43][debug][connection] source/common/network/connection_impl.cc:451] [C366] remote close
[2018-08-20 22:33:27.353][43][debug][connection] source/common/network/connection_impl.cc:133] [C366] closing socket: 0
[2018-08-20 22:33:27.353][43][debug][main] source/server/connection_handler_impl.cc:51] [C366] adding to cleanup list

Envoy Logs (without http_proxy set):

[2018-08-20 22:19:48.285][37][debug][http] source/common/http/conn_manager_impl.cc:190] [C138] new stream
[2018-08-20 22:19:48.285][37][debug][http] source/common/http/conn_manager_impl.cc:490] [C138][S6159270451548231813] request headers complete (end_stream=false):
':method', 'POST'
':scheme', 'http'
':path', '/message.Abr/GetBitrate'
':authority', 'rpc-video-abr-agent.service.consul:80'
'content-type', 'application/grpc'
'user-agent', 'grpc-go/1.11.1'
'te', 'trailers'
'grpc-timeout', '29972342n'

[2018-08-20 22:19:48.285][37][debug][router] source/common/router/router.cc:252] [C138][S6159270451548231813] cluster 'rpc-video-abr-agent-master' match for URL '/message.Abr/GetBitrate'
[2018-08-20 22:19:48.285][37][debug][router] source/common/router/router.cc:303] [C138][S6159270451548231813] router decoding headers:
':method', 'POST'
':scheme', 'http'
':path', '/message.Abr/GetBitrate'
':authority', 'rpc-video-abr-agent.service.consul:80'
'content-type', 'application/grpc'
'user-agent', 'grpc-go/1.11.1'
'te', 'trailers'
'grpc-timeout', '29972342n'
'x-forwarded-proto', 'http'
'x-request-id', '0426547f-7998-44dc-bd72-59e74fc4e859'
'x-envoy-expected-rq-timeout-ms', '15000'

[2018-08-20 22:19:48.285][37][debug][pool] source/common/http/http2/conn_pool.cc:97] [C171] creating stream

[zanes2016]

The behavior seems to be related to what's described here grpc/grpc-go#1243

[stale]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or other activity occurs. Thank you for your contributions.

[ghost]

I believe Envoy not supporting CONNECT is the root of the problem, and is an open issue: #1451

[stale]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or other activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as "help wanted". Thank you for your contributions.