Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Users want Ambassador to be able to route on headers added by extauth #1226

Closed
iNoahNothing opened this issue Feb 14, 2019 · 9 comments
Closed
Assignees
Milestone

Comments

@iNoahNothing
Copy link
Contributor

It appears as though Envoy is not respecting a header set in the AuthService when routing to backend services.

Senario:

  • I have an external AuthService that sets the header x-database-id.
      ---
      apiVersion: ambassador/v1
      kind: AuthService
      name: authentication
      proto: http
      auth_service: ambassador-pro-auth
      allowed_request_headers:
      - "Client-Id"
      - "Client-Secret"
      - "x-database-id"
      allowed_authorization_headers:
      - "Authorization"
      - "Client-Id"
      - "Client-Secret"
      - "x-database-id"
  • I have a couple of Mappings that will route requests based off of this header:
      ---
      apiVersion: ambassador/v1
      kind:  Mapping
      name:  qotm_even_mapping
      prefix: /qotm/
      rewrite: /quote/2
      headers:
        X-Database-Id: even
      service: qotm
      ---
      apiVersion: ambassador/v1
      kind:  Mapping
      name:  qotm_odd_mapping
      prefix: /qotm/
      rewrite: /quote/3
      headers:
        X-Database-Id: odd
      service: qotm
      ---
      apiVersion: ambassador/v1
      kind:  Mapping
      name:  qotm_other_mapping
      prefix: /qotm/
      rewrite: /quote/0
      service: qotm
  • When sending a request, the AuthService correctly sets the header but the correct Mapping is not selected. Requests that have the AuthService set x-database-id: even will get routed to the Mapping qotm_other_mapping.
  • I see the header in my backend service so I know it it being set
  • You can even see it set in the response from the AuthService in Ambassador's logs at Debug level
[2019-02-14 21:01:09.294][000048][debug][http] [source/common/http/conn_manager_impl.cc:200] [C479] new stream
[2019-02-14 21:01:09.295][000048][debug][http] [source/common/http/conn_manager_impl.cc:888] [C479][S12274528733847657824] request end stream
[2019-02-14 21:01:09.295][000048][debug][http] [source/common/http/conn_manager_impl.cc:515] [C479][S12274528733847657824] request headers complete (end_stream=true):
':authority', 'domain2.nkrause.k736.net'
':path', '/qotm/?db=201'
':method', 'GET'
'user-agent', 'curl/7.63.0'
'accept', '*/*'

[2019-02-14 21:01:09.296][000048][debug][router] [source/common/router/router.cc:262] [C0][S6937176304599555599] cluster 'cluster_extauth_ambassador_pro_auth' match for URL '/qotm/?db=201'
[2019-02-14 21:01:09.296][000048][debug][router] [source/common/router/router.cc:318] [C0][S6937176304599555599] router decoding headers:
':authority', 'domain2.nkrause.k736.net'
':method', 'GET'
':path', '/qotm/?db=201'
':scheme', 'http'
'content-length', '0'
'x-forwarded-proto', 'https'
'x-forwarded-for', '10.142.15.200,10.40.65.13'
'user-agent', 'curl/7.63.0'
'x-envoy-internal', 'true'
'x-envoy-expected-rq-timeout-ms', '5000'

[2019-02-14 21:01:09.297][000048][debug][pool] [source/common/http/http1/conn_pool.cc:90] [C306] using existing connection
[2019-02-14 21:01:09.297][000048][debug][router] [source/common/router/router.cc:1000] [C0][S6937176304599555599] pool ready
[2019-02-14 21:01:09.298][000048][debug][router] [source/common/router/router.cc:602] [C0][S6937176304599555599] upstream headers complete: end_stream=false
[2019-02-14 21:01:09.298][000048][debug][http] [source/common/http/async_client_impl.cc:96] async http request response headers (end_stream=false):
':status', '200'
'x-database-id', 'odd'
'date', 'Thu, 14 Feb 2019 21:01:09 GMT'
'content-length', '0'
'x-envoy-upstream-service-time', '0'

[2019-02-14 21:01:09.299][000048][debug][client] [source/common/http/codec_client.cc:95] [C306] response complete
[2019-02-14 21:01:09.299][000048][debug][filter] [source/extensions/filters/http/ext_authz/ext_authz.cc:191] [C479][S12274528733847657824] ext_authz accepted the request
[2019-02-14 21:01:09.299][000048][debug][router] [source/common/router/router.cc:262] [C479][S12274528733847657824] cluster 'cluster_qotm' match for URL '/qotm/?db=201'
[2019-02-14 21:01:09.300][000048][debug][router] [source/common/router/router.cc:318] [C479][S12274528733847657824] router decoding headers:
':authority', 'domain2.nkrause.k736.net'
':path', '/quote/0?db=201'
':method', 'GET'
':scheme', 'http'
'user-agent', 'curl/7.63.0'
'accept', '*/*'
'x-forwarded-for', '10.142.15.200'
'x-forwarded-proto', 'https'
'x-envoy-internal', 'true'
'x-request-id', '53a9da17-a3fe-438e-9047-7aee88885881'
'x-database-id', 'odd'
'x-envoy-expected-rq-timeout-ms', '3000'
'x-envoy-original-path', '/qotm/?db=201'

[2019-02-14 21:01:09.300][000048][debug][pool] [source/common/http/http1/conn_pool.cc:79] creating a new connection
[2019-02-14 21:01:09.300][000048][debug][client] [source/common/http/codec_client.cc:26] [C480] connecting
[2019-02-14 21:01:09.300][000048][debug][connection] [source/common/network/connection_impl.cc:634] [C480] connecting to 10.43.245.117:80
[2019-02-14 21:01:09.301][000048][debug][connection] [source/common/network/connection_impl.cc:643] [C480] connection in progress
[2019-02-14 21:01:09.301][000048][debug][pool] [source/common/http/http1/conn_pool.cc:107] queueing request due to no available connections
[2019-02-14 21:01:09.301][000048][debug][pool] [source/common/http/http1/conn_pool.cc:209] [C306] response complete
[2019-02-14 21:01:09.301][000048][debug][pool] [source/common/http/http1/conn_pool.cc:247] [C306] moving to ready
[2019-02-14 21:01:09.302][000048][debug][connection] [source/common/network/connection_impl.cc:516] [C480] connected
[2019-02-14 21:01:09.302][000048][debug][client] [source/common/http/codec_client.cc:64] [C480] connected
[2019-02-14 21:01:09.302][000048][debug][pool] [source/common/http/http1/conn_pool.cc:252] [C480] attaching to next request
[2019-02-14 21:01:09.302][000048][debug][router] [source/common/router/router.cc:1000] [C479][S12274528733847657824] pool ready
[2019-02-14 21:01:09.319][000048][debug][router] [source/common/router/router.cc:602] [C479][S12274528733847657824] upstream headers complete: end_stream=false
[2019-02-14 21:01:09.319][000048][debug][http] [source/common/http/conn_manager_impl.cc:1096] [C479][S12274528733847657824] encoding headers via codec (end_stream=false):
':status', '200'
'content-type', 'application/json'
'content-length', '151'
'server', 'envoy'
'date', 'Thu, 14 Feb 2019 21:01:09 GMT'
'x-envoy-upstream-service-time', '18'

[2019-02-14 21:01:09.319][000048][debug][client] [source/common/http/codec_client.cc:95] [C480] response complete
[2019-02-14 21:01:09.319][000048][debug][pool] [source/common/http/http1/conn_pool.cc:209] [C480] response complete
[2019-02-14 21:01:09.319][000048][debug][pool] [source/common/http/http1/conn_pool.cc:247] [C480] moving to ready
[2019-02-14 21:01:09.319][000048][debug][connection] [source/common/network/connection_impl.cc:501] [C480] remote close
[2019-02-14 21:01:09.319][000048][debug][connection] [source/common/network/connection_impl.cc:183] [C480] closing socket: 0
[2019-02-14 21:01:09.319][000048][debug][client] [source/common/http/codec_client.cc:82] [C480] disconnect. resetting 0 pending requests
[2019-02-14 21:01:09.319][000048][debug][pool] [source/common/http/http1/conn_pool.cc:123] [C480] client disconnected
[2019-02-14 21:01:09.348][000048][debug][connection] [source/common/network/connection_impl.cc:501] [C479] remote close
[2019-02-14 21:01:09.348][000048][debug][connection] [source/common/network/connection_impl.cc:183] [C479] closing socket: 0
[2019-02-14 21:01:09.348][000048][debug][connection] [source/common/ssl/ssl_socket.cc:229] [C479] SSL shutdown: rc=1
[2019-02-14 21:01:09.348][000048][debug][main] [source/server/connection_handler_impl.cc:68] [C479] adding to cleanup list
  • The header is is respected when set in the cURL request
$ curl -v -k -H "x-database-id: odd" https://{AMBASSADOR_DOMAIN}/qotm/
@kflynn kflynn added this to the 0.51 milestone Feb 14, 2019
@kflynn kflynn changed the title Does Envoy Respect Headers Set in AuthService Users want Ambassador to be able to route on headers added by extauth Feb 14, 2019
@iNoahNothing
Copy link
Contributor Author

Same behavior with Host

@gsagula
Copy link
Contributor

gsagula commented Feb 14, 2019

@nbkrause Do you have the Envoy config? Can you paste it here when you have a chance, please?

@iNoahNothing
Copy link
Contributor Author

iNoahNothing commented Feb 14, 2019

AuthService sets Host header:

{
    "@type": "/envoy.config.bootstrap.v2.Bootstrap",
    "static_resources": {
        "clusters": [
            {
                "connect_timeout": "3s",
                "dns_lookup_family": "V4_ONLY",
                "lb_policy": "ROUND_ROBIN",
                "load_assignment": {
                    "cluster_name": "cluster_127_0_0_1_8877",
                    "endpoints": [
                        {
                            "lb_endpoints": [
                                {
                                    "endpoint": {
                                        "address": {
                                            "socket_address": {
                                                "address": "127.0.0.1",
                                                "port_value": 8877,
                                                "protocol": "TCP"
                                            }
                                        }
                                    }
                                }
                            ]
                        }
                    ]
                },
                "name": "cluster_127_0_0_1_8877",
                "type": "STRICT_DNS"
            },
            {
                "connect_timeout": "3s",
                "dns_lookup_family": "V4_ONLY",
                "lb_policy": "ROUND_ROBIN",
                "load_assignment": {
                    "cluster_name": "cluster_acme_challenge_service",
                    "endpoints": [
                        {
                            "lb_endpoints": [
                                {
                                    "endpoint": {
                                        "address": {
                                            "socket_address": {
                                                "address": "acme-challenge-service",
                                                "port_value": 80,
                                                "protocol": "TCP"
                                            }
                                        }
                                    }
                                }
                            ]
                        }
                    ]
                },
                "name": "cluster_acme_challenge_service",
                "type": "STRICT_DNS"
            },
            {
                "connect_timeout": "3s",
                "dns_lookup_family": "V4_ONLY",
                "lb_policy": "ROUND_ROBIN",
                "load_assignment": {
                    "cluster_name": "cluster_ambassador_pro_auth",
                    "endpoints": [
                        {
                            "lb_endpoints": [
                                {
                                    "endpoint": {
                                        "address": {
                                            "socket_address": {
                                                "address": "ambassador-pro-auth",
                                                "port_value": 80,
                                                "protocol": "TCP"
                                            }
                                        }
                                    }
                                }
                            ]
                        }
                    ]
                },
                "name": "cluster_ambassador_pro_auth",
                "type": "STRICT_DNS"
            },
            {
                "connect_timeout": "3s",
                "dns_lookup_family": "V4_ONLY",
                "lb_policy": "ROUND_ROBIN",
                "load_assignment": {
                    "cluster_name": "cluster_echo_websocket_org",
                    "endpoints": [
                        {
                            "lb_endpoints": [
                                {
                                    "endpoint": {
                                        "address": {
                                            "socket_address": {
                                                "address": "echo.websocket.org",
                                                "port_value": 80,
                                                "protocol": "TCP"
                                            }
                                        }
                                    }
                                }
                            ]
                        }
                    ]
                },
                "name": "cluster_echo_websocket_org",
                "type": "STRICT_DNS"
            },
            {
                "connect_timeout": "3s",
                "dns_lookup_family": "V4_ONLY",
                "http2_protocol_options": {},
                "lb_policy": "ROUND_ROBIN",
                "load_assignment": {
                    "cluster_name": "cluster_example_rate_limit_5000",
                    "endpoints": [
                        {
                            "lb_endpoints": [
                                {
                                    "endpoint": {
                                        "address": {
                                            "socket_address": {
                                                "address": "example-rate-limit",
                                                "port_value": 5000,
                                                "protocol": "TCP"
                                            }
                                        }
                                    }
                                }
                            ]
                        }
                    ]
                },
                "name": "cluster_example_rate_limit_5000",
                "type": "STRICT_DNS"
            },
            {
                "connect_timeout": "3s",
                "dns_lookup_family": "V4_ONLY",
                "lb_policy": "ROUND_ROBIN",
                "load_assignment": {
                    "cluster_name": "cluster_extauth_ambassador_pro_auth",
                    "endpoints": [
                        {
                            "lb_endpoints": [
                                {
                                    "endpoint": {
                                        "address": {
                                            "socket_address": {
                                                "address": "ambassador-pro-auth",
                                                "port_value": 80,
                                                "protocol": "TCP"
                                            }
                                        }
                                    }
                                }
                            ]
                        }
                    ]
                },
                "name": "cluster_extauth_ambassador_pro_auth",
                "type": "STRICT_DNS"
            },
            {
                "connect_timeout": "3s",
                "dns_lookup_family": "V4_ONLY",
                "http2_protocol_options": {},
                "lb_policy": "ROUND_ROBIN",
                "load_assignment": {
                    "cluster_name": "cluster_grpc_py_test",
                    "endpoints": [
                        {
                            "lb_endpoints": [
                                {
                                    "endpoint": {
                                        "address": {
                                            "socket_address": {
                                                "address": "grpc-py.test",
                                                "port_value": 80,
                                                "protocol": "TCP"
                                            }
                                        }
                                    }
                                }
                            ]
                        }
                    ]
                },
                "name": "cluster_grpc_py_test",
                "type": "STRICT_DNS"
            },
            {
                "connect_timeout": "3s",
                "dns_lookup_family": "V4_ONLY",
                "lb_policy": "ROUND_ROBIN",
                "load_assignment": {
                    "cluster_name": "cluster_httpbin_org_80",
                    "endpoints": [
                        {
                            "lb_endpoints": [
                                {
                                    "endpoint": {
                                        "address": {
                                            "socket_address": {
                                                "address": "httpbin.org",
                                                "port_value": 80,
                                                "protocol": "TCP"
                                            }
                                        }
                                    }
                                }
                            ]
                        }
                    ]
                },
                "name": "cluster_httpbin_org_80",
                "type": "STRICT_DNS"
            },
            {
                "connect_timeout": "3s",
                "dns_lookup_family": "V4_ONLY",
                "lb_policy": "ROUND_ROBIN",
                "load_assignment": {
                    "cluster_name": "cluster_microdonut_10001",
                    "endpoints": [
                        {
                            "lb_endpoints": [
                                {
                                    "endpoint": {
                                        "address": {
                                            "socket_address": {
                                                "address": "microdonut",
                                                "port_value": 10001,
                                                "protocol": "TCP"
                                            }
                                        }
                                    }
                                }
                            ]
                        }
                    ]
                },
                "name": "cluster_microdonut_10001",
                "type": "STRICT_DNS"
            },
            {
                "connect_timeout": "3s",
                "dns_lookup_family": "V4_ONLY",
                "lb_policy": "ROUND_ROBIN",
                "load_assignment": {
                    "cluster_name": "cluster_qotm",
                    "endpoints": [
                        {
                            "lb_endpoints": [
                                {
                                    "endpoint": {
                                        "address": {
                                            "socket_address": {
                                                "address": "qotm",
                                                "port_value": 80,
                                                "protocol": "TCP"
                                            }
                                        }
                                    }
                                }
                            ]
                        }
                    ]
                },
                "name": "cluster_qotm",
                "type": "STRICT_DNS"
            },
            {
                "connect_timeout": "3s",
                "dns_lookup_family": "V4_ONLY",
                "lb_policy": "ROUND_ROBIN",
                "load_assignment": {
                    "cluster_name": "cluster_test_service",
                    "endpoints": [
                        {
                            "lb_endpoints": [
                                {
                                    "endpoint": {
                                        "address": {
                                            "socket_address": {
                                                "address": "test-service",
                                                "port_value": 80,
                                                "protocol": "TCP"
                                            }
                                        }
                                    }
                                }
                            ]
                        }
                    ]
                },
                "name": "cluster_test_service",
                "type": "STRICT_DNS"
            }
        ],
        "listeners": [
            {
                "address": {
                    "socket_address": {
                        "address": "0.0.0.0",
                        "port_value": 80,
                        "protocol": "TCP"
                    }
                },
                "filter_chains": [
                    {
                        "filters": [
                            {
                                "config": {
                                    "access_log": [
                                        {
                                            "config": {
                                                "format": "ACCESS [%START_TIME%] \"%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%\" %RESPONSE_CODE% %RESPONSE_FLAGS% %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% \"%REQ(X-FORWARDED-FOR)%\" \"%REQ(USER-AGENT)%\" \"%REQ(X-REQUEST-ID)%\" \"%REQ(:AUTHORITY)%\" \"%UPSTREAM_HOST%\"\n",
                                                "path": "/dev/fd/1"
                                            },
                                            "name": "envoy.file_access_log"
                                        }
                                    ],
                                    "http_filters": [
                                        {
                                            "config": {
                                                "http_service": {
                                                    "allowed_authorization_headers": [
                                                        "www-authenticate",
                                                        "location",
                                                        ":authority",
                                                        "client-id",
                                                        "authorization",
                                                        "client-secret",
                                                        "x-database-id",
                                                        "set-cookie",
                                                        "proxy-authenticate"
                                                    ],
                                                    "allowed_request_headers": [
                                                        ":authority",
                                                        "from",
                                                        "proxy-authorization",
                                                        "client-id",
                                                        "authorization",
                                                        "user-agent",
                                                        "x-forwarded-for",
                                                        "client-secret",
                                                        "x-forwarded-proto",
                                                        "x-database-id",
                                                        "x-forwarded-host",
                                                        "cookie"
                                                    ],
                                                    "path_prefix": null,
                                                    "server_uri": {
                                                        "cluster": "cluster_extauth_ambassador_pro_auth",
                                                        "timeout": "5.000s",
                                                        "uri": "http://"
                                                    }
                                                },
                                                "send_request_data": false
                                            },
                                            "name": "envoy.ext_authz"
                                        },
                                        {
                                            "config": {
                                                "domain": "ambassador",
                                                "request_type": "both",
                                                "timeout": "0.020s"
                                            },
                                            "name": "envoy.rate_limit"
                                        },
                                        {
                                            "name": "envoy.cors"
                                        },
                                        {
                                            "name": "envoy.router"
                                        }
                                    ],
                                    "route_config": {
                                        "virtual_hosts": [
                                            {
                                                "domains": [
                                                    "*"
                                                ],
                                                "name": "backend",
                                                "routes": [
                                                    {
                                                        "match": {
                                                            "case_sensitive": true,
                                                            "prefix": "/.well-known/acme-challenge"
                                                        },
                                                        "route": {
                                                            "priority": null,
                                                            "timeout": "3.000s",
                                                            "weighted_clusters": {
                                                                "clusters": [
                                                                    {
                                                                        "name": "cluster_acme_challenge_service",
                                                                        "weight": 100.0
                                                                    }
                                                                ]
                                                            }
                                                        }
                                                    },
                                                    {
                                                        "match": {
                                                            "case_sensitive": true,
                                                            "prefix": "/ambassador/v0/check_ready"
                                                        },
                                                        "route": {
                                                            "prefix_rewrite": "/ambassador/v0/check_ready",
                                                            "priority": null,
                                                            "timeout": "10.000s",
                                                            "weighted_clusters": {
                                                                "clusters": [
                                                                    {
                                                                        "name": "cluster_127_0_0_1_8877",
                                                                        "weight": 100.0
                                                                    }
                                                                ]
                                                            }
                                                        }
                                                    },
                                                    {
                                                        "match": {
                                                            "case_sensitive": true,
                                                            "prefix": "/ambassador/v0/check_alive"
                                                        },
                                                        "route": {
                                                            "prefix_rewrite": "/ambassador/v0/check_alive",
                                                            "priority": null,
                                                            "timeout": "10.000s",
                                                            "weighted_clusters": {
                                                                "clusters": [
                                                                    {
                                                                        "name": "cluster_127_0_0_1_8877",
                                                                        "weight": 100.0
                                                                    }
                                                                ]
                                                            }
                                                        }
                                                    },
                                                    {
                                                        "match": {
                                                            "case_sensitive": true,
                                                            "prefix": "/hello.Greeter/"
                                                        },
                                                        "route": {
                                                            "prefix_rewrite": "/hello.Greeter/",
                                                            "priority": null,
                                                            "timeout": "3.000s",
                                                            "weighted_clusters": {
                                                                "clusters": [
                                                                    {
                                                                        "name": "cluster_grpc_py_test",
                                                                        "weight": 100.0
                                                                    }
                                                                ]
                                                            }
                                                        }
                                                    },
                                                    {
                                                        "match": {
                                                            "case_sensitive": true,
                                                            "prefix": "/ambassador/v0/"
                                                        },
                                                        "route": {
                                                            "prefix_rewrite": "/ambassador/v0/",
                                                            "priority": null,
                                                            "timeout": "10.000s",
                                                            "weighted_clusters": {
                                                                "clusters": [
                                                                    {
                                                                        "name": "cluster_127_0_0_1_8877",
                                                                        "weight": 100.0
                                                                    }
                                                                ]
                                                            }
                                                        }
                                                    },
                                                    {
                                                        "match": {
                                                            "case_sensitive": true,
                                                            "prefix": "/microdonut/"
                                                        },
                                                        "route": {
                                                            "prefix_rewrite": "/",
                                                            "priority": null,
                                                            "timeout": "3.000s",
                                                            "weighted_clusters": {
                                                                "clusters": [
                                                                    {
                                                                        "name": "cluster_microdonut_10001",
                                                                        "weight": 100.0
                                                                    }
                                                                ]
                                                            }
                                                        }
                                                    },
                                                    {
                                                        "match": {
                                                            "case_sensitive": true,
                                                            "prefix": "/httpbin/"
                                                        },
                                                        "route": {
                                                            "host_rewrite": "httpbin.org",
                                                            "prefix_rewrite": "/",
                                                            "priority": null,
                                                            "timeout": "3.000s",
                                                            "weighted_clusters": {
                                                                "clusters": [
                                                                    {
                                                                        "name": "cluster_httpbin_org_80",
                                                                        "weight": 100.0
                                                                    }
                                                                ]
                                                            }
                                                        }
                                                    },
                                                    {
                                                        "match": {
                                                            "case_sensitive": true,
                                                            "prefix": "/callback"
                                                        },
                                                        "route": {
                                                            "prefix_rewrite": "/",
                                                            "priority": null,
                                                            "timeout": "3.000s",
                                                            "weighted_clusters": {
                                                                "clusters": [
                                                                    {
                                                                        "name": "cluster_ambassador_pro_auth",
                                                                        "weight": 100.0
                                                                    }
                                                                ]
                                                            }
                                                        }
                                                    },
                                                    {
                                                        "match": {
                                                            "case_sensitive": true,
                                                            "headers": [
                                                                {
                                                                    "exact_match": "even.domain2.nkrause.k736.net",
                                                                    "name": ":authority"
                                                                }
                                                            ],
                                                            "prefix": "/qotm/"
                                                        },
                                                        "route": {
                                                            "prefix_rewrite": "/quote/1",
                                                            "priority": null,
                                                            "timeout": "3.000s",
                                                            "weighted_clusters": {
                                                                "clusters": [
                                                                    {
                                                                        "name": "cluster_qotm",
                                                                        "weight": 100.0
                                                                    }
                                                                ]
                                                            }
                                                        }
                                                    },
                                                    {
                                                        "match": {
                                                            "case_sensitive": true,
                                                            "headers": [
                                                                {
                                                                    "exact_match": "odd.domain2.nkrause.k736.net",
                                                                    "name": ":authority"
                                                                }
                                                            ],
                                                            "prefix": "/qotm/"
                                                        },
                                                        "route": {
                                                            "prefix_rewrite": "/quote/2",
                                                            "priority": null,
                                                            "timeout": "3.000s",
                                                            "weighted_clusters": {
                                                                "clusters": [
                                                                    {
                                                                        "name": "cluster_qotm",
                                                                        "weight": 100.0
                                                                    }
                                                                ]
                                                            }
                                                        }
                                                    },
                                                    {
                                                        "match": {
                                                            "case_sensitive": true,
                                                            "prefix": "/qotm/"
                                                        },
                                                        "route": {
                                                            "prefix_rewrite": "/quote/0",
                                                            "priority": null,
                                                            "timeout": "3.000s",
                                                            "weighted_clusters": {
                                                                "clusters": [
                                                                    {
                                                                        "name": "cluster_qotm",
                                                                        "weight": 100.0
                                                                    }
                                                                ]
                                                            }
                                                        }
                                                    },
                                                    {
                                                        "match": {
                                                            "case_sensitive": true,
                                                            "prefix": "/echo/"
                                                        },
                                                        "route": {
                                                            "host_rewrite": "echo.websocket.org",
                                                            "prefix_rewrite": "/",
                                                            "priority": null,
                                                            "timeout": "3.000s",
                                                            "weighted_clusters": {
                                                                "clusters": [
                                                                    {
                                                                        "name": "cluster_echo_websocket_org",
                                                                        "weight": 100.0
                                                                    }
                                                                ]
                                                            }
                                                        }
                                                    },
                                                    {
                                                        "match": {
                                                            "case_sensitive": true,
                                                            "prefix": "/foo"
                                                        },
                                                        "route": {
                                                            "cors": {
                                                                "allow_headers": "Keep-Alive, User-Agent",
                                                                "allow_methods": "GET, PUT, OPTIONS",
                                                                "allow_origin": [
                                                                    "*"
                                                                ],
                                                                "enabled": true
                                                            },
                                                            "prefix_rewrite": "/",
                                                            "priority": null,
                                                            "timeout": "3.000s",
                                                            "weighted_clusters": {
                                                                "clusters": [
                                                                    {
                                                                        "name": "cluster_test_service",
                                                                        "weight": 100.0
                                                                    }
                                                                ]
                                                            }
                                                        }
                                                    }
                                                ]
                                            }
                                        ]
                                    },
                                    "stat_prefix": "ingress_http",
                                    "upgrade_configs": [
                                        {
                                            "upgrade_type": "websocket"
                                        }
                                    ],
                                    "use_remote_address": true
                                },
                                "name": "envoy.http_connection_manager"
                            }
                        ],
                        "use_proxy_proto": false
                    }
                ],
                "name": "ambassador-listener-80"
            }
        ]
    }
}

AuthService sets x-database-id:

{
    "@type": "/envoy.config.bootstrap.v2.Bootstrap",
    "static_resources": {
        "clusters": [
            {
                "connect_timeout": "3s",
                "dns_lookup_family": "V4_ONLY",
                "lb_policy": "ROUND_ROBIN",
                "load_assignment": {
                    "cluster_name": "cluster_127_0_0_1_8877",
                    "endpoints": [
                        {
                            "lb_endpoints": [
                                {
                                    "endpoint": {
                                        "address": {
                                            "socket_address": {
                                                "address": "127.0.0.1",
                                                "port_value": 8877,
                                                "protocol": "TCP"
                                            }
                                        }
                                    }
                                }
                            ]
                        }
                    ]
                },
                "name": "cluster_127_0_0_1_8877",
                "type": "STRICT_DNS"
            },
            {
                "connect_timeout": "3s",
                "dns_lookup_family": "V4_ONLY",
                "lb_policy": "ROUND_ROBIN",
                "load_assignment": {
                    "cluster_name": "cluster_acme_challenge_service",
                    "endpoints": [
                        {
                            "lb_endpoints": [
                                {
                                    "endpoint": {
                                        "address": {
                                            "socket_address": {
                                                "address": "acme-challenge-service",
                                                "port_value": 80,
                                                "protocol": "TCP"
                                            }
                                        }
                                    }
                                }
                            ]
                        }
                    ]
                },
                "name": "cluster_acme_challenge_service",
                "type": "STRICT_DNS"
            },
            {
                "connect_timeout": "3s",
                "dns_lookup_family": "V4_ONLY",
                "lb_policy": "ROUND_ROBIN",
                "load_assignment": {
                    "cluster_name": "cluster_ambassador_pro_auth",
                    "endpoints": [
                        {
                            "lb_endpoints": [
                                {
                                    "endpoint": {
                                        "address": {
                                            "socket_address": {
                                                "address": "ambassador-pro-auth",
                                                "port_value": 80,
                                                "protocol": "TCP"
                                            }
                                        }
                                    }
                                }
                            ]
                        }
                    ]
                },
                "name": "cluster_ambassador_pro_auth",
                "type": "STRICT_DNS"
            },
            {
                "connect_timeout": "3s",
                "dns_lookup_family": "V4_ONLY",
                "lb_policy": "ROUND_ROBIN",
                "load_assignment": {
                    "cluster_name": "cluster_echo_websocket_org",
                    "endpoints": [
                        {
                            "lb_endpoints": [
                                {
                                    "endpoint": {
                                        "address": {
                                            "socket_address": {
                                                "address": "echo.websocket.org",
                                                "port_value": 80,
                                                "protocol": "TCP"
                                            }
                                        }
                                    }
                                }
                            ]
                        }
                    ]
                },
                "name": "cluster_echo_websocket_org",
                "type": "STRICT_DNS"
            },
            {
                "connect_timeout": "3s",
                "dns_lookup_family": "V4_ONLY",
                "http2_protocol_options": {},
                "lb_policy": "ROUND_ROBIN",
                "load_assignment": {
                    "cluster_name": "cluster_example_rate_limit_5000",
                    "endpoints": [
                        {
                            "lb_endpoints": [
                                {
                                    "endpoint": {
                                        "address": {
                                            "socket_address": {
                                                "address": "example-rate-limit",
                                                "port_value": 5000,
                                                "protocol": "TCP"
                                            }
                                        }
                                    }
                                }
                            ]
                        }
                    ]
                },
                "name": "cluster_example_rate_limit_5000",
                "type": "STRICT_DNS"
            },
            {
                "connect_timeout": "3s",
                "dns_lookup_family": "V4_ONLY",
                "lb_policy": "ROUND_ROBIN",
                "load_assignment": {
                    "cluster_name": "cluster_extauth_ambassador_pro_auth",
                    "endpoints": [
                        {
                            "lb_endpoints": [
                                {
                                    "endpoint": {
                                        "address": {
                                            "socket_address": {
                                                "address": "ambassador-pro-auth",
                                                "port_value": 80,
                                                "protocol": "TCP"
                                            }
                                        }
                                    }
                                }
                            ]
                        }
                    ]
                },
                "name": "cluster_extauth_ambassador_pro_auth",
                "type": "STRICT_DNS"
            },
            {
                "connect_timeout": "3s",
                "dns_lookup_family": "V4_ONLY",
                "http2_protocol_options": {},
                "lb_policy": "ROUND_ROBIN",
                "load_assignment": {
                    "cluster_name": "cluster_grpc_py_test",
                    "endpoints": [
                        {
                            "lb_endpoints": [
                                {
                                    "endpoint": {
                                        "address": {
                                            "socket_address": {
                                                "address": "grpc-py.test",
                                                "port_value": 80,
                                                "protocol": "TCP"
                                            }
                                        }
                                    }
                                }
                            ]
                        }
                    ]
                },
                "name": "cluster_grpc_py_test",
                "type": "STRICT_DNS"
            },
            {
                "connect_timeout": "3s",
                "dns_lookup_family": "V4_ONLY",
                "lb_policy": "ROUND_ROBIN",
                "load_assignment": {
                    "cluster_name": "cluster_httpbin_org_80",
                    "endpoints": [
                        {
                            "lb_endpoints": [
                                {
                                    "endpoint": {
                                        "address": {
                                            "socket_address": {
                                                "address": "httpbin.org",
                                                "port_value": 80,
                                                "protocol": "TCP"
                                            }
                                        }
                                    }
                                }
                            ]
                        }
                    ]
                },
                "name": "cluster_httpbin_org_80",
                "type": "STRICT_DNS"
            },
            {
                "connect_timeout": "3s",
                "dns_lookup_family": "V4_ONLY",
                "lb_policy": "ROUND_ROBIN",
                "load_assignment": {
                    "cluster_name": "cluster_microdonut_10001",
                    "endpoints": [
                        {
                            "lb_endpoints": [
                                {
                                    "endpoint": {
                                        "address": {
                                            "socket_address": {
                                                "address": "microdonut",
                                                "port_value": 10001,
                                                "protocol": "TCP"
                                            }
                                        }
                                    }
                                }
                            ]
                        }
                    ]
                },
                "name": "cluster_microdonut_10001",
                "type": "STRICT_DNS"
            },
            {
                "connect_timeout": "3s",
                "dns_lookup_family": "V4_ONLY",
                "lb_policy": "ROUND_ROBIN",
                "load_assignment": {
                    "cluster_name": "cluster_qotm",
                    "endpoints": [
                        {
                            "lb_endpoints": [
                                {
                                    "endpoint": {
                                        "address": {
                                            "socket_address": {
                                                "address": "qotm",
                                                "port_value": 80,
                                                "protocol": "TCP"
                                            }
                                        }
                                    }
                                }
                            ]
                        }
                    ]
                },
                "name": "cluster_qotm",
                "type": "STRICT_DNS"
            },
            {
                "connect_timeout": "3s",
                "dns_lookup_family": "V4_ONLY",
                "lb_policy": "ROUND_ROBIN",
                "load_assignment": {
                    "cluster_name": "cluster_test_service",
                    "endpoints": [
                        {
                            "lb_endpoints": [
                                {
                                    "endpoint": {
                                        "address": {
                                            "socket_address": {
                                                "address": "test-service",
                                                "port_value": 80,
                                                "protocol": "TCP"
                                            }
                                        }
                                    }
                                }
                            ]
                        }
                    ]
                },
                "name": "cluster_test_service",
                "type": "STRICT_DNS"
            }
        ],
        "listeners": [
            {
                "address": {
                    "socket_address": {
                        "address": "0.0.0.0",
                        "port_value": 80,
                        "protocol": "TCP"
                    }
                },
                "filter_chains": [
                    {
                        "filters": [
                            {
                                "config": {
                                    "access_log": [
                                        {
                                            "config": {
                                                "format": "ACCESS [%START_TIME%] \"%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%\" %RESPONSE_CODE% %RESPONSE_FLAGS% %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% \"%REQ(X-FORWARDED-FOR)%\" \"%REQ(USER-AGENT)%\" \"%REQ(X-REQUEST-ID)%\" \"%REQ(:AUTHORITY)%\" \"%UPSTREAM_HOST%\"\n",
                                                "path": "/dev/fd/1"
                                            },
                                            "name": "envoy.file_access_log"
                                        }
                                    ],
                                    "http_filters": [
                                        {
                                            "config": {
                                                "http_service": {
                                                    "allowed_authorization_headers": [
                                                        "www-authenticate",
                                                        "location",
                                                        ":authority",
                                                        "client-id",
                                                        "authorization",
                                                        "client-secret",
                                                        "x-database-id",
                                                        "set-cookie",
                                                        "proxy-authenticate"
                                                    ],
                                                    "allowed_request_headers": [
                                                        ":authority",
                                                        "from",
                                                        "proxy-authorization",
                                                        "client-id",
                                                        "authorization",
                                                        "user-agent",
                                                        "x-forwarded-for",
                                                        "client-secret",
                                                        "x-forwarded-proto",
                                                        "x-database-id",
                                                        "x-forwarded-host",
                                                        "cookie"
                                                    ],
                                                    "path_prefix": null,
                                                    "server_uri": {
                                                        "cluster": "cluster_extauth_ambassador_pro_auth",
                                                        "timeout": "5.000s",
                                                        "uri": "http://"
                                                    }
                                                },
                                                "send_request_data": false
                                            },
                                            "name": "envoy.ext_authz"
                                        },
                                        {
                                            "config": {
                                                "domain": "ambassador",
                                                "request_type": "both",
                                                "timeout": "0.020s"
                                            },
                                            "name": "envoy.rate_limit"
                                        },
                                        {
                                            "name": "envoy.cors"
                                        },
                                        {
                                            "name": "envoy.router"
                                        }
                                    ],
                                    "route_config": {
                                        "virtual_hosts": [
                                            {
                                                "domains": [
                                                    "*"
                                                ],
                                                "name": "backend",
                                                "routes": [
                                                    {
                                                        "match": {
                                                            "case_sensitive": true,
                                                            "prefix": "/.well-known/acme-challenge"
                                                        },
                                                        "route": {
                                                            "priority": null,
                                                            "timeout": "3.000s",
                                                            "weighted_clusters": {
                                                                "clusters": [
                                                                    {
                                                                        "name": "cluster_acme_challenge_service",
                                                                        "weight": 100.0
                                                                    }
                                                                ]
                                                            }
                                                        }
                                                    },
                                                    {
                                                        "match": {
                                                            "case_sensitive": true,
                                                            "prefix": "/ambassador/v0/check_ready"
                                                        },
                                                        "route": {
                                                            "prefix_rewrite": "/ambassador/v0/check_ready",
                                                            "priority": null,
                                                            "timeout": "10.000s",
                                                            "weighted_clusters": {
                                                                "clusters": [
                                                                    {
                                                                        "name": "cluster_127_0_0_1_8877",
                                                                        "weight": 100.0
                                                                    }
                                                                ]
                                                            }
                                                        }
                                                    },
                                                    {
                                                        "match": {
                                                            "case_sensitive": true,
                                                            "prefix": "/ambassador/v0/check_alive"
                                                        },
                                                        "route": {
                                                            "prefix_rewrite": "/ambassador/v0/check_alive",
                                                            "priority": null,
                                                            "timeout": "10.000s",
                                                            "weighted_clusters": {
                                                                "clusters": [
                                                                    {
                                                                        "name": "cluster_127_0_0_1_8877",
                                                                        "weight": 100.0
                                                                    }
                                                                ]
                                                            }
                                                        }
                                                    },
                                                    {
                                                        "match": {
                                                            "case_sensitive": true,
                                                            "prefix": "/hello.Greeter/"
                                                        },
                                                        "route": {
                                                            "prefix_rewrite": "/hello.Greeter/",
                                                            "priority": null,
                                                            "timeout": "3.000s",
                                                            "weighted_clusters": {
                                                                "clusters": [
                                                                    {
                                                                        "name": "cluster_grpc_py_test",
                                                                        "weight": 100.0
                                                                    }
                                                                ]
                                                            }
                                                        }
                                                    },
                                                    {
                                                        "match": {
                                                            "case_sensitive": true,
                                                            "prefix": "/ambassador/v0/"
                                                        },
                                                        "route": {
                                                            "prefix_rewrite": "/ambassador/v0/",
                                                            "priority": null,
                                                            "timeout": "10.000s",
                                                            "weighted_clusters": {
                                                                "clusters": [
                                                                    {
                                                                        "name": "cluster_127_0_0_1_8877",
                                                                        "weight": 100.0
                                                                    }
                                                                ]
                                                            }
                                                        }
                                                    },
                                                    {
                                                        "match": {
                                                            "case_sensitive": true,
                                                            "prefix": "/microdonut/"
                                                        },
                                                        "route": {
                                                            "prefix_rewrite": "/",
                                                            "priority": null,
                                                            "timeout": "3.000s",
                                                            "weighted_clusters": {
                                                                "clusters": [
                                                                    {
                                                                        "name": "cluster_microdonut_10001",
                                                                        "weight": 100.0
                                                                    }
                                                                ]
                                                            }
                                                        }
                                                    },
                                                    {
                                                        "match": {
                                                            "case_sensitive": true,
                                                            "prefix": "/httpbin/"
                                                        },
                                                        "route": {
                                                            "host_rewrite": "httpbin.org",
                                                            "prefix_rewrite": "/",
                                                            "priority": null,
                                                            "timeout": "3.000s",
                                                            "weighted_clusters": {
                                                                "clusters": [
                                                                    {
                                                                        "name": "cluster_httpbin_org_80",
                                                                        "weight": 100.0
                                                                    }
                                                                ]
                                                            }
                                                        }
                                                    },
                                                    {
                                                        "match": {
                                                            "case_sensitive": true,
                                                            "prefix": "/callback"
                                                        },
                                                        "route": {
                                                            "prefix_rewrite": "/",
                                                            "priority": null,
                                                            "timeout": "3.000s",
                                                            "weighted_clusters": {
                                                                "clusters": [
                                                                    {
                                                                        "name": "cluster_ambassador_pro_auth",
                                                                        "weight": 100.0
                                                                    }
                                                                ]
                                                            }
                                                        }
                                                    },
                                                    {
                                                        "match": {
                                                            "case_sensitive": true,
                                                            "headers": [
                                                                {
                                                                    "exact_match": "even",
                                                                    "name": "X-Database-Id"
                                                                }
                                                            ],
                                                            "prefix": "/qotm/"
                                                        },
                                                        "route": {
                                                            "prefix_rewrite": "/quote/2",
                                                            "priority": null,
                                                            "timeout": "3.000s",
                                                            "weighted_clusters": {
                                                                "clusters": [
                                                                    {
                                                                        "name": "cluster_qotm",
                                                                        "weight": 100.0
                                                                    }
                                                                ]
                                                            }
                                                        }
                                                    },
                                                    {
                                                        "match": {
                                                            "case_sensitive": true,
                                                            "headers": [
                                                                {
                                                                    "exact_match": "odd",
                                                                    "name": "X-Database-Id"
                                                                }
                                                            ],
                                                            "prefix": "/qotm/"
                                                        },
                                                        "route": {
                                                            "prefix_rewrite": "/quote/3",
                                                            "priority": null,
                                                            "timeout": "3.000s",
                                                            "weighted_clusters": {
                                                                "clusters": [
                                                                    {
                                                                        "name": "cluster_qotm",
                                                                        "weight": 100.0
                                                                    }
                                                                ]
                                                            }
                                                        }
                                                    },
                                                    {
                                                        "match": {
                                                            "case_sensitive": true,
                                                            "prefix": "/qotm/"
                                                        },
                                                        "route": {
                                                            "prefix_rewrite": "/quote/0",
                                                            "priority": null,
                                                            "timeout": "3.000s",
                                                            "weighted_clusters": {
                                                                "clusters": [
                                                                    {
                                                                        "name": "cluster_qotm",
                                                                        "weight": 100.0
                                                                    }
                                                                ]
                                                            }
                                                        }
                                                    },
                                                    {
                                                        "match": {
                                                            "case_sensitive": true,
                                                            "prefix": "/echo/"
                                                        },
                                                        "route": {
                                                            "host_rewrite": "echo.websocket.org",
                                                            "prefix_rewrite": "/",
                                                            "priority": null,
                                                            "timeout": "3.000s",
                                                            "weighted_clusters": {
                                                                "clusters": [
                                                                    {
                                                                        "name": "cluster_echo_websocket_org",
                                                                        "weight": 100.0
                                                                    }
                                                                ]
                                                            }
                                                        }
                                                    },
                                                    {
                                                        "match": {
                                                            "case_sensitive": true,
                                                            "prefix": "/foo"
                                                        },
                                                        "route": {
                                                            "cors": {
                                                                "allow_headers": "Keep-Alive, User-Agent",
                                                                "allow_methods": "GET, PUT, OPTIONS",
                                                                "allow_origin": [
                                                                    "*"
                                                                ],
                                                                "enabled": true
                                                            },
                                                            "prefix_rewrite": "/",
                                                            "priority": null,
                                                            "timeout": "3.000s",
                                                            "weighted_clusters": {
                                                                "clusters": [
                                                                    {
                                                                        "name": "cluster_test_service",
                                                                        "weight": 100.0
                                                                    }
                                                                ]
                                                            }
                                                        }
                                                    }
                                                ]
                                            }
                                        ]
                                    },
                                    "stat_prefix": "ingress_http",
                                    "upgrade_configs": [
                                        {
                                            "upgrade_type": "websocket"
                                        }
                                    ],
                                    "use_remote_address": true
                                },
                                "name": "envoy.http_connection_manager"
                            }
                        ],
                        "use_proxy_proto": false
                    }
                ],
                "name": "ambassador-listener-80"
            }
        ]
    }
}

@gsagula
Copy link
Contributor

gsagula commented Feb 15, 2019

Ok, couple problems here. First, looking at the Envoy router logic:

if (weighted_clusters_.empty()) {
    if (!cluster_name_.empty() || isDirectResponse()) {
      return shared_from_this();
    } else {
      ASSERT(!cluster_header_name_.get().empty());
      const Http::HeaderEntry* entry = headers.get(cluster_header_name_);
      std::string final_cluster_name;
      if (entry) {
        final_cluster_name = entry->value().c_str();
      }

      // NOTE: Though we return a shared_ptr here, the current ownership model assumes that
      //       the route table sticks around. See snapped_route_config_ in
      //       ConnectionManagerImpl::ActiveStream.
      return std::make_shared<DynamicRouteEntry>(this, final_cluster_name);
    }
  }

It doesn't seem possible to have both headers and weighted_clusters playing together. The second one will always take precence making header's cofiguration irrelevant.

Second, loooking at where cluster_header_name_ and its constructor, not that the correct config is taken from route.route().cluster_header():

    // Envoy will determine the cluster to route to by reading the value of the
    // HTTP header named by cluster_header from the request headers. If the
    // header is not found or the referenced cluster does not exist, Envoy will
    // return a 404 response.
    //
    // .. attention::
    //
    //   Internally, Envoy always uses the HTTP/2 *:authority* header to represent the HTTP/1
    //   *Host* header. Thus, if attempting to match on *Host*, match on *:authority* instead.
    string cluster_header = 2 [(validate.rules).string.min_bytes = 1];

In this case the correct config for header-base routing should look more like this:

routes:
   - match: { prefix: "/bin" }
      route: { host_rewrite: httpbin.org, prefix_rewrite: "/", cluster: httpbin_cluster }
   - match: { prefix: "/bin" }
      route: { host_rewrite: httpbin.org,  prefix_rewrite: "/", cluster_header: "x-httpbin-2" }

Where x-httpbin-2 is the header dispatched by the authorization and its correspondent value should be the cluster's name.

FYI @kflynn @nbkrause

@gsagula
Copy link
Contributor

gsagula commented Feb 15, 2019

After chatting @kflynn, I realized that this ^^ is not the correct use-case. It seems that the external authorization filter should be clearing the route cache callbacks_->clearRouteCache() before dispatching the new headers to the upstream. I totally missed that from the initial design, but hopefully, the new image will fix the problem.

kflynn pushed a commit that referenced this issue Feb 15, 2019
@gsagula
Copy link
Contributor

gsagula commented Feb 15, 2019

Awesome @kflynn
It looks like the test passed with the new image!

@bingosummer
Copy link

bingosummer commented Oct 16, 2020

@kflynn @gsagula I can still see the same issue. The auth service sets a header. What I expect is Ambassador routes the request based on the header. But the actual behavior is not like this. Seems the issue is not fixed.

@bingosummer
Copy link

I figured it out. I need to provide a default mapping. What the service is doesn't matter.

apiVersion: getambassador.io/v2
kind: Mapping
metadata:
  name: default
spec:
  prefix: /
  service: "127.0.0.1:443"
  tls: my-tls-context

If the default mapping doesn't exist, the ambassador won't send the request to extauth and return no_route.
If the default mapping exists, the ambassador will send the request to extauth. The extauth will set the header, and clear the route cache. Then the ambassador will route the request based on the header.

@pinkyjpainadath
Copy link

i am facing same issue. was there any fix identified for this

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants