External Service: no Route matched with those values

[mikyll]

I followed the External Service example guide on the KIC documentation, but when I try to send an HTTP request to localhost/httpbin/anything, I get the classic message "no Route matched with those values", instead of what I would expect.
I tried adding the echo service & route and everything is working correctly... it's just the external service that is causing problems

Is there anything I've done wrong or maybe a step in the documentation is missing something?

Here's a StackOverflow thread I made, where I explain exactly each step I followed: https://stackoverflow.com/questions/77489459/kong-external-service-no-route-matched-with-those-values

[mikyll]

Am I the only one that has this problem? Have anybody tried to configure the external service to httpbin, following the guide, who could tell me if they get the same result or provide any kind of feedback?

[programmer04]

I tried it on my machine (instead of minikube I use kind + lima) and couldn't reproduce the reported problem. I exactly followed this guide. Could you check logs of Controller and Kong Gateway? Maybe they tell us something more

[mikyll]

Here are the logs from both (I'm not sure if that's the correct approach to retrieve them)

Controller Logs

kubectl logs -n kong kong-controller-549fcc4d84-k9wts -c ingress-controller:

Show/Hide logs

2023-11-20T13:45:22Z    info    Diagnostics server disabled     {"v": 0}
2023-11-20T13:45:22Z    info    setup   Starting controller manager     {"v": 0, "release": "3.0.0", "repo": "https://github.com/Kong/kubernetes-ingress-controller.git", "commit": "9585f2484639d0df2b671c1b3655de7d3dea8718"}
2023-11-20T13:45:22Z    info    setup   The ingress class name has been set     {"v": 0, "value": "kong"}
2023-11-20T13:45:22Z    info    setup   Getting enabled options and features    {"v": 0}
2023-11-20T13:45:22Z    info    setup   Getting the kubernetes client configuration     {"v": 0}
W1120 13:45:22.881212       1 client_config.go:618] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
2023-11-20T13:45:22Z    info    setup   Starting standalone health check server {"v": 0}
2023-11-20T13:45:22Z    info    setup   Getting the kong admin api client configuration {"v": 0}
W1120 13:45:22.881470       1 client_config.go:618] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
2023-11-20T13:46:17Z    info    setup   Retrying kong admin api client call after error {"v": 0, "retries": "0/60", "error": "making HTTP request: Get \"https://10.244.0.4:8444/\": dial tcp 10.244.0.4:8444: connect: connection refused"}
2023-11-20T13:46:18Z    info    setup   Retrying kong admin api client call after error {"v": 0, "retries": "1/60", "error": "making HTTP request: Get \"https://10.244.0.4:8444/\": dial tcp 10.244.0.4:8444: connect: connection refused"}
2023-11-20T13:46:19Z    info    setup   Retrying kong admin api client call after error {"v": 0, "retries": "2/60", "error": "making HTTP request: Get \"https://10.244.0.4:8444/\": dial tcp 10.244.0.4:8444: connect: connection refused"}
2023-11-20T13:46:20Z    info    setup   Retrying kong admin api client call after error {"v": 0, "retries": "3/60", "error": "making HTTP request: Get \"https://10.244.0.4:8444/\": dial tcp 10.244.0.4:8444: connect: connection refused"}
2023-11-20T13:46:21Z    info    setup   Tag filtering enabled   {"v": 0, "tags": ["managed-by-ingress-controller"]}
2023-11-20T13:46:21Z    info    setup   Configuring and building the controller manager {"v": 0}
2023-11-20T13:46:21Z    info    setup   Building the manager runtime scheme and loading apis into the scheme    {"v": 0}
2023-11-20T13:46:21Z    info    setup   DB-less mode detected with service detection, enabling leader election  {"v": 0}
2023-11-20T13:46:21Z    info    setup   Initializing Dataplane Client   {"v": 0}
2023-11-20T13:46:21Z    info    setup   Running AdminAPIClientsManager loop     {"v": 0}
2023-11-20T13:46:21Z    info    Gateway is running with non-expression router flavor    {"v": 0, "flavor": "traditional"}
2023-11-20T13:46:21Z    info    setup   Starting Admission Server       {"v": 0}
2023-11-20T13:46:21Z    info    controller-runtime.certwatcher  Updated current TLS certificate {"v": 0}
2023-11-20T13:46:21Z    info    controller-runtime.certwatcher  Starting certificate watcher    {"v": 0}
2023-11-20T13:46:21Z    info    setup   Initializing Dataplane Synchronizer     {"v": 0}
2023-11-20T13:46:21Z    info    setup   Starting Status Updater {"v": 0}
2023-11-20T13:46:21Z    info    setup   Initializing Dataplane address Discovery        {"v": 0}
2023-11-20T13:46:21Z    info    setup   Falling back to a default address finder for UDP        {"v": 0, "reason": "no publish status address or publish service were provided"}
2023-11-20T13:46:21Z    info    setup   Starting Enabled Controllers    {"v": 0}
2023-11-20T13:46:21Z    info    setup   Add readiness probe to health server    {"v": 0}
2023-11-20T13:46:21Z    info    telemetry       starting telemetry manager      {"v": 0}
2023-11-20T13:46:21Z    info    setup   Anonymous reports enabled       {"v": 0}
2023-11-20T13:46:21Z    info    setup   Starting manager        {"v": 0}
2023-11-20T13:46:21Z    info    controller-runtime.metrics      Starting metrics server {"v": 0}
2023-11-20T13:46:21Z    info    controller-runtime.metrics      Serving metrics server  {"v": 0, "bindAddress": ":10255", "secure": false}
I1120 13:46:21.316673       1 leaderelection.go:250] attempting to acquire leader lease kong/kong-ingress-controller-leader-kong...
I1120 13:46:21.329612       1 leaderelection.go:260] successfully acquired lease kong/kong-ingress-controller-leader-kong
2023-11-20T13:46:21Z    info    controllers.IngressClass.netv1  Starting EventSource    {"v": 0, "source": "kind source: *v1.IngressClass"}
2023-11-20T13:46:21Z    info    controllers.IngressClass.netv1  Starting Controller     {"v": 0}
2023-11-20T13:46:21Z    info    controllers.Ingress.netv1       Starting EventSource    {"v": 0, "source": "channel source: 0xc000c14c40"}
2023-11-20T13:46:21Z    info    controllers.TCPIngress  Starting EventSource    {"v": 0, "source": "channel source: 0xc000c14e40"}
2023-11-20T13:46:21Z    info    controllers.TCPIngress  Starting EventSource    {"v": 0, "source": "kind source: *v1.IngressClass"}
2023-11-20T13:46:21Z    info    controllers.TCPIngress  Starting EventSource    {"v": 0, "source": "kind source: *v1beta1.TCPIngress"}
2023-11-20T13:46:21Z    info    controllers.TCPIngress  Starting Controller     {"v": 0}
2023-11-20T13:46:21Z    info    controllers.Service     Starting EventSource    {"v": 0, "source": "kind source: *v1.Service"}
2023-11-20T13:46:21Z    info    controllers.Service     Starting Controller     {"v": 0}
2023-11-20T13:46:21Z    info    controllers.Ingress.netv1       Starting EventSource    {"v": 0, "source": "kind source: *v1.IngressClass"}
2023-11-20T13:46:21Z    info    controllers.Ingress.netv1       Starting EventSource    {"v": 0, "source": "kind source: *v1.Ingress"}
2023-11-20T13:46:21Z    info    controllers.Ingress.netv1       Starting Controller     {"v": 0}
2023-11-20T13:46:21Z    info    controllers.KongAdminAPIService Starting EventSource    {"v": 0, "source": "kind source: *v1.EndpointSlice"}
2023-11-20T13:46:21Z    info    controllers.KongAdminAPIService Starting Controller     {"v": 0}
2023-11-20T13:46:21Z    info    controllers.KongClusterPlugin   Starting EventSource    {"v": 0, "source": "kind source: *v1.IngressClass"}
2023-11-20T13:46:21Z    info    controllers.KongClusterPlugin   Starting EventSource    {"v": 0, "source": "kind source: *v1.KongClusterPlugin"}
2023-11-20T13:46:21Z    info    controllers.KongClusterPlugin   Starting Controller     {"v": 0}
2023-11-20T13:46:21Z    info    controllers.Gateway     Starting EventSource    {"v": 0, "source": "kind source: *v1.Gateway"}
2023-11-20T13:46:21Z    info    controllers.Gateway     Starting EventSource    {"v": 0, "source": "kind source: *v1.GatewayClass"}
2023-11-20T13:46:21Z    info    controllers.UDPIngress  Starting EventSource    {"v": 0, "source": "channel source: 0xc000c14dc0"}
2023-11-20T13:46:21Z    info    controllers.Gateway     Starting EventSource    {"v": 0, "source": "kind source: *v1.Service"}
2023-11-20T13:46:21Z    info    controllers.Gateway     Starting EventSource    {"v": 0, "source": "kind source: *v1.HTTPRoute"}
2023-11-20T13:46:21Z    info    controllers.Gateway     Starting EventSource    {"v": 0, "source": "kind source: *v1beta1.ReferenceGrant"}
2023-11-20T13:46:21Z    info    controllers.Gateway     Starting Controller     {"v": 0}
2023-11-20T13:46:21Z    info    controllers.UDPIngress  Starting EventSource    {"v": 0, "source": "kind source: *v1.IngressClass"}
2023-11-20T13:46:21Z    info    controllers.UDPIngress  Starting EventSource    {"v": 0, "source": "kind source: *v1beta1.UDPIngress"}
2023-11-20T13:46:21Z    info    controllers.ReferenceGrant      Starting EventSource    {"v": 0, "source": "kind source: *v1beta1.ReferenceGrant"}
2023-11-20T13:46:21Z    info    controllers.ReferenceGrant      Starting Controller     {"v": 0}
2023-11-20T13:46:21Z    info    controllers.IngressClassParameters      Starting EventSource    {"v": 0, "source": "kind source: *v1alpha1.IngressClassParameters"}
2023-11-20T13:46:21Z    info    controllers.IngressClassParameters      Starting Controller     {"v": 0}
2023-11-20T13:46:21Z    info    controllers.Secrets     Starting EventSource    {"v": 0, "source": "kind source: *v1.Secret"}
2023-11-20T13:46:21Z    info    controllers.Secrets     Starting Controller     {"v": 0}
2023-11-20T13:46:21Z    info    controllers.KongIngress Starting EventSource    {"v": 0, "source": "kind source: *v1.KongIngress"}
2023-11-20T13:46:21Z    info    controllers.KongIngress Starting Controller     {"v": 0}
2023-11-20T13:46:21Z    info    controllers.Gateway.V1Beta1GatewayClass Starting EventSource    {"v": 0, "source": "kind source: *v1.GatewayClass"}
2023-11-20T13:46:21Z    info    controllers.Gateway.V1Beta1GatewayClass Starting Controller     {"v": 0}
2023-11-20T13:46:21Z    info    controllers.KongPlugin  Starting EventSource    {"v": 0, "source": "kind source: *v1.KongPlugin"}
2023-11-20T13:46:21Z    info    controllers.KongPlugin  Starting Controller     {"v": 0}
2023-11-20T13:46:21Z    info    controllers.KongUpstreamPolicy  Starting EventSource    {"v": 0, "source": "kind source: *v1beta1.KongUpstreamPolicy"}
2023-11-20T13:46:21Z    info    controllers.KongUpstreamPolicy  Starting Controller     {"v": 0}
2023-11-20T13:46:21Z    info    controllers.HTTPRoute   Starting EventSource    {"v": 0, "source": "kind source: *v1.GatewayClass"}
2023-11-20T13:46:21Z    info    controllers.HTTPRoute   Starting EventSource    {"v": 0, "source": "kind source: *v1.Gateway"}
2023-11-20T13:46:21Z    info    controllers.HTTPRoute   Starting EventSource    {"v": 0, "source": "kind source: *v1beta1.ReferenceGrant"}
2023-11-20T13:46:21Z    info    controllers.HTTPRoute   Starting EventSource    {"v": 0, "source": "channel source: 0xc0007acb80"}
2023-11-20T13:46:21Z    info    controllers.KongConsumer        Starting EventSource    {"v": 0, "source": "channel source: 0xc000c14f80"}
2023-11-20T13:46:21Z    info    controllers.KongConsumer        Starting EventSource    {"v": 0, "source": "kind source: *v1.IngressClass"}
2023-11-20T13:46:21Z    info    controllers.KongConsumer        Starting EventSource    {"v": 0, "source": "kind source: *v1.KongConsumer"}
2023-11-20T13:46:21Z    info    controllers.UDPIngress  Starting Controller     {"v": 0}
2023-11-20T13:46:21Z    info    controllers.KongConsumer        Starting Controller     {"v": 0}
2023-11-20T13:46:21Z    info    controllers.HTTPRoute   Starting EventSource    {"v": 0, "source": "kind source: *v1.HTTPRoute"}
2023-11-20T13:46:21Z    info    controllers.EndpointSlice       Starting EventSource    {"v": 0, "source": "kind source: *v1.EndpointSlice"}
2023-11-20T13:46:21Z    info    controllers.EndpointSlice       Starting Controller     {"v": 0}
2023-11-20T13:46:21Z    info    controllers.HTTPRoute   Starting Controller     {"v": 0}
2023-11-20T13:46:21Z    info    controllers.KongConsumerGroup   Starting EventSource    {"v": 0, "source": "channel source: 0xc000c15000"}
2023-11-20T13:46:21Z    info    controllers.KongConsumerGroup   Starting EventSource    {"v": 0, "source": "kind source: *v1.IngressClass"}
2023-11-20T13:46:21Z    info    controllers.KongConsumerGroup   Starting EventSource    {"v": 0, "source": "kind source: *v1beta1.KongConsumerGroup"}
2023-11-20T13:46:21Z    info    controllers.KongConsumerGroup   Starting Controller     {"v": 0}
2023-11-20T13:46:21Z    info    controllers.IngressClass.netv1  Starting workers        {"v": 0, "worker count": 1}
2023-11-20T13:46:21Z    info    controllers.KongAdminAPIService Starting workers        {"v": 0, "worker count": 1}
2023-11-20T13:46:21Z    info    controllers.EndpointSlice       Starting workers        {"v": 0, "worker count": 1}
2023-11-20T13:46:21Z    info    controllers.Gateway.V1Beta1GatewayClass Starting workers        {"v": 0, "worker count": 1}
2023-11-20T13:46:21Z    info    controllers.KongAdminAPIService Reconciling Admin API EndpointSlice     {"v": 0, "namespace": "kong", "name": "kong-gateway-admin-6gfb6"}
2023-11-20T13:46:21Z    info    controllers.TCPIngress  Starting workers        {"v": 0, "worker count": 1}
2023-11-20T13:46:21Z    info    controllers.Service     Starting workers        {"v": 0, "worker count": 1}
2023-11-20T13:46:21Z    info    controllers.KongClusterPlugin   Starting workers        {"v": 0, "worker count": 1}
2023-11-20T13:46:21Z    info    controllers.KongConsumer        Starting workers        {"v": 0, "worker count": 1}
2023-11-20T13:46:21Z    info    controllers.Secrets     Starting workers        {"v": 0, "worker count": 1}
2023-11-20T13:46:21Z    info    controllers.UDPIngress  Starting workers        {"v": 0, "worker count": 1}
2023-11-20T13:46:21Z    info    controllers.KongPlugin  Starting workers        {"v": 0, "worker count": 1}
2023-11-20T13:46:21Z    info    controllers.KongIngress Starting workers        {"v": 0, "worker count": 1}
2023-11-20T13:46:21Z    info    controllers.Ingress.netv1       Starting workers        {"v": 0, "worker count": 1}
2023-11-20T13:46:21Z    info    controllers.IngressClassParameters      Starting workers        {"v": 0, "worker count": 1}
2023-11-20T13:46:21Z    info    controllers.KongUpstreamPolicy  Starting workers        {"v": 0, "worker count": 1}
2023-11-20T13:46:21Z    info    controllers.ReferenceGrant      Starting workers        {"v": 0, "worker count": 1}
2023-11-20T13:46:21Z    info    controllers.KongConsumerGroup   Starting workers        {"v": 0, "worker count": 1}
2023-11-20T13:46:21Z    info    controllers.HTTPRoute   Starting workers        {"v": 0, "worker count": 1}
2023-11-20T13:46:21Z    info    controllers.Gateway     Starting workers        {"v": 0, "worker count": 1}
2023-11-20T13:46:21Z    info    controllers.Gateway     Marking gateway as accepted     {"GatewayV1Gateway": {"name":"kong","namespace":"default"}, "v": 0, "namespace": "default", "name": "kong"}
2023-11-20T13:46:21Z    info    controllers.Gateway     Marking gateway as accepted     {"GatewayV1Gateway": {"name":"kong","namespace":"default"}, "v": 0, "namespace": "default", "name": "kong"}
2023-11-20T13:46:24Z    info    Successfully synced configuration to Kong       {"url": "https://10.244.0.4:8444", "update_strategy": "InMemory", "v": 0}
2023-11-20T13:46:27Z    info    controllers.KongAdminAPIService Reconciling Admin API EndpointSlice     {"v": 0, "namespace": "kong", "name": "kong-gateway-admin-6gfb6"}
2023-11-20T13:47:15Z    info    Successfully synced configuration to Kong       {"url": "https://10.244.0.4:8444", "update_strategy": "InMemory", "v": 0}
2023-11-20T13:47:15Z    info    controllers.HTTPRoute   HTTPRoute has been configured on the data-plane {"GatewayV1HTTPRoute": {"name":"proxy-from-k8s-to-httpbin","namespace":"default"}, "v": 0, "namespace": "default", "name": "proxy-from-k8s-to-httpbin"}
2023-11-20T13:47:15Z    info    controllers.HTTPRoute   HTTPRoute has been configured on the data-plane {"GatewayV1HTTPRoute": {"name":"proxy-from-k8s-to-httpbin","namespace":"default"}, "v": 0, "namespace": "default", "name": "proxy-from-k8s-to-httpbin"}
2023-11-20T13:47:15Z    info    controllers.HTTPRoute   HTTPRoute has been configured on the data-plane {"GatewayV1HTTPRoute": {"name":"proxy-from-k8s-to-httpbin","namespace":"default"}, "v": 0, "namespace": "default", "name": "proxy-from-k8s-to-httpbin"}

Gateway Logs

kubectl logs -n kong kong-gateway-786f56968-v8tp9 -c proxy:

Show/Hide logs

2023/11/20 13:46:19 [warn] 1#0: the "user" directive makes sense only if the master process runs with super-user privileges, ignored in /kong_prefix/nginx.conf:7
nginx: [warn] the "user" directive makes sense only if the master process runs with super-user privileges, ignored in /kong_prefix/nginx.conf:7
2023/11/20 13:46:21 [notice] 1#0: using the "epoll" event method
2023/11/20 13:46:21 [notice] 1#0: openresty/1.21.4.1
2023/11/20 13:46:21 [notice] 1#0: OS: Linux 5.10.102.1-microsoft-standard-WSL2
2023/11/20 13:46:21 [notice] 1#0: getrlimit(RLIMIT_NOFILE): 1048576:1048576
2023/11/20 13:46:21 [notice] 1#0: start worker processes
2023/11/20 13:46:21 [notice] 1#0: start worker process 1262
2023/11/20 13:46:21 [notice] 1#0: start worker process 1263
2023/11/20 13:46:21 [notice] 1263#0: *2 [lua] globalpatches.lua:73: sleep(): executing a blocking 'sleep' (0.001 seconds), context: init_worker_by_lua*
2023/11/20 13:46:21 [notice] 1263#0: *2 [lua] globalpatches.lua:73: sleep(): executing a blocking 'sleep' (0.002 seconds), context: init_worker_by_lua*
2023/11/20 13:46:21 [notice] 1263#0: *2 [lua] globalpatches.lua:73: sleep(): executing a blocking 'sleep' (0.004 seconds), context: init_worker_by_lua*
2023/11/20 13:46:21 [notice] 1262#0: *1 [lua] init.lua:259: purge(): [DB cache] purging (local) cache, context: init_worker_by_lua*
2023/11/20 13:46:21 [notice] 1262#0: *1 [lua] init.lua:259: purge(): [DB cache] purging (local) cache, context: init_worker_by_lua*
10.244.0.3 - - [20/Nov/2023:13:46:21 +0000] "GET / HTTP/2.0" 200 14089 "-" "Go-http-client/2.0"
10.244.0.3 - - [20/Nov/2023:13:46:21 +0000] "GET /tags HTTP/2.0" 200 23 "-" "Go-http-client/2.0"
10.244.0.3 - - [20/Nov/2023:13:46:21 +0000] "GET / HTTP/2.0" 200 14089 "-" "Go-http-client/2.0"
10.244.0.3 - - [20/Nov/2023:13:46:21 +0000] "GET /status HTTP/2.0" 200 1175 "-" "Go-http-client/2.0"
10.244.0.3 - - [20/Nov/2023:13:46:21 +0000] "GET / HTTP/2.0" 200 14089 "-" "Go-http-client/2.0"
10.244.0.3 - - [20/Nov/2023:13:46:21 +0000] "GET / HTTP/2.0" 200 14089 "-" "Go-http-client/2.0"
10.244.0.3 - - [20/Nov/2023:13:46:21 +0000] "GET / HTTP/2.0" 200 14089 "-" "Go-http-client/2.0"
10.244.0.3 - - [20/Nov/2023:13:46:24 +0000] "GET /mtls-auths?size=1000 HTTP/2.0" 404 23 "-" "Go-http-client/2.0"
10.244.0.3 - - [20/Nov/2023:13:46:24 +0000] "GET /consumer_groups?size=1000 HTTP/2.0" 404 23 "-" "Go-http-client/2.0"
10.244.0.3 - - [20/Nov/2023:13:46:24 +0000] "GET /services?size=1000 HTTP/2.0" 200 23 "-" "Go-http-client/2.0"
10.244.0.3 - - [20/Nov/2023:13:46:24 +0000] "GET /routes?size=1000 HTTP/2.0" 200 23 "-" "Go-http-client/2.0"
10.244.0.3 - - [20/Nov/2023:13:46:24 +0000] "GET /plugins?size=1000 HTTP/2.0" 200 23 "-" "Go-http-client/2.0"
10.244.0.3 - - [20/Nov/2023:13:46:24 +0000] "GET /certificates?size=1000 HTTP/2.0" 200 23 "-" "Go-http-client/2.0"
10.244.0.3 - - [20/Nov/2023:13:46:24 +0000] "GET /ca_certificates?size=1000 HTTP/2.0" 200 23 "-" "Go-http-client/2.0"
10.244.0.3 - - [20/Nov/2023:13:46:24 +0000] "GET /snis?size=1000 HTTP/2.0" 200 23 "-" "Go-http-client/2.0"
10.244.0.3 - - [20/Nov/2023:13:46:24 +0000] "GET /upstreams?size=1000 HTTP/2.0" 200 23 "-" "Go-http-client/2.0"
10.244.0.3 - - [20/Nov/2023:13:46:24 +0000] "GET /consumers?size=1000 HTTP/2.0" 200 23 "-" "Go-http-client/2.0"
10.244.0.3 - - [20/Nov/2023:13:46:24 +0000] "GET /key-auths?size=1000 HTTP/2.0" 200 23 "-" "Go-http-client/2.0"
10.244.0.3 - - [20/Nov/2023:13:46:24 +0000] "GET /hmac-auths?size=1000 HTTP/2.0" 200 23 "-" "Go-http-client/2.0"
10.244.0.3 - - [20/Nov/2023:13:46:24 +0000] "GET /jwts?size=1000 HTTP/2.0" 200 23 "-" "Go-http-client/2.0"
10.244.0.3 - - [20/Nov/2023:13:46:24 +0000] "GET /basic-auths?size=1000 HTTP/2.0" 200 23 "-" "Go-http-client/2.0"
10.244.0.3 - - [20/Nov/2023:13:46:24 +0000] "GET /oauth2?size=1000 HTTP/2.0" 200 23 "-" "Go-http-client/2.0"
10.244.0.3 - - [20/Nov/2023:13:46:24 +0000] "GET /acls?size=1000 HTTP/2.0" 200 23 "-" "Go-http-client/2.0"
10.244.0.3 - - [20/Nov/2023:13:46:24 +0000] "GET /vaults?size=1000 HTTP/2.0" 200 23 "-" "Go-http-client/2.0"
10.244.0.3 - - [20/Nov/2023:13:46:24 +0000] "GET /status HTTP/2.0" 200 1176 "-" "Go-http-client/2.0"
2023/11/20 13:46:24 [notice] 1262#0: *297 [lua] init.lua:259: purge(): [DB cache] purging (local) cache, client: 10.244.0.3, server: kong_admin, request: "POST /config?check_hash=1&flatten_errors=1 HTTP/2.0", host: "10.244.0.4:8444"
2023/11/20 13:46:24 [notice] 1262#0: *297 [lua] init.lua:259: purge(): [DB cache] purging (local) cache, client: 10.244.0.3, server: kong_admin, request: "POST /config?check_hash=1&flatten_errors=1 HTTP/2.0", host: "10.244.0.4:8444"
10.244.0.3 - - [20/Nov/2023:13:46:24 +0000] "POST /config?check_hash=1&flatten_errors=1 HTTP/2.0" 201 850 "-" "Go-http-client/2.0"
2023/11/20 13:46:24 [notice] 1263#0: *8 [lua] init.lua:259: purge(): [DB cache] purging (local) cache, context: ngx.timer
2023/11/20 13:46:24 [notice] 1263#0: *8 [lua] init.lua:259: purge(): [DB cache] purging (local) cache, context: ngx.timer
2023/11/20 13:46:24 [notice] 1262#0: *4 [lua] init.lua:259: purge(): [DB cache] purging (local) cache, context: ngx.timer
2023/11/20 13:46:24 [notice] 1262#0: *4 [lua] init.lua:259: purge(): [DB cache] purging (local) cache, context: ngx.timer
10.244.0.3 - - [20/Nov/2023:13:46:27 +0000] "GET /status HTTP/2.0" 200 1176 "-" "Go-http-client/2.0"
10.244.0.3 - - [20/Nov/2023:13:46:30 +0000] "GET /status HTTP/2.0" 200 1176 "-" "Go-http-client/2.0"
10.244.0.3 - - [20/Nov/2023:13:46:31 +0000] "GET /status HTTP/2.0" 200 1176 "-" "Go-http-client/2.0"
10.244.0.3 - - [20/Nov/2023:13:46:33 +0000] "GET /status HTTP/2.0" 200 1176 "-" "Go-http-client/2.0"
10.244.0.3 - - [20/Nov/2023:13:46:36 +0000] "GET /status HTTP/2.0" 200 1176 "-" "Go-http-client/2.0"
10.244.0.3 - - [20/Nov/2023:13:46:39 +0000] "GET /status HTTP/2.0" 200 1176 "-" "Go-http-client/2.0"
10.244.0.3 - - [20/Nov/2023:13:46:41 +0000] "GET /status HTTP/2.0" 200 1176 "-" "Go-http-client/2.0"
10.244.0.3 - - [20/Nov/2023:13:46:42 +0000] "GET /status HTTP/2.0" 200 1176 "-" "Go-http-client/2.0"
10.244.0.1 - - [20/Nov/2023:13:46:44 +0000] "GET / HTTP/1.1" 404 52 "-" "curl/7.76.1"
10.244.0.3 - - [20/Nov/2023:13:46:45 +0000] "GET /status HTTP/2.0" 200 1176 "-" "Go-http-client/2.0"
10.244.0.3 - - [20/Nov/2023:13:46:48 +0000] "GET /status HTTP/2.0" 200 1178 "-" "Go-http-client/2.0"
10.244.0.1 - - [20/Nov/2023:13:46:51 +0000] "GET / HTTP/1.1" 404 52 "-" "curl/8.2.1"
10.244.0.3 - - [20/Nov/2023:13:46:51 +0000] "GET /status HTTP/2.0" 200 1178 "-" "Go-http-client/2.0"
10.244.0.3 - - [20/Nov/2023:13:46:51 +0000] "GET /status HTTP/2.0" 200 1178 "-" "Go-http-client/2.0"
10.244.0.3 - - [20/Nov/2023:13:46:54 +0000] "GET /status HTTP/2.0" 200 1178 "-" "Go-http-client/2.0"
10.244.0.3 - - [20/Nov/2023:13:46:57 +0000] "GET /status HTTP/2.0" 200 1178 "-" "Go-http-client/2.0"
10.244.0.3 - - [20/Nov/2023:13:47:00 +0000] "GET /status HTTP/2.0" 200 1178 "-" "Go-http-client/2.0"
10.244.0.3 - - [20/Nov/2023:13:47:01 +0000] "GET /status HTTP/2.0" 200 1178 "-" "Go-http-client/2.0"
10.244.0.3 - - [20/Nov/2023:13:47:03 +0000] "GET /status HTTP/2.0" 200 1178 "-" "Go-http-client/2.0"
10.244.0.3 - - [20/Nov/2023:13:47:06 +0000] "GET /status HTTP/2.0" 200 1178 "-" "Go-http-client/2.0"
10.244.0.3 - - [20/Nov/2023:13:47:09 +0000] "GET /status HTTP/2.0" 200 1178 "-" "Go-http-client/2.0"
10.244.0.3 - - [20/Nov/2023:13:47:11 +0000] "GET /status HTTP/2.0" 200 1178 "-" "Go-http-client/2.0"
10.244.0.3 - - [20/Nov/2023:13:47:12 +0000] "GET /status HTTP/2.0" 200 1178 "-" "Go-http-client/2.0"
10.244.0.3 - - [20/Nov/2023:13:47:13 +0000] "POST /schemas/routes/validate HTTP/2.0" 200 42 "-" "Go-http-client/2.0"
10.244.0.3 - - [20/Nov/2023:13:47:13 +0000] "GET / HTTP/2.0" 200 14088 "-" "Go-http-client/2.0"
10.244.0.3 - - [20/Nov/2023:13:47:13 +0000] "GET / HTTP/2.0" 200 14088 "-" "Go-http-client/2.0"
2023/11/20 13:47:15 [notice] 1262#0: *297 [lua] init.lua:259: purge(): [DB cache] purging (local) cache, client: 10.244.0.3, server: kong_admin, request: "POST /config?check_hash=1&flatten_errors=1 HTTP/2.0", host: "10.244.0.4:8444"
2023/11/20 13:47:15 [notice] 1262#0: *297 [lua] init.lua:259: purge(): [DB cache] purging (local) cache, client: 10.244.0.3, server: kong_admin, request: "POST /config?check_hash=1&flatten_errors=1 HTTP/2.0", host: "10.244.0.4:8444"
10.244.0.3 - - [20/Nov/2023:13:47:15 +0000] "POST /config?check_hash=1&flatten_errors=1 HTTP/2.0" 201 2479 "-" "Go-http-client/2.0"
2023/11/20 13:47:15 [notice] 1263#0: *8 [lua] init.lua:259: purge(): [DB cache] purging (local) cache, context: ngx.timer
2023/11/20 13:47:15 [notice] 1263#0: *8 [lua] init.lua:259: purge(): [DB cache] purging (local) cache, context: ngx.timer
2023/11/20 13:47:15 [notice] 1262#0: *4 [lua] init.lua:259: purge(): [DB cache] purging (local) cache, context: ngx.timer
2023/11/20 13:47:15 [notice] 1262#0: *4 [lua] init.lua:259: purge(): [DB cache] purging (local) cache, context: ngx.timer
10.244.0.3 - - [20/Nov/2023:13:47:15 +0000] "GET / HTTP/2.0" 200 14088 "-" "Go-http-client/2.0"
10.244.0.3 - - [20/Nov/2023:13:47:15 +0000] "GET / HTTP/2.0" 200 14088 "-" "Go-http-client/2.0"
10.244.0.3 - - [20/Nov/2023:13:47:18 +0000] "GET /status HTTP/2.0" 200 1178 "-" "Go-http-client/2.0"
10.244.0.1 - - [20/Nov/2023:13:47:19 +0000] "GET /anything HTTP/1.1" 404 52 "-" "curl/8.2.1"
10.244.0.1 - - [20/Nov/2023:13:47:19 +0000] "GET /httpbin/anything HTTP/1.1" 404 52 "-" "curl/8.2.1"
10.244.0.3 - - [20/Nov/2023:13:47:21 +0000] "GET /status HTTP/2.0" 200 1178 "-" "Go-http-client/2.0"
10.244.0.3 - - [20/Nov/2023:13:47:21 +0000] "GET /status HTTP/2.0" 200 1178 "-" "Go-http-client/2.0"
10.244.0.3 - - [20/Nov/2023:13:47:24 +0000] "GET /status HTTP/2.0" 200 1178 "-" "Go-http-client/2.0"

[mikyll]

I cannot find anything unusual, just the 404 from

10.244.0.1 - - [20/Nov/2023:13:47:19 +0000] "GET /anything HTTP/1.1" 404 52 "-" "curl/8.2.1"
10.244.0.1 - - [20/Nov/2023:13:47:19 +0000] "GET /httpbin/anything HTTP/1.1" 404 52 "-" "curl/8.2.1"

Meanwhile I can try it with Kind and see if anything changes - and in case provide a feedback

[mikyll]

Update

I tried installing Kong both via Kind and via Docker Desktop Kubernetes, but yet I experience the same issue, only with external services: "no Route matched with those values".

---

I have some Azure credits to waste, so I think I'm going to try and setup Kong on an AKS cluster to see what I can get 🥲