Skip to content
This repository has been archived by the owner on Mar 19, 2024. It is now read-only.

API-Gateway-Controller fails to resolve Vault Service registered in Consul #299

Open
tyler-wendys opened this issue Aug 4, 2022 · 4 comments
Open

API-Gateway-Controller fails to resolve Vault Service registered in Consul #299

tyler-wendys opened this issue Aug 4, 2022 · 4 comments
Labels
type/bug Something isn't working

Comments

@tyler-wendys
Copy link

tyler-wendys commented Aug 4, 2022 •
edited
Loading

Overview of the Issue

The TL;DR:
The api-gateway-controller fails to resolve the consul service registration on deployment of httproutes for vault when deploying Vault with a consul HA backend.

The longer version:
I have a GKE cluster that I’m running Vault with Consul serving as the HA backend. I’ve installed Vault and Consul via helm charts, everything between those two installations appears to be playing nice. I’m now attempting to set up Consul API Gateway with that Consul cluster to set up some (internal) ingress traffic for a vault-injector living in another cluster. Everything is working together flawlessly until it comes to setting up the HTTPRoute, when doing so the api-gateway-controller fails to resolve the the consul service registration for Vault.

Reproduction Steps

Here's all of the things we've got setup for the installations of Vault + Consul + API Gateway:

Consul helm overrides:

global:
  name: consul
  gossipEncryption: 
    autoGenerate: true
  tls:
    enabled: true
    enableAutoEncrypt: true
    verify: true
    httpsOnly: false
  acls:
    enabled: true
    manageSystemACLs: true
    default_policy: "allow"
    enable_token_persistence: true
ui:
  enabled: true
  type: "LoadBalancer"
client:
  enabled: true
connectInject:
  enabled: true
controller:
  enabled: true
terminatingGateways:
  enabled: false
ingressGateways:
  enabled: false
apiGateway:
  enabled: true
  image: hashicorp/consul-api-gateway:0.3.0
  managedGatewayClass:
    copyAnnotations:
      service: 
        annotations: |
          - 'networking.gke.io/load-balancer-type'

Vault helm overrides:

global:
  enabled: true
  tlsDisable: true
injector:
  enabled: false
  logLevel: debug
  webhook:
    failurePolicy: Fail
  image:
    repository: hashicorp/vault-k8s
    tag: latest
  resources:
    requests:
      memory: 256Mi
      cpu: 250m
    limits:
      memory: 256Mi
      cpu: 250m
server:
  resources:
    requests:
      memory: 8Gi
      cpu: 2000m
    limits:
      memory: 16Gi
      cpu: 2000m
  readinessProbe:
    enabled: true
    path: /v1/sys/health?standbyok=true&sealedcode=204&uninitcode=204
  livenessProbe:
    enabled: true
    path: /v1/sys/health?standbyok=true
    initialDelaySeconds: 60
  auditStorage:
    enabled: true
  standalone:
    enabled: false
  ha:
    enabled: true
    replicas: 5
    config: |

      ui = true

      listener "tcp" {
        tls_disable = true
        address = "[::]:8200"
        cluster_address = "[::]:8201"    
      }

      storage "consul" {
          path = "vault/"
          address = "<consul_addr>"
          token = "<consul_token>"
          scheme = "https"
          tls_skip_verify = true
      }

API Gateway resources: (I'm including the TLS block in case it turns out to be a clue, but I have validated that the api gateway at least receives traffic ok over ssl)

apiVersion: gateway.networking.k8s.io/v1alpha2
kind: Gateway
metadata:
  name: vault-gateway
  namespace: vault
  annotations:
    networking.gke.io/load-balancer-type: 'Internal'
spec:
  gatewayClassName: consul-api-gateway
  listeners:
    - protocol: HTTPS
      hostname: internal.vault.hostname
      port: 443
      name: https
      allowedRoutes:
        namespaces:
          from: Same
      tls:
        certificateRefs:
          - name: vault-ingress-certificate
---
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: HTTPRoute
metadata:
  name: vault-route
  namespace: vault
spec:
  parentRefs:
    - name: vault-gateway
  rules:
    - backendRefs:
        - kind: Service
          name: vault
          namespace: vault
          port: 8200
---
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: ReferencePolicy
metadata:
  name: vault-ref-policy-route
  namespace: vault
spec:
  from:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
      namespace: vault
  to:
    - group: ""
      kind: Service
      name: vault

Logs

2022-08-04T20:42:28.917Z [ERROR] service/resolver.go:249: consul-api-gateway-server.k8s.Reconciler: could not resolve consul service: error="consul service vault/vault not found"
2022-08-04T20:43:00.370Z [ERROR] service/resolver.go:249: consul-api-gateway-server.k8s.Reconciler: could not resolve consul service: error="consul service vault/vault not found"
2022-08-04T20:43:31.315Z [ERROR] service/resolver.go:249: consul-api-gateway-server.k8s.Reconciler: could not resolve consul service: error="consul service vault/vault not found"

The HTTPRoute registration

apiVersion: v1
items:
- apiVersion: gateway.networking.k8s.io/v1alpha2
  kind: HTTPRoute
  metadata:
    name: vault-route
    namespace: vault
  spec:
    parentRefs:
    - group: gateway.networking.k8s.io
      kind: Gateway
      name: vault-gateway
    rules:
    - backendRefs:
      - group: ""
        kind: Service
        name: vault
        port: 8200
        weight: 1
      matches:
      - path:
          type: PathPrefix
          value: /
  status:
    parents:
    - conditions:
      - lastTransitionTime: "2022-08-04T20:42:28Z"
        message: Route accepted.
        observedGeneration: 3
        reason: Accepted
        status: "True"
        type: Accepted
      - lastTransitionTime: "2022-08-04T20:42:28Z"
        message: 'consul: consul service vault/vault not found'
        observedGeneration: 3
        reason: ConsulServiceNotFound
        status: "False"
        type: ResolvedRefs
      controllerName: hashicorp.com/consul-api-gateway-controller
      parentRef:
        group: gateway.networking.k8s.io
        kind: Gateway
        name: vault-gateway
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

Expected behavior

The HTTPRoute registers both the kubernetes and consul service registrations for vault and allows for the api gateway to route traffic to vault.

Environment details

  • consul-api-gateway version: 0.3.0
  • configuration used to deploy the gateway controller: Listed above as part of the consul helm chart overrides
  • Kubernetes version: v1.21.12-gke.1700 (couldn't find the exact k8s version this ties back to, not sure if it's a 1 to 1)
  • Consul Server version: v1.10.1
  • Cloud Provider (If self-hosted, the Kubernetes provider utilized): GKE,
  • Networking CNI plugin in use: None (GKE is using default kubenet as far as I can tell)

Additional Context

In case this helps, below is also a listing of the vault service registered in consul.

{
  "vault:10.4.0.59:8200": {
    "ID": "vault:10.4.0.59:8200",
    "Service": "vault",
    "Tags": [
      "standby",
      "initialized"
    ],
    "Meta": {
      "external-source": "vault"
    },
    "Port": 8200,
    "Address": "10.4.0.59",
    "TaggedAddresses": {
      "lan_ipv4": {
        "Address": "10.4.0.59",
        "Port": 8200
      },
      "wan_ipv4": {
        "Address": "10.4.0.59",
        "Port": 8200
      }
    },
    "Weights": {
      "Passing": 1,
      "Warning": 1
    },
    "EnableTagOverride": false,
    "Datacenter": "dc1"
  },
  "vault:10.4.1.89:8200": {
    "ID": "vault:10.4.1.89:8200",
    "Service": "vault",
    "Tags": [
      "active",
      "initialized"
    ],
    "Meta": {
      "external-source": "vault"
    },
    "Port": 8200,
    "Address": "10.4.1.89",
    "TaggedAddresses": {
      "lan_ipv4": {
        "Address": "10.4.1.89",
        "Port": 8200
      },
      "wan_ipv4": {
        "Address": "10.4.1.89",
        "Port": 8200
      }
    },
    "Weights": {
      "Passing": 1,
      "Warning": 1
    },
    "EnableTagOverride": false,
    "Datacenter": "dc1"
  },
  "vault:10.4.3.5:8200": {
    "ID": "vault:10.4.3.5:8200",
    "Service": "vault",
    "Tags": [
      "standby",
      "initialized"
    ],
    "Meta": {
      "external-source": "vault"
    },
    "Port": 8200,
    "Address": "10.4.3.5",
    "TaggedAddresses": {
      "lan_ipv4": {
        "Address": "10.4.3.5",
        "Port": 8200
      },
      "wan_ipv4": {
        "Address": "10.4.3.5",
        "Port": 8200
      }
    },
    "Weights": {
      "Passing": 1,
      "Warning": 1
    },
    "EnableTagOverride": false,
    "Datacenter": "dc1"
  }
}

Thank you for taking a look at this!
@mikemorris
Copy link
Contributor

mikemorris commented Aug 8, 2022 •
edited
Loading

It looks like the error you're hitting is https://github.com/hashicorp/consul-api-gateway/blob/v0.3.0/internal/k8s/service/resolver.go#L244 - could you try capturing Consul API Gateway controller logs with logLevel: trace configured in the Helm chart? (trace doesn't appear to be documented in the Helm config, but should pass through to set https://pkg.go.dev/github.com/hashicorp/go-hclog#Level) which will hopefully offer more detail as to where specifically this lookup is failing.

The code path you're getting an error from iterates over services registered to all Consul nodes - what I'm not quite sure about is how the Vault backend integration registers itself with Consul - it may be as an external service which could be an edge case our logic doesn't cover yet.

@tyler-wendys
Copy link
Author

Thank you so much for getting back to me! We tried bumping the logLevel to trace, resulting in this message back from the controller:

2022-08-08T19:30:03.455Z [TRACE] reconciler/route.go:251: consul-api-gateway-server.k8s.Reconciler.route: syncing route status: name=vault-route
  status=
  | {
  |   "parents": [
  |     {
  |       "parentRef": {
  |         "group": "gateway.networking.k8s.io",
  |         "kind": "Gateway",
  |         "name": "vault-gateway"
  |       },
  |       "controllerName": "hashicorp.com/consul-api-gateway-controller",
  |       "conditions": [
  |         {
  |           "type": "Accepted",
  |           "status": "True",
  |           "observedGeneration": 4,
  |           "lastTransitionTime": "2022-08-08T19:30:03Z",
  |           "reason": "Accepted",
  |           "message": "Route accepted."
  |         },
  |         {
  |           "type": "ResolvedRefs",
  |           "status": "False",
  |           "observedGeneration": 4,
  |           "lastTransitionTime": "2022-08-08T19:30:03Z",
  |           "reason": "ConsulServiceNotFound",
  |           "message": "consul: consul service vault/vault not found"
  |         }
  |       ]
  |     }
  |   ]
  | }
  
    
2022-08-08T19:30:03.465Z [TRACE] reconciler/gateway.go:523: consul-api-gateway-server.k8s.Reconciler.gateway: created or updated gateway service: name=vault-gateway namespace=vault
  service=
  | {
  |   "kind": "Service",
  |   "apiVersion": "v1",
  |   "metadata": {
  |     "name": "vault-gateway",
  |     "namespace": "vault",
  |     "uid": "5d9dd04e-a5a0-4778-a69b-5a0cec17e0b2",
  |     "resourceVersion": "846248",
  |     "creationTimestamp": "2022-08-04T20:42:28Z",
  |     "labels": {
  |       "api-gateway.consul.hashicorp.com/created": "1659645717",
  |       "api-gateway.consul.hashicorp.com/managed": "true",
  |       "api-gateway.consul.hashicorp.com/name": "vault-gateway",
  |       "api-gateway.consul.hashicorp.com/namespace": "vault"
  |     },
  |     "annotations": {
  |       "cloud.google.com/neg": "{\"ingress\":true}",
  |       "external-dns.alpha.kubernetes.io/hostname": "example.hostname",
  |       "networking.gke.io/load-balancer-type": "Internal"
  |     },
  |     "ownerReferences": [
  |       {
  |         "apiVersion": "gateway.networking.k8s.io/v1alpha2",
  |         "kind": "Gateway",
  |         "name": "vault-gateway",
  |         "uid": "0b9a4e07-836d-41e8-b5bf-58301dfec862",
  |         "controller": true,
  |         "blockOwnerDeletion": true
  |       }
  |     ],
  |     "finalizers": [
  |       "gke.networking.io/l4-ilb-v1",
  |       "service.kubernetes.io/load-balancer-cleanup"
  |     ],
  |     "managedFields": [
  |       {
  |         "manager": "consul-api-gateway",
  |         "operation": "Update",
  |         "apiVersion": "v1",
  |         "time": "2022-08-04T20:42:28Z",
  |         "fieldsType": "FieldsV1",
  |         "fieldsV1": {
  |           "f:metadata": {
  |             "f:annotations": {
  |               ".": {},
  |               "f:external-dns.alpha.kubernetes.io/hostname": {},
  |               "f:networking.gke.io/load-balancer-type": {}
  |             },
  |             "f:labels": {
  |               ".": {},
  |               "f:api-gateway.consul.hashicorp.com/created": {},
  |               "f:api-gateway.consul.hashicorp.com/managed": {},
  |               "f:api-gateway.consul.hashicorp.com/name": {},
  |               "f:api-gateway.consul.hashicorp.com/namespace": {}
  |             },
  |             "f:ownerReferences": {
  |               ".": {},
  |               "k:{\"uid\":\"0b9a4e07-836d-41e8-b5bf-58301dfec862\"}": {
  |                 ".": {},
  |                 "f:apiVersion": {},
  |                 "f:blockOwnerDeletion": {},
  |                 "f:controller": {},
  |                 "f:kind": {},
  |                 "f:name": {},
  |                 "f:uid": {}
  |               }
  |             }
  |           },
  |           "f:spec": {
  |             "f:externalTrafficPolicy": {},
  |             "f:ports": {
  |               ".": {},
  |               "k:{\"port\":443,\"protocol\":\"TCP\"}": {
  |                 ".": {},
  |                 "f:name": {},
  |                 "f:port": {},
  |                 "f:protocol": {},
  |                 "f:targetPort": {}
  |               }
  |             },
  |             "f:selector": {
  |               ".": {},
  |               "f:api-gateway.consul.hashicorp.com/created": {},
  |               "f:api-gateway.consul.hashicorp.com/managed": {},
  |               "f:api-gateway.consul.hashicorp.com/name": {},
  |               "f:api-gateway.consul.hashicorp.com/namespace": {}
  |             },
  |             "f:sessionAffinity": {},
  |             "f:type": {}
  |           }
  |         }
  |       },
  |       {
  |         "manager": "kube-controller-manager",
  |         "operation": "Update",
  |         "apiVersion": "v1",
  |         "time": "2022-08-04T20:43:06Z",
  |         "fieldsType": "FieldsV1",
  |         "fieldsV1": {
  |           "f:metadata": {
  |             "f:finalizers": {
  |               ".": {},
  |               "v:\"gke.networking.io/l4-ilb-v1\"": {},
  |               "v:\"service.kubernetes.io/load-balancer-cleanup\"": {}
  |             }
  |           },
  |           "f:status": {
  |             "f:loadBalancer": {
  |               "f:ingress": {}
  |             }
  |           }
  |         }
  |       }
  |     ]
  |   },
  |   "spec": {
  |     "ports": [
  |       {
  |         "name": "https",
  |         "protocol": "TCP",
  |         "port": 443,
  |         "targetPort": 443,
  |         "nodePort": 32520
  |       }
  |     ],
  |     "selector": {
  |       "api-gateway.consul.hashicorp.com/created": "1659645717",
  |       "api-gateway.consul.hashicorp.com/managed": "true",
  |       "api-gateway.consul.hashicorp.com/name": "vault-gateway",
  |       "api-gateway.consul.hashicorp.com/namespace": "vault"
  |     },
  |     "clusterIP": "10.192.51.251",
  |     "clusterIPs": [
  |       "10.192.51.251"
  |     ],
  |     "type": "LoadBalancer",
  |     "sessionAffinity": "None",
  |     "externalTrafficPolicy": "Cluster",
  |     "ipFamilies": [
  |       "IPv4"
  |     ],
  |     "ipFamilyPolicy": "SingleStack"
  |   },
  |   "status": {
  |     "loadBalancer": {
  |       "ingress": [
  |         {
  |           "ip": "192.2.0.12"
  |         }
  |       ]
  |     }
  |   }
  | }
  
2022-08-08T19:30:03.467Z [TRACE] memory/store.go:227: consul-api-gateway-server.state: detected gateway state change: service=vault-gateway namespace=""
2022-08-08T19:30:03.467Z [TRACE] memory/gateway.go:53: consul-api-gateway-server.state: checking if route can bind to gateway: gateway.consul.namespace="" gateway.consul.service=vault-gateway route=http-vault/vault-route
2022-08-08T19:30:03.467Z [TRACE] memory/gateway.go:58: consul-api-gateway-server.state: checking if route can bind to listener: gateway.consul.namespace="" gateway.consul.service=vault-gateway listener=https route=http-vault/vault-route
2022-08-08T19:30:03.467Z [TRACE] reconciler/listener.go:336: consul-api-gateway-server.k8s.Reconciler.gateway.listener: checking route parent ref: listener=https name=vault-gateway namespace=vault name=vault-gateway
2022-08-08T19:30:03.467Z [TRACE] reconciler/listener.go:339: consul-api-gateway-server.k8s.Reconciler.gateway.listener: checking gateway match: listener=https name=vault-gateway namespace=vault expected=vault/vault-gateway found=vault/vault-gateway
2022-08-08T19:30:03.467Z [TRACE] reconciler/listener.go:356: consul-api-gateway-server.k8s.Reconciler.gateway.listener: listener not ready, unable to bind: listener=https name=vault-gateway namespace=vault route=http-vault/vault-route
2022-08-08T19:30:03.477Z [TRACE] reconciler/gateway.go:523: consul-api-gateway-server.k8s.Reconciler.gateway: created or updated gateway service: name=vault-gateway namespace=vault
  service=
  | {
  |   "kind": "Service",
  |   "apiVersion": "v1",
  |   "metadata": {
  |     "name": "vault-gateway",
  |     "namespace": "vault",
  |     "uid": "5d9dd04e-a5a0-4778-a69b-5a0cec17e0b2",
  |     "resourceVersion": "846248",
  |     "creationTimestamp": "2022-08-04T20:42:28Z",
  |     "labels": {
  |       "api-gateway.consul.hashicorp.com/created": "1659645717",
  |       "api-gateway.consul.hashicorp.com/managed": "true",
  |       "api-gateway.consul.hashicorp.com/name": "vault-gateway",
  |       "api-gateway.consul.hashicorp.com/namespace": "vault"
  |     },
  |     "annotations": {
  |       "cloud.google.com/neg": "{\"ingress\":true}",
  |       "external-dns.alpha.kubernetes.io/hostname": "example.hostname",
  |       "networking.gke.io/load-balancer-type": "Internal"
  |     },
  |     "ownerReferences": [
  |       {
  |         "apiVersion": "gateway.networking.k8s.io/v1alpha2",
  |         "kind": "Gateway",
  |         "name": "vault-gateway",
  |         "uid": "0b9a4e07-836d-41e8-b5bf-58301dfec862",
  |         "controller": true,
  |         "blockOwnerDeletion": true
  |       }
  |     ],
  |     "finalizers": [
  |       "gke.networking.io/l4-ilb-v1",
  |       "service.kubernetes.io/load-balancer-cleanup"
  |     ],
  |     "managedFields": [
  |       {
  |         "manager": "consul-api-gateway",
  |         "operation": "Update",
  |         "apiVersion": "v1",
  |         "time": "2022-08-04T20:42:28Z",
  |         "fieldsType": "FieldsV1",
  |         "fieldsV1": {
  |           "f:metadata": {
  |             "f:annotations": {
  |               ".": {},
  |               "f:external-dns.alpha.kubernetes.io/hostname": {},
  |               "f:networking.gke.io/load-balancer-type": {}
  |             },
  |             "f:labels": {
  |               ".": {},
  |               "f:api-gateway.consul.hashicorp.com/created": {},
  |               "f:api-gateway.consul.hashicorp.com/managed": {},
  |               "f:api-gateway.consul.hashicorp.com/name": {},
  |               "f:api-gateway.consul.hashicorp.com/namespace": {}
  |             },
  |             "f:ownerReferences": {
  |               ".": {},
  |               "k:{\"uid\":\"0b9a4e07-836d-41e8-b5bf-58301dfec862\"}": {
  |                 ".": {},
  |                 "f:apiVersion": {},
  |                 "f:blockOwnerDeletion": {},
  |                 "f:controller": {},
  |                 "f:kind": {},
  |                 "f:name": {},
  |                 "f:uid": {}
  |               }
  |             }
  |           },
  |           "f:spec": {
  |             "f:externalTrafficPolicy": {},
  |             "f:ports": {
  |               ".": {},
  |               "k:{\"port\":443,\"protocol\":\"TCP\"}": {
  |                 ".": {},
  |                 "f:name": {},
  |                 "f:port": {},
  |                 "f:protocol": {},
  |                 "f:targetPort": {}
  |               }
  |             },
  |             "f:selector": {
  |               ".": {},
  |               "f:api-gateway.consul.hashicorp.com/created": {},
  |               "f:api-gateway.consul.hashicorp.com/managed": {},
  |               "f:api-gateway.consul.hashicorp.com/name": {},
  |               "f:api-gateway.consul.hashicorp.com/namespace": {}
  |             },
  |             "f:sessionAffinity": {},
  |             "f:type": {}
  |           }
  |         }
  |       },
  |       {
  |         "manager": "kube-controller-manager",
  |         "operation": "Update",
  |         "apiVersion": "v1",
  |         "time": "2022-08-04T20:43:06Z",
  |         "fieldsType": "FieldsV1",
  |         "fieldsV1": {
  |           "f:metadata": {
  |             "f:finalizers": {
  |               ".": {},
  |               "v:\"gke.networking.io/l4-ilb-v1\"": {},
  |               "v:\"service.kubernetes.io/load-balancer-cleanup\"": {}
  |             }
  |           },
  |           "f:status": {
  |             "f:loadBalancer": {
  |               "f:ingress": {}
  |             }
  |           }
  |         }
  |       }
  |     ]
  |   },
  |   "spec": {
  |     "ports": [
  |       {
  |         "name": "https",
  |         "protocol": "TCP",
  |         "port": 443,
  |         "targetPort": 443,
  |         "nodePort": 32520
  |       }
  |     ],
  |     "selector": {
  |       "api-gateway.consul.hashicorp.com/created": "1659645717",
  |       "api-gateway.consul.hashicorp.com/managed": "true",
  |       "api-gateway.consul.hashicorp.com/name": "vault-gateway",
  |       "api-gateway.consul.hashicorp.com/namespace": "vault"
  |     },
  |     "clusterIP": "<cluster_ip>",
  |     "clusterIPs": [
  |       "<cluster_ip>"
  |     ],
  |     "type": "LoadBalancer",
  |     "sessionAffinity": "None",
  |     "externalTrafficPolicy": "Cluster",
  |     "ipFamilies": [
  |       "IPv4"
  |     ],
  |     "ipFamilyPolicy": "SingleStack"
  |   },
  |   "status": {
  |     "loadBalancer": {
  |       "ingress": [
  |         {
  |           "ip": "192.2.0.12"
  |         }
  |       ]
  |     }
  |   }
  | }

Definitely more info coming back, but we haven't seen anything to clue us in on whether or not vault is registering itself as an external service. We're going to keep doing some digging on our end as to whether or not that's the case. Out of curiosity what is your timeline for external services support?

@mikemorris
Copy link
Contributor

mikemorris commented Aug 8, 2022 •
edited
Loading

Hmm, those logs mostly look like the gateway itself spinning up normally, the only thing that looked off is

reconciler/listener.go:356: consul-api-gateway-server.k8s.Reconciler.gateway.listener: listener not ready, unable to bind: listener=https name=vault-gateway namespace=vault route=http-vault/vault-route

but I'm guessing that was a temporary state given that the route eventually gets an Accepted { status: true } condition.

It may be worth checking the ListenerStatus field on the Gateway to make sure everything looks okay first, then checking the controller logs specifically for errors originating from service/resolver.go to see if anything shows up from https://github.com/hashicorp/consul-api-gateway/blob/v0.3.0/internal/k8s/service/resolver.go#L282-L331

(EDIT: Looking more closely, it appears that Vault uses a standard agent.ServiceRegister call to register itself in Consul, so I don't think external services would be relevant for this case.)

@nathancoleman nathancoleman added the type/bug Something isn't working label Nov 29, 2022
@nathancoleman
Copy link
Member

nathancoleman commented Nov 30, 2022 •
edited
Loading

I did a little bit of digging here today and now understand more as to why Consul API Gateway's controller isn't able to resolve the reference to the vault Service.

The ServiceMeta that the controller filters on here - specifically k8s-namespace and k8s-service-name - are not present on the Consul service corresponding with the Kubernetes Service named "vault".

For a "normal" Service that I create myself, you can see the metadata referenced above is present:

$ curl https://localhost:8501/v1/catalog/service/echo-1 --insecure --silent | jq
[
  {
    ...
    "ServiceID": "echo-1-bf966d7c9-nt5dh-echo-1",
    "ServiceName": "echo-1",
    "ServiceTags": [],
    "ServiceMeta": {
      "k8s-namespace": "default",
      "k8s-service-name": "echo-1",
      "managed-by": "consul-k8s-endpoints-controller",
      "pod-name": "echo-1-bf966d7c9-nt5dh",
      "synthetic-node": "true"
    },
    "CreateIndex": 2508,
    "ModifyIndex": 2508
  }
]

For the Service created by the vault Helm chart, it is missing:

$ curl https://localhost:8501/v1/catalog/service/vault --insecure --silent | jq
[
  {
    ...
    "ServiceID": "vault:10.60.0.9:8200",
    "ServiceName": "vault",
    "ServiceTags": [
      "standby"
    ],
    "ServiceMeta": {
      "external-source": "vault"
    },
    "CreateIndex": 1737,
    "ModifyIndex": 1737
  }
  ...
]

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
type/bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@mikemorris @nathancoleman @tyler-wendys