No endpoints for Consul service registered in datacenter not named 'dc1'

[ewbankkit]

Describe the bug

We have Consul services registered in a Consul datacenter NOT named dc1 but Ambassador is not resolving endpoints for this service to pass on to Envoy and so the service is not routable.

On the Ambassador Kubernetes service:

  annotations:
    getambassador.io/config: |
      ---
      apiVersion: ambassador/v1
      kind: Module
      name: ambassador
      config:
        service_port: 8080
      ---
      apiVersion: ambassador/v1
      kind: ConsulResolver
      name: consul-not-dc1
      address: "${HOST_IP}:8500"
      datacenter: not-dc1

On a Kubernetes service:

apiVersion: v1
kind: Service
metadata:
  name: consul-demoapp
  annotations:
    getambassador.io/config: |
      ---
      apiVersion: ambassador/v1
      kind:  Mapping
      name:  consul_demoapp_mapping
      prefix: /demoapp
      resolver: consul-not-dc1
      service: demoapp
      load_balancer:
        policy: round_robin
spec:
  clusterIP: None

In snapshot.yaml I see

{
    "Consul": {
            "demoapp": {
                "Id": "not-dc1",
                "Service": "demoapp",
                "Endpoints": [
                    {
                        "SystemID": "consul::00000000-0000-0000-0000-000000000000",
                        "ID": "_nomad-task-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
                        "Service": "demoapp",
                        "Address": "x.x.x.x",
                        "Port": 26457,
                        "Tags": [
                            "urlprefix-/demoapp strip=/demoapp",
                            "demoapp",
                            "blue"
                        ]
                    },
                    {
                        "SystemID": "consul::00000000-0000-0000-0000-000000000000",
                        "ID": "_nomad-task-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
                        "Service": "demoapp",
                        "Address": "x.x.x.x",
                        "Port": 30147,
                        "Tags": [
                            "urlprefix-/demoapp strip=/demoapp",
                            "demoapp",
                            "blue"
                        ]
                    },
                    {
                        "SystemID": "consul::00000000-0000-0000-0000-000000000000",
                        "ID": "_nomad-task-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
                        "Service": "demoapp",
                        "Address": "x.x.x.x",
                        "Port": 24860,
                        "Tags": [
                            "urlprefix-/demoapp strip=/demoapp",
                            "demoapp",
                            "blue"
                        ]
                    }
                ]
            }
        }
    },
    "Kubernetes": {
        ...
    }
}

but in aconf.json I see:

        "consul-demoapp-dc1": {
            "_referenced_by": {},
            "ambassador_id": "default",
            "apiVersion": "ambassador/v1",
            "datacenter": "dc1",
            "endpoints": {
                "*": [
                    {
                        "ip": "x.x.x.x",
                        "port": 26457,
                        "target_kind": "Consul"
                    },
                    {
                        "ip": "x.x.x.x",
                        "port": 30147,
                        "target_kind": "Consul"
                    },
                    {
                        "ip": "x.x.x.x",
                        "port": 24860,
                        "target_kind": "Consul"
                    }
                ]
            },
            "kind": "Service",
            "location": "consul-demoapp-dc1",
            "name": "demoapp",
            "rkey": "consul-demoapp-dc1",
            "serialization": "ambassador_id: default\napiVersion: ambassador/v1\ndatacenter: dc1\nendpoints:\n  '*':\n  - ip: x.x.x.x\n    port: 26457\n    target_kind: Consul\n  - ip: x.x.x.x\n    port: 30147\n    target_kind: Consul\n  - ip: x.x.x.x\n    port: 24860\n    target_kind: Consul\nkind: Service\nname: demoapp\n"
        },

the datacenter value has reverted to dc1 and I think this is causing the failure to find endpoints.

It looks like this code

https://github.com/datawire/ambassador/blob/0e1e8ffbc83f7a96b815ff68ed786b59be09cce3/ambassador/ambassador/config/resourcefetcher.py#L536

is incorrect and 'id' should be 'Id' as that is the name of the datacenter value in snapshot.yaml.

Versions (please complete the following information):

• Ambassador: 0.61.0
• Kubernetes environment BYO Kubernetes on AWS
• Version 1.14

Additional context
Related:

• IR stack dump for a Consul service with no endpoints #1532