ERROR ERROR ERROR Starting with configuration errors - already defined Module Ambassador with multi-ambassador setup

[pkalantri-ebsco]

Describe the bug
We have defined 2 emissary-ingresses in 2 different namespaces - api-emissary and medsapi-emissary.

Below are the non-substituted values files for both configs :
medsapi-ambassador ->

replicaCount: {{ .Values | get "replicaCount" }}
fullnameOverride: medsapi-ambassador-ingress-{{ .Values | get "env" }}
createDefaultListeners: false
podLabels:
  tags.datadoghq.com/env: {{ .Values | get "env" }}
  team: cd.sre
affinity:
  podAntiAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchExpressions:
            - key: kube_app_name
              operator: In
              values:
                - medsapi-ambassador-ingress
        topologyKey: "kubernetes.io/hostname"
env:
  AMBASSADOR_ID: medsapi-ambassador-ingress

ingressClassResource:
  enabled: false
agent:
  enabled: false

resources:
  limits:
    cpu: 1000m
    memory: 2048Mi
  requests:
    cpu: 200m
    memory: 300Mi

service:
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: {{ .Values | get "certificateARN" }}
    service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "http"
    getambassador.io/config: |
        ---
        apiVersion: getambassador.io/v3alpha1
        kind:  Module
        name:  ambassador
        ambassador_id: [ medsapi-ambassador-ingress ]
        config:
          envoy_log_type: json
  ports:
    - name: https
      port: 443
      targetPort: 8443
    - name: http
      port: 80
      targetPort: 8080

module: {}

agent:
  enabled: false

api-ambassador ->

replicaCount: {{ .Values | get "replicaCount" }}
fullnameOverride: api-ambassador-ingress-{{ .Values | get "env" }}
createDefaultListeners: true
podLabels:
  tags.datadoghq.com/env: {{ .Values | get "env" }}
  team: cd.sre
affinity:
  podAntiAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchExpressions:
            - key: kube_app_name
              operator: In
              values:
                - api-ambassador-ingress
        topologyKey: "kubernetes.io/hostname"
env:
  AMBASSADOR_ID: api-ambassador-ingress
resources:
  limits:
    cpu: 1000m
    memory: 2048Mi
  requests:
    cpu: 200m
    memory: 300Mi

service:
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: {{ .Values | get "certificateARN" }}
    service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "http"
    getambassador.io/config: |
        ---
        apiVersion: getambassador.io/v3alpha1
        kind:  Module
        name:  ambassador
        ambassador_id: [ api-ambassador-ingress ]
        config:
          envoy_log_type: json
          x_forwarded_proto_redirect: true
          use_remote_address: false

  ports:
    - name: https
      port: 443
      targetPort: 8443
    - name: http
      port: 80
      targetPort: 8080

module: {}

agent:
  enabled: false

We are getting the below error within our emissary logs ->

2024-07-26 14:16:58 diagd 3.9.1 [P18TAEW] INFO: api-ambassador-ingress-live.api-emissary.1: <RichStatus BAD error='<Module api-ambassador-ingress-live.api-emissary.1> defines Module ambassador, which is already defined by ambassador.api-emissary.1' hostname='api-ambassador-ingress-live-677fdd85c9-qgfgz' version='3.9.1'>
2024-07-26 14:16:58 diagd 3.9.1 [P18TAEW] ERROR: ERROR ERROR ERROR Starting with configuration errors

Similar error shows up in medsapi-ambassador logs.

We think it has something to do with the module but we have different ambassador-id's and different namespaces for both modules as given in the documentation here : https://www.getambassador.io/docs/emissary/latest/topics/running/ambassador

Expected behavior
Since both of our ambassadors are set up in different namespaces with different ambassador-id annotations, we should not be getting the above config error log.

Versions (please complete the following information):
Emissary - Chart Version 8.9.1
EKS - v1.29

Additional context
Any help on this would be really appreciated.

[pkalantri-ebsco]

Is there any update on this?

[cindymullins-dw]

HI @pkalantri-ebsco , naming the module 'ambassador' and using annotations looks correct. Is it possible there is another module defined, perhaps by default? I think you can run something like kubectl describe module to check for that.

[cindymullins-dw]

I see you're using annotations - your Module is getting created but perhaps not with the right config. Pls see config example here. You may be able to use annotations if you follow this format and I'll attach an example for reference. Note that the Modules in the annotations must use apiVersion: ambassador/v2 for them to take effect if you are on v3 of Edge Stack and also the ambassador_id is not specified as an array.

getambassador.io/config: |
    ---
    apiVersion: ambassador/v2
    kind:  Module
    name:  ambassador
    ambassador_id: api-ambassador-ingress
    config:
      envoy_log_type: json
      x_forwarded_proto_redirect: true
      use_remote_address: false