Wrong IP address is being populated on an ingress

[snoby]

Using version v0.6.1 mate choses the public ip address of the node instead of the ingress controller alias to populate the route53 address. I'm using a simple example here:
kops ingress-nginx
and I've changed the ingress yaml in the above example to point to route53 zone instead of foo:
here is the original I changed it to foo.c.tropo.com and bar.c.tropo.com

Here is a snippet from my route53 console

bar.c.tropo.com.  A 54.147.161.188 - - 300
bar.c.tropo.com.  TXT "mate:mate-managed" - - 300

default-ingress-nginx.c.tropo.com.  A ALIAS ac2f3241cffbc11e6bab612aafb8d2fa-1674343966.us-east-1.elb.amazonaws.com. (z35sxdotrq7x7k) Yes -
default-ingress-nginx.c.tropo.com.  TXT "mate:mate-managed" - - 300

foo.c.tropo.com.  A 54.147.161.188 - - 300
foo.c.tropo.com.  TXT "mate:mate-managed" - - 300

Here is the snippet from the mate container:

time="2017-03-03T03:10:20Z" level=info msg="ADDED: default/echomap"
time="2017-03-03T03:10:20Z" level=warning msg="[Ingress] The load balancer of ingress 'default/echomap' does not have any ingress."
time="2017-03-03T03:10:20Z" level=info msg="MODIFIED: default/echomap"
time="2017-03-03T03:10:20Z" level=info msg="[AWS] Processing (foo.c.tropo.com., 54.147.161.188, )\n"
time="2017-03-03T03:10:20Z" level=info msg="[AWS] Processing (bar.c.tropo.com., 54.147.161.188, )\n"
time="2017-03-03T03:10:44Z" level=info msg="[Synchronize] Synchronizing DNS entries..."

I setup the mate deployment to look like below, only change I put in was I put the annotation to use a specific policy role that I created:

/work/Official/kops-installs/mate$ cat mate-deployment.yml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: mate
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: mate
      annotations:
        iam.amazonaws.com/role: mate-route53-role
    spec:
      containers:
      - name: mate
        image: registry.opensource.zalan.do/teapot/mate:v0.6.1
        env:
        - name: AWS_REGION
          value: us-east-1
        args:
        - --producer=kubernetes
        - --kubernetes-format={{.Namespace}}-{{.Name}}.c.tropo.com
        - --consumer=aws
        - --aws-record-group-id=mate-managed

Here are my two mate policies basically i'm trying to limit mate to just two zoneid's

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1488504089000",
            "Effect": "Allow",
            "Action": [
                "route53:ChangeResourceRecordSets"
            ],
            "Resource": [
                "arn:aws:route53::::hostedzone/ZVO7JZ181S8PP"
            ]
        },
        {
            "Sid": "Stmt1488504193000",
            "Effect": "Allow",
            "Action": [
                "route53:ChangeResourceRecordSets"
            ],
            "Resource": [
                "arn:aws:route53::::hostedzone/Z1N71K8UHPS27L"
            ]
        },
        {
            "Sid": "Stmt1488504240000",
            "Effect": "Allow",
            "Action": [
                "route53:GetChange"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}

and the elb policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1488504551000",
            "Effect": "Allow",
            "Action": [
                "elasticloadbalancing:*"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}

[ideahitme]

could you please do

kubectl get -o yaml ingress echomap

[snoby]

Looks strange...

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: '{"kind":"Ingress","apiVersion":"extensions/v1beta1","metadata":{"name":"echomap","creationTimestamp":null},"spec":{"rules":[{"host":"foo.c.tropo.com","http":{"paths":[{"path":"/foo","backend":{"serviceName":"echoheaders-x","servicePort":80}}]}},{"host":"bar.c.tropo.com","http":{"paths":[{"path":"/bar","backend":{"serviceName":"echoheaders-y","servicePort":80}},{"path":"/foo","backend":{"serviceName":"echoheaders-x","servicePort":80}}]}}]},"status":{"loadBalancer":{}}}'
  creationTimestamp: 2017-03-03T03:10:20Z
  generation: 1
  name: echomap
  namespace: default
  resourceVersion: "23478101"
  selfLink: /apis/extensions/v1beta1/namespaces/default/ingresses/echomap
  uid: ef2ca658-ffbe-11e6-bab6-12aafb8d2fa6
spec:
  rules:
  - host: foo.c.tropo.com
    http:
      paths:
      - backend:
          serviceName: echoheaders-x
          servicePort: 80
        path: /foo
  - host: bar.c.tropo.com
    http:
      paths:
      - backend:
          serviceName: echoheaders-y
          servicePort: 80
        path: /bar
      - backend:
          serviceName: echoheaders-x
          servicePort: 80
        path: /foo
status:
  loadBalancer:
    ingress:
    - ip: 54.147.161.188

[ideahitme]

@snoby we already discussed this issue previously here: #85

As mentioned in the issue, Mate does its job integrating with ingress-controllers by reading whatever loadBalancervalue is provided by ingress resource, and that's where Mate's responsibility ends. You can also find some more useful information there. Feel free to discuss in the upstream issue, closing this one

[linki]

@snoby mate doesn't modify the Ingress object, so it's your ingress controller putting the "wrong" IP there. Are you using the nginx-ingress-controller?

[snoby]

Yes I am. I guess I need to find a way to make the nginx-ingress-controller put the right dns alias in that field. Honestly I suspect 80% of the people using k8s are going to be using nginx-ingress controller.

[linki]

@snoby @ideahitme I did some digging and have good news.

https://github.com/kubernetes/ingress/tree/master/examples/static-ip/nginx#acquiring-an-ip perfectly describes our issue. They mention to put a Type=LoadBalancer in front of the nginx controller to get an ELB for it. In this case one should pass the --publish-service flag to nginx-controller so that:

--publish-service string

Service fronting the ingress controllers. Takes the form
namespace/name. The controller will set the endpoint records on the
ingress objects to reflect those on the service.

This flag only exists in the non-released 0.9 version of nginx-controller. https://github.com/kubernetes/ingress/tree/master/examples/static-ip/nginx has the corresponding manifest files.

However, I also found https://github.com/kubernetes/kops/tree/master/addons/ingress-nginx#creating-a-simple-ingress where the Ingress clearly receives the DNS name of the exposed ELB. Though, looking at the manifests they seem to use 0.8.3 and also put a service in front of the controller. So, I guess it also somehow works with this version. Maybe it discovers the service by a hardcoded name and in 0.9 the name is just configurable?

@snoby You want to give 0.9 pointing it to your service a try?

[snoby]

@linki
I did some playing around with this today and discovered a couple of things:

1. in the 0.9.0-beta.2 release of the nginx ingress controller they changed the argument to the binary from --nginx-configmaps to just --configmaps.
   2.) I added to my service annotation :
   service.beta.kubernetes.io/external-traffic: OnlyLocal

Everything looks correct and MATE correctly sets the ip address to the ELB alias now!, unfortunately this is definitely a beta because the health check in the ELB suggests that none of my nodes were in service.
All that being said @linki 's suggestion was correct. Here is my updated ingress:

kind: Service
apiVersion: v1
metadata:
  name: nginx-default-backend
  namespace: default
  labels:
    k8s-addon: ingress-nginx.addons.k8s.io
spec:
  ports:
  - port: 80
    targetPort: http
  selector:
    app: nginx-default-backend

---

kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: nginx-default-backend
  labels:
    k8s-addon: ingress-nginx.addons.k8s.io
spec:
  replicas: 1
  template:
    metadata:
      labels:
        k8s-addon: ingress-nginx.addons.k8s.io
        app: nginx-default-backend
    spec:
      terminationGracePeriodSeconds: 60
      containers:
      - name: default-http-backend
        image: gcr.io/google_containers/defaultbackend:1.0
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 30
          timeoutSeconds: 5
        resources:
          limits:
            cpu: 10m
            memory: 20Mi
          requests:
            cpu: 10m
            memory: 20Mi
        ports:
        - name: http
          containerPort: 8080
          protocol: TCP

---

kind: ConfigMap
apiVersion: v1
metadata:
  name: ingress-nginx
  namespace: default
  labels:
    k8s-addon: ingress-nginx.addons.k8s.io
data:
  use-proxy-protocol: 'true'
  enable-sticky-sessions: 'true'
  enale-vts-status: 'true'

---
#
# must include external-traffic: OnlyLocal to preserve the source ip address and so that
# the load balancer Ip address will be set instead of the node ip address.
#
kind: Service
apiVersion: v1
metadata:
  name: ingress-nginx
  labels:
    k8s-addon: ingress-nginx.addons.k8s.io
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: '*'
    service.beta.kubernetes.io/external-traffic: OnlyLocal
spec:
  type: LoadBalancer
  selector:
    app: ingress-nginx
  ports:
  - name: http
    port: 80
    targetPort: http
  - name: https
    port: 443
    targetPort: https

[snoby]

Follow up to the previous post. I have it all working now,
DO NOT USE the OnlyLocal annotation. So take out the above line:
service.beta.kubernetes.io/external-traffic: OnlyLocal

[ideahitme]

@snoby it would be awesome to have this use-case documented in the mate readme files. If you wish to do so, please let us know and kindly create a PR. Otherwise, me or @linki will do that 👍

[snoby]

I will put something together and submit a PR,
Thanks for your help!