Traefik with Kubernetes backend - keep getting 401 on all GET requests to kube-apiserver

[spk83]

What version of Traefik are you using (traefik version)?

v1.1.2

What is your environment & configuration (arguments, toml...)?

# Deployment Definition
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: traefik-ingress-controller
  namespace: kube-system
  labels:
    app: traefik-ingress-lb
    kubernetes.io/cluster-service: "true"
spec:
  replicas: 1
  selector:
    matchLabels:
      app: traefik-ingress-lb
  template:
    metadata:
      labels:
        app: traefik-ingress-lb
    spec:
      terminationGracePeriodSeconds: 60
      hostNetwork: true
      containers:
      - image: traefik:v1.1.2
        name: traefik-ingress-lb
        resources:
          limits:
            cpu: 200m
            memory: 30Mi
          requests:
            cpu: 100m
            memory: 20Mi
        ports:
        - name: http
          containerPort: 80
        - name: admin
          containerPort: 8081
        args:
        - --web
        - --web.address=:8081
        - --kubernetes
        - --logLevel=DEBUG
---
# Service Definition for UI
apiVersion: v1
kind: Service
metadata:
  name: traefik-web-ui
  namespace: kube-system
  labels:
    app: traefik-ingress-lb
    kubernetes.io/cluster-service: "true"
spec:
  selector:
    app: traefik-ingress-lb
  ports:
  - name: web
    port: 80
    targetPort: 8081
---
# Ingress Definition for UI
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: traefik-web-ui
  namespace: kube-system
  labels:
    app: traefik-web-ui
    kubernetes.io/cluster-service: "true"
spec:
  rules:
  - host: traefik-ui.xyz.com
    http:
      paths:
      - path: /
        backend:
          serviceName: traefik-web-ui
          servicePort: web

What did you do?

Tried deploying traefik on k8s cluster as ingress controller with default service account.

What did you expect to see?

traefik should be able to listen to kube-apiserver for ingresses and changes

What did you see instead?

traefik throwing 401 in logs on all GET requests to kube-apiserver. I can see traefik loading default service account token and ca certs in kube-system namespace and verified that they are correct.

Token value and domain names are changed in log below.

2017-02-16T13:49:07.489821005Z time="2017-02-16T13:49:07Z" level=info msg="Traefik version v1.1.2 built on 2016-12-15_10:21:15AM" 
2017-02-16T13:49:07.489912561Z time="2017-02-16T13:49:07Z" level=debug msg="Global configuration loaded {\"GraceTimeOut\":10,\"Debug\":false,\"CheckNewVersion\":true,\"AccessLogsFile\":\"\",\"TraefikLogsFile\":\"\",\"LogLevel\":\"DEBUG\",\"EntryPoints\":{\"http\":{\"Network\":\"\",\"Address\":\":80\",\"TLS\":null,\"Redirect\":null,\"Auth\":null,\"Compress\":false}},\"Cluster\":null,\"Constraints\":[],\"ACME\":null,\"DefaultEntryPoints\":[\"http\"],\"ProvidersThrottleDuration\":2000000000,\"MaxIdleConnsPerHost\":200,\"InsecureSkipVerify\":false,\"Retry\":null,\"Docker\":null,\"File\":null,\"Web\":{\"Address\":\":8081\",\"CertFile\":\"\",\"KeyFile\":\"\",\"ReadOnly\":false,\"Auth\":null},\"Marathon\":null,\"Consul\":null,\"ConsulCatalog\":null,\"Etcd\":null,\"Zookeeper\":null,\"Boltdb\":null,\"Kubernetes\":{\"Watch\":true,\"Filename\":\"\",\"Constraints\":[],\"Endpoint\":\"\",\"DisablePassHostHeaders\":false,\"Namespaces\":null,\"LabelSelector\":\"\"},\"Mesos\":null}" 
2017-02-16T13:49:07.489944656Z time="2017-02-16T13:49:07Z" level=info msg="Preparing server http &{Network: Address::80 TLS:<nil> Redirect:<nil> Auth:<nil> Compress:false}" 
2017-02-16T13:49:07.489957691Z time="2017-02-16T13:49:07Z" level=info msg="Starting provider *main.WebProvider {\"Address\":\":8081\",\"CertFile\":\"\",\"KeyFile\":\"\",\"ReadOnly\":false,\"Auth\":null}" 
2017-02-16T13:49:07.489970373Z time="2017-02-16T13:49:07Z" level=info msg="Starting provider *provider.Kubernetes {\"Watch\":true,\"Filename\":\"\",\"Constraints\":[],\"Endpoint\":\"\",\"DisablePassHostHeaders\":false,\"Namespaces\":null,\"LabelSelector\":\"\"}" 
2017-02-16T13:49:07.490236651Z time="2017-02-16T13:49:07Z" level=info msg="Starting server on :80" 
2017-02-16T13:49:07.490796634Z time="2017-02-16T13:49:07Z" level=debug msg="Kubernetes token: Lm5hbWUiOiJ0cmFlZmlrLXNhLXRva2VuLXNuOThjIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6InRyYWVmaWstc2EiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiJhZTM0NDkxNC1mNDRlLTExZTYtYTVjNC0wMmIxYWRlMTIwMDgiRUIaBE4BdY6T1Pfx2m2XukmQEafiW5tKPAmCxrHdxAygJmQl7i4_z6AGdOBuHp77o" 
2017-02-16T13:49:07.490815615Z time="2017-02-16T13:49:07Z" level=debug msg="Kubernetes CA cert: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt" 
2017-02-16T13:49:07.490822814Z time="2017-02-16T13:49:07Z" level=debug msg="Using environment provided kubernetes endpoint" 
2017-02-16T13:49:07.490828015Z time="2017-02-16T13:49:07Z" level=debug msg="Kubernetes endpoint: https://10.254.0.1:443" 
2017-02-16T13:49:07.490837173Z time="2017-02-16T13:49:07Z" level=debug msg="Using label selector: ''" 
2017-02-16T13:49:07.800588683Z time="2017-02-16T13:49:07Z" level=debug msg="Configuration received from provider kubernetes: {\"frontends\":{\"traefik-ui.xyz.com/\":{\"entryPoints\":[\"http\"],\"backend\":\"traefik-ui.xyz.com/\",\"routes\":{\"/\":{\"rule\":\"PathPrefix:/\"},\"traefik-ui.xyz.com\":{\"rule\":\"Host:traefik-ui.xyz.com\"}},\"passHostHeader\":true,\"priority\":1}}}" 
2017-02-16T13:49:07.800680976Z time="2017-02-16T13:49:07Z" level=debug msg="Last kubernetes config received more than 2s, OK" 
2017-02-16T13:49:07.800691996Z time="2017-02-16T13:49:07Z" level=debug msg="Creating frontend traefik-ui.xyz.com/" 
2017-02-16T13:49:07.800701222Z time="2017-02-16T13:49:07Z" level=debug msg="Wiring frontend traefik-ui.xyz.com/ to entryPoint http" 
2017-02-16T13:49:07.800710075Z time="2017-02-16T13:49:07Z" level=debug msg="Creating route / PathPrefix:/" 
2017-02-16T13:49:07.800719215Z time="2017-02-16T13:49:07Z" level=debug msg="Creating route traefik-ui.xyz.com Host:traefik-ui.xyz.com" 
2017-02-16T13:49:07.800727727Z time="2017-02-16T13:49:07Z" level=debug msg="Creating backend traefik-ui.xyz.com/" 
2017-02-16T13:49:07.800736889Z time="2017-02-16T13:49:07Z" level=error msg="Undefined backend 'traefik-ui.xyz.com/' for frontend traefik-ui.xyz.com/" 
2017-02-16T13:49:07.800746610Z time="2017-02-16T13:49:07Z" level=error msg="Skipping frontend traefik-ui.xyz.com/..." 
2017-02-16T13:49:07.800756330Z time="2017-02-16T13:49:07Z" level=info msg="Server configuration reloaded on :80" 
2017-02-16T13:49:08.054695760Z time="2017-02-16T13:49:08Z" level=debug msg="Received event from kubernetes map[type:MODIFIED object:map[kind:Endpoints apiVersion:v1 metadata:map[name:kube-controller-manager namespace:kube-system selfLink:/api/v1/namespaces/kube-system/endpoints/kube-controller-manager uid:ecbb4d95-f3b0-11e6-a5c4-02b1ade12008 resourceVersion:137659 creationTimestamp:2017-02-15T18:59:49Z annotations:map[control-plane.alpha.kubernetes.io/leader:{\"holderIdentity\":\"kube-b-1.xyz.com\",\"leaseDurationSeconds\":15,\"acquireTime\":\"2017-02-15T18:59:49Z\",\"renewTime\":\"2017-02-16T13:49:08Z\",\"leaderTransitions\":0}]] subsets:[]]]" 
2017-02-16T13:49:08.092750793Z time="2017-02-16T13:49:08Z" level=debug msg="Skipping event from kubernetes map[object:map[metadata:map[uid:ecbb4d95-f3b0-11e6-a5c4-02b1ade12008 resourceVersion:137659 creationTimestamp:2017-02-15T18:59:49Z annotations:map[control-plane.alpha.kubernetes.io/leader:{\"holderIdentity\":\"kube-b-1.xyz.com\",\"leaseDurationSeconds\":15,\"acquireTime\":\"2017-02-15T18:59:49Z\",\"renewTime\":\"2017-02-16T13:49:08Z\",\"leaderTransitions\":0}] name:kube-controller-manager namespace:kube-system selfLink:/api/v1/namespaces/kube-system/endpoints/kube-controller-manager] subsets:[] kind:Endpoints apiVersion:v1] type:MODIFIED]" 
2017-02-16T13:49:08.414440290Z time="2017-02-16T13:49:08Z" level=debug msg="Received event from kubernetes map[object:map[kind:Endpoints apiVersion:v1 metadata:map[creationTimestamp:2017-02-16T13:49:04Z labels:map[app:traefik-ingress-lb kubernetes.io/cluster-service:true] name:traefik-web-ui namespace:kube-system selfLink:/api/v1/namespaces/kube-system/endpoints/traefik-web-ui uid:ae420b89-f44e-11e6-a5c4-02b1ade12008 resourceVersion:137662] subsets:[map[addresses:[map[ip:45.12.79.88 nodeName:kube-b-3.xyz.com targetRef:map[namespace:kube-system name:traefik-ingress-controller-542373120-h0x5b uid:ae4d9527-f44e-11e6-a5c4-02b1ade12008 resourceVersion:137660 kind:Pod]]] ports:[map[protocol:TCP name:web port:8081]]]]] type:MODIFIED]"  
2017-02-16T13:49:08.514713468Z time="2017-02-16T13:49:08Z" level=debug msg="Load balancer method '<nil>' for backend traefik-ui.xyz.com/: Invalid method, using default. Using default wrr." 
2017-02-16T13:49:08.514756390Z time="2017-02-16T13:49:08Z" level=debug msg="Configuration received from provider kubernetes: {\"backends\":{\"traefik-ui.xyz.com/\":{\"servers\":{\"traefik-ingress-controller-542373120-h0x5b\":{\"url\":\"http://45.12.79.88:8081\",\"weight\":0}},\"loadBalancer\":{\"method\":\"wrr\"}}},\"frontends\":{\"traefik-ui.xyz.com/\":{\"entryPoints\":[\"http\"],\"backend\":\"traefik-ui.xyz.com/\",\"routes\":{\"/\":{\"rule\":\"PathPrefix:/\"},\"traefik-ui.xyz.com\":{\"rule\":\"Host:traefik-ui.xyz.com\"}},\"passHostHeader\":true,\"priority\":1}}}" 
2017-02-16T13:49:08.514769344Z time="2017-02-16T13:49:08Z" level=debug msg="Last kubernetes config received less than 2s, waiting..." 
2017-02-16T13:49:09.104454313Z time="2017-02-16T13:49:09Z" level=debug msg="Received event from kubernetes map[type:MODIFIED object:map[kind:Endpoints apiVersion:v1 metadata:map[name:kubernetes namespace:default selfLink:/api/v1/namespaces/default/endpoints/kubernetes uid:1d1609ad-f3b0-11e6-844d-02b1ade12008 resourceVersion:137664 creationTimestamp:2017-02-15T18:54:00Z] subsets:[map[addresses:[map[ip:45.12.79.88]] ports:[map[name:https port:6443 protocol:TCP]]]]]]" 
2017-02-16T13:49:09.202312276Z time="2017-02-16T13:49:09Z" level=warning msg="Error retrieving services: failed to create services request: GET \"https://10.254.0.1:443/api/v1/namespaces/kube-system/services/traefik-web-ui\" : http error 401 GET \"https://10.254.0.1:443/api/v1/namespaces/kube-system/services/traefik-web-ui\": \"Unauthorized\\n\"" 

[spk83]

Issue was with kubernetes authentication setup. Closing this issue. Thanks.