Error when using HA acme in kubernetes with etcd

[pdressel]

I am running traefik v1.1.0-rc1 in kubernetes, and I get the following error when I try to run it with etcd as KV store and HA acme:

kubectl logs traefik-2153114016-656zf
level=info msg="Traefik version v1.1.0-rc1 built on 2016-09-30_03:01:06PM" 
level=info msg="Using TOML configuration file /etc/traefik/config.toml" 
level=info msg="Preparing server https &{Network: Address::443 TLS:0xc4201b1ec0 Redirect:<nil> Auth:<nil> Compress:false}" 
level=error msg="Error creating TLS config Unknown entrypoint  for ACME configuration" 
level=fatal msg="Error preparing server: Unknown entrypoint  for ACME configuration" 

I use the following TOML:

logLevel = "INFO"
defaultEntryPoints = ["http", "https"]

[etcd]
endpoint = "etcd-0.etcd:2379,etcd-1.etcd:2379,etcd-2.etcd:2379"

[acme]
email = "some valid email"
entryPoint = "https"
OnHostRule = true
storage = "traefik/acme/account"

[entryPoints]
[entryPoints.http]
address = ":80"
[entryPoints.http.redirect]
entryPoint = "https"
[entryPoints.https]
address = ":443"
[entryPoints.https.tls]

[web]
address = ":8080"

[kubernetes]
labelselector = "authenticationProvider in (public-access)"

I checked that the etcd PetSet is working, and traefik does store data in it under traefik/acme/account.

Any advice on how to debug this?

[emilevauge]

Can you post your DEBUG logs?

[pdressel]

Sure, this is the DEBUG log:

level=info msg="Traefik version v1.1.0-rc1 built on 2016-09-30_03:01:06PM" 
level=info msg="Using TOML configuration file /etc/traefik/config.toml" 
level=debug msg="Global configuration loaded {\"GraceTimeOut\":10,\"Debug\":false,\"AccessLogsFile\":\"\",\"TraefikLogsFile\":\"\",\"LogLevel\":\"DEBUG\",\"EntryPoints\":{\"http\":{\"Network\":\"\",\"Address\":\":80\",\"TLS\":null,\"Redirect\":{\"EntryPoint\":\"https\",\"Regex\":\"\",\"Replacement\":\"\"},\"Auth\":null,\"Compress\":false},\"https\":{\"Network\":\"\",\"Address\":\":443\",\"TLS\":{\"MinVersion\":\"\",\"CipherSuites\":null,\"Certificates\":null,\"ClientCAFiles\":null},\"Redirect\":null,\"Auth\":null,\"Compress\":false}},\"Cluster\":{\"Node\":\"97dae5aa-e9ce-42db-86cb-5519d284df1d\",\"Store\":{\"Store\":{},\"Prefix\":\"/traefik\"}},\"Constraints\":[],\"ACME\":{\"Email\":\"\",\"Domains\":null,\"Storage\":\"\",\"StorageFile\":\"\",\"OnDemand\":false,\"OnHostRule\":false,\"CAServer\":\"\",\"EntryPoint\":\"\"},\"DefaultEntryPoints\":[\"http\",\"https\"],\"ProvidersThrottleDuration\":2000000000,\"MaxIdleConnsPerHost\":200,\"InsecureSkipVerify\":false,\"Retry\":null,\"Docker\":null,\"File\":null,\"Web\":{\"Address\":\":8080\",\"CertFile\":\"\",\"KeyFile\":\"\",\"ReadOnly\":false,\"Auth\":null},\"Marathon\":null,\"Consul\":null,\"ConsulCatalog\":null,\"Etcd\":{\"Watch\":true,\"Filename\":\"\",\"Constraints\":[],\"Endpoint\":\"etcd-0.etcd:2379,etcd-1.etcd:2379,etcd-2.etcd:2379\",\"Prefix\":\"/traefik\",\"TLS\":null},\"Zookeeper\":null,\"Boltdb\":null,\"Kubernetes\":{\"Watch\":true,\"Filename\":\"\",\"Constraints\":[],\"Endpoint\":\"\",\"DisablePassHostHeaders\":false,\"Namespaces\":null,\"LabelSelector\":\"authenticationProvider in (public-access)\"},\"Mesos\":null}" 
level=info msg="Preparing server https &{Network: Address::443 TLS:0xc4202d9d40 Redirect:<nil> Auth:<nil> Compress:false}" 
level=error msg="Error creating TLS config Unknown entrypoint  for ACME configuration" 
level=fatal msg="Error preparing server: Unknown entrypoint  for ACME configuration" 

This is the JSON stored in etcd under /traefik/acme/account/object (decoded base64):

{
  "Email": "<<valid email>>",
  "Registration": {
    "body": {
      "resource": "reg",
      "id": <<id>>,
      "key": {
        "kty": "RSA",
        "n": "<<valid looking key>>",
        "e": "AQAB"
      },
      "contact": [
        "mailto:<<valid email>>"
      ],
      "agreement": "https://letsencrypt.org/documents/LE-SA-v1.1.1-August-1-2016.pdf"
    },
    "uri": "https://acme-v01.api.letsencrypt.org/acme/reg/<<id>>",
    "new_authzr_uri": "https://acme-v01.api.letsencrypt.org/acme/new-authz",
    "terms_of_service": "https://letsencrypt.org/documents/LE-SA-v1.1.1-August-1-2016.pdf"
  },
  "PrivateKey": "<<valid looking private key>>",
  "DomainsCertificate": {
    "Certs": []
  },
  "ChallengeCerts": {}
}

While testing i found something else: when there are no values stored in etcd under /traefik, the first instance of traefik behaves differently (log is longer so I attached it).
log.txt

[emilevauge]

@pdressel did you use traefik storeconfig to populate etcd with traefik configuration http://docs.traefik.io/user-guide/cluster?

[pdressel]

@emilevauge I used a kubernetes ConfigMap to mount the toml file into every container. Is the storeconfig required?

[emilevauge]

Yes, as you told traefik to use etcd to get its configuration, it will override the toml config using etcd config. As it's probably empty, you get this error. You should use storeconfig once, and then remove the toml :)

[pdressel]

I added a kubernetes job to run the storeconfig command on initial deployment, that fixed it. Thanks! 🎉

[Smana]

@pdressel could you please describe in more detail how you proceeded ?

[pdressel]

@Smana sure, this is the kubernetes job I used to populate the KV store (which must be running first, of course)

apiVersion: batch/v1
kind: Job
metadata:
  name: traefik-config
spec:
  template:
    metadata:
      name: traefik-config
    spec:
      containers:
      - name: traefik-config
        image: {{ image: traefik }}
        args:
        - storeconfig
        - --configfile=/etc/traefik/traefik.toml
        volumeMounts:
        - name: traefik-config
          mountPath: /etc/traefik
      restartPolicy: Never
      volumes:
      - name: traefik-config
        configMap:
          name: traefik-config

In conjunction with a configmap like so:

metadata:
  name: traefik-config
data:
  traefik.toml: |-
    logLevel = "DEBUG"
    [etcd]
    endpoint = "etcd-0.etcd:2379,etcd-1.etcd:2379,etcd-2.etcd:2379"
...

After this, you can run traefik as a regular deployment and point it to your KV store. In this case, I used a StatefulSet of three etcd servers as the KV store.

I never moved this into production though, since at the time HA acme was in a very early release candidate and I needed a stable solution for production.

[Smana]

@pdressel Thank you for these details.
Actually i'm trying to use InitContainers and etcd operator in order to run the K/V cluster.
Currently i just need to fix an issue with the acme config, i got the following error when running the storeconfig

2017/01/09 08:19:36 Error running traefik: 102: Not a file (/traefik/acme)

But anyway this is not related to this issue :)

[glennschmidt]

Hi @emilevauge , in the docs here https://docs.traefik.io/basics/#static-trfk-configuration it says

Træfɪk can be configured using many configuration sources with the following precedence order.

• Key-value Store
• Arguments
• Configuration file
• Default

This makes it sound like you can use more than one config source at the same time, and Traefik will search for properties in all of them, in the sequence given, stopping when it finds a value. Is that actually incorrect?