Traefik stopped serving on upgrade to v1.1.0-rc3

[ShakataGaNai]

I was getting tired of all the LetsEncrypt renewal reminder emails and I saw the issues had been resolved in the latest, so I figured I'd give the latest RC a try (via DockerHub). I shutdown & removed the old traefik, pulled & launched the latest version and it didn't serve any traffic. The healthpage worked and I think HTTP traffic worked, but TLS based traffic did not (which is 100% of what I use).

Here's the logs from starting the container. The truncated section in the middle is where its reading out all the docker configs.

time="2016-11-06T20:47:20Z" level=info msg="Traefik version v1.1.0-rc3 built on 2016-10-26_04:12:45PM"
time="2016-11-06T20:47:20Z" level=info msg="Using TOML configuration file /etc/traefik/traefik.toml"
time="2016-11-06T20:47:20Z" level=debug msg="Global configuration loaded {\"GraceTimeOut\":10,\"Debug\":false,\"AccessLogsFile\":\"/dev/stdout\",\"TraefikLogsFile\":\"/dev/stdout\",\"LogLevel\":\"DEBUG\",\"EntryPoints\":{\"http\":{\"Network\":\"\",\"Address\":\":80\",\"TLS\":null,\"Redirect\":{\"EntryPoint\":\"https\",\"Regex\":\"\",\"Replacement\":\"\"},\"Auth\":null,\"Compress\":false},\"https\":{\"Network\":\"\",\"Address\":\":443\",\"TLS\":{\"MinVersion\":\"\",\"CipherSuites\":null,\"Certificates\":null,\"ClientCAFiles\":null},\"Redirect\":null,\"Auth\":null,\"Compress\":false}},\"Cluster\":null,\"Constraints\":[],\"ACME\":{\"Email\":\"fakeaddress@fakedomain.tld\",\"Domains\":[{\"Main\":\"domain.host.tld\",\"SANs\":null}],\"Storage\":\"\",\"StorageFile\":\"/etc/traefik/acme/acme.json\",\"OnDemand\":true,\"OnHostRule\":false,\"CAServer\":\"\",\"EntryPoint\":\"https\"},\"DefaultEntryPoints\":[\"http\",\"https\"],\"ProvidersThrottleDuration\":2000000000,\"MaxIdleConnsPerHost\":200,\"InsecureSkipVerify\":false,\"Retry\":null,\"Docker\":{\"Watch\":true,\"Filename\":\"\",\"Constraints\":null,\"Endpoint\":\"unix:///var/run/docker.sock\",\"Domain\":\"docker.localhost\",\"TLS\":null,\"ExposedByDefault\":true,\"UseBindPortIP\":false,\"SwarmMode\":false},\"File\":null,\"Web\":{\"Address\":\":8080\",\"CertFile\":\"\",\"KeyFile\":\"\",\"ReadOnly\":false,\"Auth\":null},\"Marathon\":null,\"Consul\":null,\"ConsulCatalog\":null,\"Etcd\":null,\"Zookeeper\":null,\"Boltdb\":null,\"Kubernetes\":null,\"Mesos\":null}"
time="2016-11-06T20:47:20Z" level=info msg="Preparing server http &{Network: Address::80 TLS:<nil> Redirect:0xc4203bae10 Auth:<nil> Compress:false}"
time="2016-11-06T20:47:20Z" level=info msg="Preparing server https &{Network: Address::443 TLS:0xc4202eaea0 Redirect:<nil> Auth:<nil> Compress:false}"
time="2016-11-06T20:47:20Z" level=info msg="Starting server on :80"
time="2016-11-06T20:47:20Z" level=warning msg="ACME.StorageFile is deprecated, use ACME.Storage instead"
time="2016-11-06T20:47:20Z" level=info msg="Loading ACME Account..."
time="2016-11-06T20:47:20Z" level=info msg="Loaded ACME config from store /etc/traefik/acme/acme.json"
time="2016-11-06T20:47:20Z" level=info msg=buildACMEClient...
time="2016-11-06T20:47:20Z" level=debug msg="Building ACME client..."
time="2016-11-06T20:47:20Z" level=info msg=AgreeToTOS...
time="2016-11-06T20:47:20Z" level=info msg="Retrieving ACME certificates..."
time="2016-11-06T20:47:20Z" level=info msg="Retrieved ACME certificates"
time="2016-11-06T20:47:20Z" level=debug msg="Testing certificate renew..."
time="2016-11-06T20:47:20Z" level=info msg="Starting server on :443"
time="2016-11-06T20:47:20Z" level=info msg="Starting provider *provider.Docker {\"Watch\":true,\"Filename\":\"\",\"Constraints\":null,\"Endpoint\":\"unix:///var/run/docker.sock\",\"Domain\":\"docker.localhost\",\"TLS\":null,\"ExposedByDefault\":true,\"UseBindPortIP\":false,\"SwarmMode\":false}"
time="2016-11-06T20:47:20Z" level=info msg="Starting provider *main.WebProvider {\"Address\":\":8080\",\"CertFile\":\"\",\"KeyFile\":\"\",\"ReadOnly\":false,\"Auth\":null}"
time="2016-11-06T20:47:20Z" level=debug msg="Renewing certificate {Main:domain.host.tld SANs:[]}"
time="2016-11-06T20:47:20Z" level=debug msg="Docker connection established with docker 1.12.1 (API 1.24)"
time="2016-11-06T20:47:20Z" level=debug msg="Filtering container without port and no traefik.port label /traefik"
time="2016-11-06T20:47:20Z" level=debug msg="Filtering disabled container /xxxxxxxxxxxxxxxx_wp_1"
time="2016-11-06T20:47:20Z" level=debug msg="Filtering disabled container /xxxxxxxxxxxxxxxx_db_1"
time="2016-11-06T20:47:20Z" level=debug msg="Filtering disabled container /yyyyyyyyy_wp_1"
time="2016-11-06T20:47:20Z" level=debug msg="Filtering disabled container /yyyyyyyyy_db_1"
time="2016-11-06T20:47:20Z" level=debug msg="Filtering container without port and no traefik.port label /watchtower"
time="2016-11-06T20:47:20Z" level=debug msg="Filtering container without port and no traefik.port label /zzzzzzzzzzzzzzzz_gitlab-runner_1"
time="2016-11-06T20:47:20Z" level=debug msg="Filtering container with more than 1 port and no traefik.port label /dd-agent"
time="2016-11-06T20:47:20Z" level=debug msg="Filtering container without port and no traefik.port label /cleanup-agent"
time="2016-11-06T20:47:20Z" level=debug msg="Load balancer method '<nil>' for backend backend-qqqqqqqqqqqq: Invalid method, using default. Using default wrr."

.........

time="2016-11-06T20:47:20Z" level=debug msg="Creating backend backend-xxxxxxxxxxxxxxxx"
time="2016-11-06T20:47:20Z" level=debug msg="Creating load-balancer wrr"
time="2016-11-06T20:47:20Z" level=debug msg="Creating server server-xxxxxxxxxxxxxxxx_ngx_1 at http://172.19.0.4:80 with weight 1"
time="2016-11-06T20:47:20Z" level=info msg="Server configuration reloaded on :80"
time="2016-11-06T20:47:20Z" level=info msg="Server configuration reloaded on :443"
time="2016-11-06T20:47:21Z" level=debug msg="Challenge Present domain.host.tld"
2604:xxxx:xxxx:5b99 - - [06/Nov/2016:20:47:52 +0000] "GET / HTTP/1.1" 302 5 "" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTM

Also attached is my traefik.toml

As I was testing in production, I rolled back to latest release (1.0.3) and the system ran fine again.

[emilevauge]

@ShakataGaNai When you say

TLS based traffic did not

Can you give more details? Anything in the DEBUG logs?

[ShakataGaNai]

No pages loaded for any site over HTTPS, nor did it log any hits. The above log was everything with the one IPv6 hit at the bottom of the log was an attempt over HTTP (which is 302'd over to HTTPS per the traefik.toml config).

I could roll forward again and try something else, if you have a suggestion for getting better diagnostics out of this.

[emilevauge]

@ShakataGaNai Could you provide all your DEBUG logs (I don't see any frontend created in provided logs)?

[osixia]

I experencied the same thing on CoreOs + Kubernetes. I will try to give you some logs later on.

[deepthawtz]

Just tried v1.1.0-rc3 and proxying stopped working for me too (Marathon). Could reach traefik UX and frontend/backend entries I was expecting to see were all listed but could not reach any of them. Rolling back to v1.1.0-rc1 restored functionality.

[emilevauge]

I found the issue. Fix is coming.

[emilevauge]

Fixed by #814. Could you test using Docker image containous/traefik:acme ?

[osixia]

Hello,
i just test containous/traefik:acme this is still not working for me, no https response but this show up in the logs:

  time="2016-11-08T09:17:17Z" level=error msg="Error renewing certificate: acme: Error 429 - urn:acme:error:rateLimited - Error creating new cert :: Too many certificates already issued for exact set of domains: xxxx.com"

The certificate is still valid but must be renewed i guess. It would be better if traefik respond to https request even with an expired certificate or if the renewal failed :)

[emilevauge]

@osixia thanks for testing.
Traefik should return the expired cert anyway... Could you send me in PM on Slack all your DEBUG logs and your acme.json file?

[osixia]

Sorry it seems ok, litle caching pb 😳
Thanks

[SantoDE]

Hey @osixia,

great that you're issue is solved. Therefore, I gonna close that issue for now :)

[thomasf]

I just got the same errors when trying to upgrade from 1.0.3 to 1.1.0.. Lot's of Error renewing certificate: acme: Error 429 - urn:acme:error:rateLimited - Error creating new cert :: Too many certificates already issued for exact set of domains: ... acme errors in the log and the providers list was empty in the traefik web ui. Going back to 1.0.3 made it work again.

[emilevauge]

@thomasf your issue(s) seems not linked with this closed issue. Can you open a new one providing more info?

[thomasf]

ok