Error 503 when accessing dapps

[asemiankevich]

Problem

error 503 when accessing simpledapp.eth or ens dapp

Failed to load resource: the server responded with a status of 503 (Service Unavailable)

ciqne5onq6dbtaqlamhh6yi3bv36o7olkaaucygae5gmdd4boxceyja.infura.status.im/:1

503 Service Unavailable

No server is available to handle this request.

[annadanchenko]

1. on iOS, with 0.11.0 release, browser privacy mode is ON, Ropsten: I can open Test Dapp fine, address is resolved to https://ciqne5onq6dbtaqlamhh6yi3bv36o7olkaaucygae5gmdd4boxceyja.infura.status.im
2. on iOS, with 0.11.0 release, browser privacy mode is ON, Mainnet: 503 error if open ENS Dapp from profile, address is resolved to https://ciqhjnk4hajetn6bu3slnf5im6e5vzkwv34wlgfncrpgwvlbkwpzcxa.infura.status.im

[jakubgs]

OH:

ciqne5onq6dbtaqlamhh6yi3bv36o7olkaaucygae5gmdd4boxceyja.infura.status.im;
its security certificate expired yesterday.

[jakubgs]

Yep:

 $  curl -sv https://ciqne5onq6dbtaqlamhh6yi3bv36o7olkaaucygae5gmdd4boxceyja.infura.status.im 2>&1 | grep cert
                                                                                                                     
* successfully set certificate verify locations:
  CApath: /etc/ssl/certs
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS alert, certificate expired (557):
* SSL certificate problem: certificate has expired

[jakubgs]

We use LetsEncrypt for this cert since CloudFlare doesn't support wildcard certificates of this type, but certbot doesn't think the certificate needs a refresh:

admin@node-01.do-ams3.proxy.misc:~ % sudo certbot renew --no-self-upgrade                   
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/infura.status.im.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cert not yet due for renewal

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

The following certs are not due for renewal yet:
  /etc/letsencrypt/live/infura.status.im/fullchain.pem expires on 2019-06-21 (skipped)
No renewals were attempted.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Must be some misconfiguration. Investigating.

[jakubgs]

Looks expired to me:

 $ echo | openssl s_client -servername ciqne5onq6dbtaqlamhh6yi3bv36o7olkaaucygae5gmdd4boxceyja.infura.status.im -connect ciqne5onq6dbtaqlamhh6yi3bv36o7olkaaucygae5gmdd4boxceyja.infura.status.im:443 2>/dev/null | openssl x509 -noout -dates 
                                                                       
notBefore=Jan 22 18:55:03 2019 GMT
notAfter=Apr 22 18:55:03 2019 GMT

[jakubgs]

And now it's not:

 caspair > echo | openssl s_client -servername ciqne5onq6dbtaqlamhh6yi3bv36o7olkaaucygae5gmdd4boxceyja.infura.status.im -connect ciqne5onq6dbtaqlamhh6yi3bv36o7olkaaucygae5gmdd4boxceyja.infura.status.im:443 2>/dev/null | openssl x509 -noout -dates
                                                                                                               
notBefore=Mar 23 21:43:36 2019 GMT
notAfter=Jun 21 21:43:36 2019 GMT

And all I did is systemctl reload nginx on our proxy host.
So looks like certbot did refresh the cert on the 23rd, but failed to make Nginx notice. There are indeed two certificates now:

admin@node-01.do-ams3.proxy.misc:~ % sudo ls -l /etc/letsencrypt/archive/infura.status.im               
total 32
-rw-r--r-- 1 root root 1919 Jan 22 19:55 cert1.pem
-rw-r--r-- 1 root root 1919 Mar 23 22:43 cert2.pem
-rw-r--r-- 1 root root 1647 Jan 22 19:55 chain1.pem
-rw-r--r-- 1 root root 1647 Mar 23 22:43 chain2.pem
-rw-r--r-- 1 root root 3566 Jan 22 19:55 fullchain1.pem
-rw-r--r-- 1 root root 3566 Mar 23 22:43 fullchain2.pem
-rw------- 1 root root 1704 Jan 22 19:55 privkey1.pem
-rw-r--r-- 1 root root 1704 Mar 23 22:43 privkey2.pem

Works now:
https://ciqne5onq6dbtaqlamhh6yi3bv36o7olkaaucygae5gmdd4boxceyja.infura.status.im/

[jakubgs]

According to: https://certbot.eff.org/docs/using.html#renewing-certificates

The renew command includes hooks for running commands or scripts before or after a certificate is renewed. For example, if you have a single certificate obtained using the standalone plugin, you might need to stop the webserver before renewing so standalone can bind to the necessary ports, and then restart it after the plugin is finished.

certbot renew --pre-hook "service nginx stop" --post-hook "service nginx start"

But according to this thread one can also use the --nginx option:
certbot/certbot#5486
I'm not sure which approach is correct yet.

[jakubgs]

Ah, since we don't use the --nginx option in our initial cert creation the hook is probably the correct way.
Fix: https://github.com/status-im/infra-misc/commit/8ad24657b83deb136e903ab1879f2cf7b9afd00a
I consider this fixed. But I have added myself a calender reminder to check this in before next expiry date.