Secure local cluster Admin UI is not accessible on Chrome

[jseldess]

When running a secure local cluster, following our getting started docs, it's no longer possible to access the Admin UI on Chrome. Seems to work fine on FF and Safari.

Given Chrome's popularity and the importance of helping users have a quick and easy getting started experience via our docs and otherwise, I consider this a significant problem, potentially a release blocker, though that's not my call. At the least, we need to figure out how to adjust the docs so that copy/paste works for spinning up a local cluster and accessing the admin UI for new users.

Steps:

~$ mkdir certs my-safe-directory

~$ cockroach cert create-ca --certs-dir=certs --ca-key=my-safe-directory/ca.key

~$ cockroach cert create-node localhost $(hostname) --certs-dir=certs --ca-key=my-safe-directory/ca.key

~$ cockroach cert create-client root --certs-dir=certs --ca-key=my-safe-directory/ca.key

~$ cockroach start --certs-dir=certs --store=node1 --listen-addr=localhost:26257 --http-addr=localhost:8080 --join=localhost:26257,localhost:26258,localhost:26259 --background
*
* INFO: initial startup completed.
* Node will now attempt to join a running cluster, or wait for `cockroach init`.
* Client connections will be accepted after this completes successfully.
* Check the log file(s) for progress.
*

~$ cockroach start --certs-dir=certs --store=node2 --listen-addr=localhost:26258 --http-addr=localhost:8081 --join=localhost:26257,localhost:26258,localhost:26259 --background
*
* INFO: initial startup completed.
* Node will now attempt to join a running cluster, or wait for `cockroach init`.
* Client connections will be accepted after this completes successfully.
* Check the log file(s) for progress.
*

~$ cockroach start --certs-dir=certs --store=node3 --listen-addr=localhost:26259 --http-addr=localhost:8082 --join=localhost:26257,localhost:26258,localhost:26259 --background
*
* INFO: initial startup completed.
* Node will now attempt to join a running cluster, or wait for `cockroach init`.
* Client connections will be accepted after this completes successfully.
* Check the log file(s) for progress.
*

~$ cockroach init --certs-dir=certs --host=localhost:26257
Cluster successfully initialized

~$ grep 'node starting' node1/logs/cockroach.log -A 11
CockroachDB node starting at 2020-03-19 15:07:58.540333 +0000 UTC (took 11.5s)
build:               CCL v20.1.0-beta.2-937-g1ae9b8d2cd @ 2020/03/19 03:25:00 (go1.13.7)
webui:               https://localhost:8080
sql:                 postgresql://root@localhost:26257?sslcert=certs%2Fclient.root.crt&sslkey=certs%2Fclient.root.key&sslmode=verify-full&sslrootcert=certs%2Fca.crt
RPC client flags:    cockroach <client cmd> --host=localhost:26257 --certs-dir=certs
logs:                /Users/jesseseldess/node1/logs
temp dir:            /Users/jesseseldess/node1/cockroach-temp947498966
external I/O path:   /Users/jesseseldess/node1/extern
store[0]:            path=/Users/jesseseldess/node1
storage engine:      rocksdb
status:              initialized new cluster
clusterID:           aa69e24b-ffbd-4716-a49a-fb7371ba80fc

When you go to https://localhost:8080 in Chrome and click "Advanced" on the initial "Your connection is not private" warning page, you get:

In contrast, when you do the same think in FF, you have the option to "Accept risk and continue" and can create the Admin UI login page. Same basic path on Safari.

As suggested by @knz, I tried listing $(hostname) before localhost in the cert and starting nodes with --http-addr=$(hostname):8080 and then directing browser to https://<hostname>:8080, but no luck.

Environment:
CockroachDB version: v20.1.0-beta.2-937-g1ae9b8d2cd
Server OS: MacOS
Chrome version: Version 80.0.3987.149 (Official Build) (64-bit)
FF version: 72.0.2 (64-bit)
Safari versio: 13.0.5 (15608.5.11)

[knz]

@aaron-crl do you know what are the new requirements from Chrome? I can't deduce them from the error message alone and I don't have a chrome locally to test with

[aaron-crl]

This looks like a combination of OSX Catalina TLS changes combined with Chrome's stricter enforcement behavior.

Possibly related: "TLS server certificates must present the DNS name of the server in the Subject Alternative Name extension of the certificate. DNS names in the CommonName of a certificate are no longer trusted." - https://support.apple.com/en-us/HT210176

And: Getting NET::ERR_CERT_REVOKED error with locally-signed certificate - https://support.google.com/chrome/thread/14551925?hl=en

I'll figure out the best path to make this work for Chrome and get back to you.

[dbist]

If you are using Google Chrome as your browser, you may want to navigate here chrome://flags/#allow-insecure-localhost and set this flag to Enabled.

[aaron-crl]

There are two contributing factors to this:

(1) Chrome has an intentionally high wall around certificates that do not chain back to the system root store.
(2) OSX Catalina recently changed the rules for trusted certificates. Most important for us is:

• Use of SubjectAltName for localhost
• Certificate lifespan cannot exceed 825 days

I was able to get this working by:

• Generating certificates in openssl with modified .cnf files and command line parameters (below)
• Installing the generated CA into the OSX Keychain (login store worked fine)
• Marking the CA as trusted in the OSX Keychain

We should revisit the openssl instructions generally but I've included the deltas below here in the meantime.

---

(Re: https://www.cockroachlabs.com/docs/dev/create-security-certificates-openssl.html):

• In ca.cnf change default_days to 365
• In node.cnf add to [ extensions ] the line subjectAltName = DNS:localhost,DNS:node,IP:0.0.0.0
• In the create CA steps (https://www.cockroachlabs.com/docs/dev/create-security-certificates-openssl.html#create-the-ca-key-and-certificate-pair):

• Step 4, change -days 3660 to -days 365

• In the create node steps (https://www.cockroachlabs.com/docs/dev/create-security-certificates-openssl.html#create-the-certificate-and-key-pairs-for-nodes)

• Step 4, remove -days 1830

[aaron-crl]

If you are using Google Chrome as your browser, you may want to navigate here chrome://flags/#allow-insecure-localhost and set this flag to Enabled.

Or that ^^

Thanks @dbist !

[jseldess]

That works, @dbist! Thank you. I'll transfer this issue to the docs repo and update the docs.

[knz]

Note that changing the chrome settings degrades the user's browser security for everything, not just crdb. Some eyebrows will be raised. We need to acknowledge this is not a panacea and recommend users to reset the change when they actually deploy.