cli: unable to initialize a cluster with --cert-principal-map

[dbist]

Describe the problem

Please describe the issue you observed, and any steps we can take to reproduce it:

To Reproduce
start cluster

[vagrant@krb ~]$ cockroach start --certs-dir=certs --store=node1 --listen-addr=krb.example.com:26257 --http-addr=krb.example.com:8080 --join=krb.example.com:26257,krb.example.com:26258,krb.example.com:26259 --cert-principal-map=krb.example.com:node,max:root --background
*
* INFO: initial startup completed.
* Node will now attempt to join a running cluster, or wait for `cockroach init`.
* Client connections will be accepted after this completes successfully.
* Check the log file(s) for progress.
*
[vagrant@krb ~]$ cockroach start --certs-dir=certs --store=node2 --listen-addr=krb.example.com:26258 --http-addr=krb.example.com:8081 --join=krb.example.com:26257,krb.example.com:26258,krb.example.com:26259 --cert-principal-map=krb.example.com:node,max:root --background
*
* INFO: initial startup completed.
* Node will now attempt to join a running cluster, or wait for `cockroach init`.
* Client connections will be accepted after this completes successfully.
* Check the log file(s) for progress.
*
[vagrant@krb ~]$ cockroach start --certs-dir=certs --store=node3 --listen-addr=krb.example.com:26259 --http-addr=krb.example.com:8082 --join=krb.example.com:26257,krb.example.com:26258,krb.example.com:26259 --cert-principal-map=krb.example.com:node,max:root --background
*
* INFO: initial startup completed.
* Node will now attempt to join a running cluster, or wait for `cockroach init`.
* Client connections will be accepted after this completes successfully.
* Check the log file(s) for progress.

trying to init cluster

cockroach init --certs-dir=certs --host=krb.example.com:26257
warning: node not ready to perform cluster initialization: problem using security settings: client/server node certificate has principals ["" "krb.example.com" "example.com"], expected "node" (retrying)
warning: node not ready to perform cluster initialization: problem using security settings: client/server node certificate has principals ["" "krb.example.com" "example.com"], expected "node" (retrying)
warning: node not ready to perform cluster initialization: problem using security settings: client/server node certificate has principals ["" "krb.example.com" "example.com"], expected "node" (retrying)
warning: node not ready to perform cluster initialization: problem using security settings: client/server node certificate has principals ["" "krb.example.com" "example.com"], expected "node" (retrying)
warning: node not ready to perform cluster initialization: problem using security settings: client/server node certificate has principals ["" "krb.example.com" "example.com"], expected "node" (retrying)

cockroach init --certs-dir=certs --host=krb.example.com:26257 --cert-principal-map=krb.example.com:node,max:root
Usage:
  cockroach init [flags]

Flags:
      --certs-dir string
                                                                                    Path to the directory containing SSL certificates and keys.
                                                                                    Environment variable: COCKROACH_CERTS_DIR
                                                                                    (default "${HOME}/.cockroach-certs")
  -h, --help                                                                help for init
      --host <addr/host>[:<port>]
                                                                                    CockroachDB node to connect to. This can be specified either as an
                                                                                    address/hostname, or together with a port number as in -s myhost:26257.
                                                                                    If the port number is left unspecified, it defaults to 26257. An IPv6
                                                                                    address can also be specified with the notation [...], for example
                                                                                    [::1]:26257 or [fe80::f6f2:::]:26257.
                                                                                    Environment variable: COCKROACH_HOST
                                                                                    (default :26257)
      --insecure
                                                                                    Connect to an insecure cluster. This is strongly discouraged for
                                                                                    production usage.
                                                                                    Environment variable: COCKROACH_INSECURE

      --url postgresql://[user[:passwd]@]host[:port]/[db][?parameters...]
                                                                                    Connection URL, e.g. "postgresql://myuser@localhost:26257/mydb". If
                                                                                    left empty, the connection flags are used (host, port, user, database,
                                                                                    insecure, certs-dir).
                                                                                    Environment variable: COCKROACH_URL

Global Flags:
      --logtostderr Severity[=DEFAULT]   logs at or above this threshold go to stderr (default NONE)
      --no-color                         disable standard error log colorization
      --vmodule moduleSpec               comma-separated list of pattern=N settings for file-filtered logging (significantly hurts performance)

ERROR: unknown flag: --cert-principal-map
Failed running "init"

Expected behavior
cluster should initialize as usual.

Environment:

• CockroachDB version cockroach 20.1.rc.2
• Server OS: Centos 7.6
• Client app cockroach sql

[dbist]

@petermattis you must have a workaround as you said you tested multi-node cluster with --cert-principal-map, perhaps I'm not doing something correct?

[petermattis]

Hrmm. Yeah, that's not good. I used the pre-init version of cluster initialization via roachprod (empty --join flag passed to start). I take it --url doesn't work.

[dbist]

same story

[vagrant@krb ~]$ cockroach init --certs-dir=certs --host=krb.example.com:26257
warning: node not ready to perform cluster initialization: problem using security settings: client/server node certificate has principals ["" "krb.example.com" "example.com"], expected "node" (retrying)
warning: node not ready to perform cluster initialization: problem using security settings: client/server node certificate has principals ["" "krb.example.com" "example.com"], expected "node" (retrying)
warning: node not ready to perform cluster initialization: problem using security settings: client/server node certificate has principals ["" "krb.example.com" "example.com"], expected "node" (retrying)
warning: node not ready to perform cluster initialization: problem using security settings: client/server node certificate has principals ["" "krb.example.com" "example.com"], expected "node" (retrying)
^C
[vagrant@krb ~]$ cockroach init --host=krb.example.com:26257 --url 'postgresql://krb.example.com:26257?sslcert=certs%2Fclient.root.crt&sslkey=certs%2Fclient.root.key&sslmode=verify-full&sslrootcert=certs%2Fca.crt'
warning: node not ready to perform cluster initialization: problem using security settings: client/server node certificate has principals ["" "krb.example.com" "example.com"], expected "node" (retrying)
warning: node not ready to perform cluster initialization: problem using security settings: client/server node certificate has principals ["" "krb.example.com" "example.com"], expected "node" (retrying)

[petermattis]

Ack. Thanks for checking. I'm not surprised by the result.

[dbist]

@petermattis I was able to work around this issue by shutting down the nodes before init and restarting with certs generated by cockroach cert, initializing the cluster and shutting it down. Then restarting the nodes the same way as above. Not sure if it's correct but in my case it worked.

[petermattis]

@petermattis I was able to work around this issue by shutting down the nodes before init and restarting with certs generated by cockroach cert, initializing the cluster and shutting it down. Then restarting the nodes the same way as above. Not sure if it's correct but in my case it worked.

That would work, though I think we can agree that it is a pretty gross workaround. #47449 will be merging shortly which will make this situation a bit better. I'd love for you to take it for a spin when it merges.

[dbist]

I'll give it a spin when it's ready