ETCD Fails to start

[rmenn]

I have the following error when i try and bring up a self hosted etcd cluster

Mar 10 13:35:45 master etcd-wrapper[11781]: 2017-03-10 13:35:45.840410 I | pkg/flags: recognized and used environment variable ETCD_DATA_DIR=/var/lib/etcd
Mar 10 13:35:45 master etcd-wrapper[11781]: 2017-03-10 13:35:45.840861 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_CLUSTER=controller=http://${COREOS_PRIVATE_IPV4}:2380
Mar 10 13:35:45 master etcd-wrapper[11781]: 2017-03-10 13:35:45.841109 I | pkg/flags: recognized and used environment variable ETCD_LISTEN_CLIENT_URLS=http://0.0.0.0:2379
Mar 10 13:35:45 master etcd-wrapper[11781]: 2017-03-10 13:35:45.841405 I | pkg/flags: recognized and used environment variable ETCD_LISTEN_PEER_URLS=http://0.0.0.0:2380
Mar 10 13:35:45 master etcd-wrapper[11781]: 2017-03-10 13:35:45.841654 I | pkg/flags: recognized and used environment variable ETCD_NAME=controller
Mar 10 13:35:45 master etcd-wrapper[11781]: 2017-03-10 13:35:45.841903 W | pkg/flags: unrecognized environment variable ETCD_USER=etcd
Mar 10 13:35:45 master etcd-wrapper[11781]: 2017-03-10 13:35:45.843595 W | pkg/flags: unrecognized environment variable ETCD_IMAGE_TAG=v3.1.2
Mar 10 13:35:45 master etcd-wrapper[11781]: 2017-03-10 13:35:45.843907 E | etcdmain: error verifying flags, --advertise-client-urls is required when --listen-client-urls is set explicitly. See 'etcd --help'.
Mar 10 13:35:45 master etcd-wrapper[11781]: 2017-03-10 13:35:45.844143 E | etcdmain: When listening on specific address(es), this etcd process must advertise accessible url(s) to each connected client.

Here is the config

[Service]
EnvironmentFile=/etc/environment
Environment="ETCD_IMAGE_TAG=v3.1.2"
Environment="ETCD_NAME=controller"
Environment="ETCD_INITIAL_CLUSTER=controller=http://${COREOS_PRIVATE_IPV4}:2380"
Environment="ETCD_INITIAL_ADVERTISE_PEER_URLS=http://${COREOS_PRIVATE_IPV4}:2380"
Environment="ETCD_ADVERTISE_CLIENT_URLS=http://${COREOS_PRIVATE_IPV4}:2379"
Environment="ETCD_LISTEN_CLIENT_URLS=http://0.0.0.0:2379"
Environment="ETCD_LISTEN_PEER_URLS=http://0.0.0.0:2380"

I am passing both ETCD_ADVERTISE_CLIENT_URLS and ETCD_LISTEN_CLIENT_URLS as required but as in the logs it isnt recognizing and picking up ETCD_ADVERTISE_CLIENT_URLS.

[aaronlevy]

@rmenn When using self-hosted etcd there should be no service file on the host (everything is run as a pod - including etcd).

This PR #350 (should be merged shortly) allows launching self-hosted etcd clusters using the hack/quickstart examples - might be a good place to start from?

Unfortunately we haven't started documenting much of this (and I apologize for that) - some doc related tracking issues:

#240
#311
#302

[rmenn]

@aaronlevy i get the problems addressed by #350 but am i am still at wits end why etcd isnt accepting environment variables to start up, regardless of the face if i am using the self hosted version or not.

Please let me know if i am rolling out something which shouldnt, the etcd errors are quite beyond me

[peebs]

@rmenn: The issue your having is in regards then to running a etcd node manually on the host? Are you placing that config as a drop-in under /etc/systemd/system/etcd-member.service.d/10-etcd-member.conf? Also, what version of Container Linux are you using?

[rmenn]

I am using the latest stable and passing the config via cloud config the environment variables are passed via cloud config as well.

[peebs]

Thats very odd, my best guess would be a subtle error in your cloud-config yaml or possibly existing drop-ins sitting around on-disk. Just taking a stab in the dark here. Maybe you can post the entire cloud-config yaml here? Here is an example: https://github.com/kubernetes-incubator/bootkube/blob/master/hack/multi-node/etcd-cloud-config.yaml

Also, it has an error about ETCD_USER not being recognize, but I don't see it in your service file config. I'm not sure why it has an issue with your ETCD_IMAGE_TAG variable because that is set and looks correct.

[rmenn]

Here is the cloud config
https://gist.github.com/rmenn/90722fe72e4a5dd8dd56d540a02e9c2b

[aaronlevy]

@rmenn You shouldn't use --experimental-self-hosted-etcd and start etcd locally via systemd. At minimum there would be port conflicts - but not really sure what the resulting behavior would end up being.

If you're "bringing your own etcd" - then it could be started via systemd (or otherwise). But if you're testing self-hosted etcd - it is expected to run in the cluster as pods (not on the host) and bootkube/etcd-operator should manage that for you.

[rmenn]

@aaronlevy i realized my mistake a day ago, i should have closed this out. But still curious why the errors showed up. But can confirm that the errors dont show up when i use the self hosted and not supply the systemd config for etcd.