Skip to content
This repository has been archived by the owner on Feb 12, 2021. It is now read-only.

Latest commit

 

History

History
237 lines (164 loc) · 10 KB

getting-started-with-etcd.md

File metadata and controls

237 lines (164 loc) · 10 KB

Getting started with etcd

etcd is an open-source distributed key value store that provides shared configuration and service discovery for Container Linux clusters. etcd runs on each machine in a cluster and gracefully handles leader election during network partitions and the loss of the current leader.

Application containers running on your cluster can read and write data into etcd. Common examples are storing database connection details, cache settings, feature flags, and more. This guide will walk you through a basic example of reading and writing to etcd then proceed to other features like TTLs, directories and watching a prefix. This guide is way more fun when you've got at least one Container Linux machine up and running — try it on Amazon EC2 or locally with Vagrant.

Complete etcd API Docs

The version 3 series is the current edition of etcd. Version 3 etcd binaries are not included in the Container Linux filesystem. Instead, etcd is distributed in a container fetched by an included system service. Read on below in setting up etcd and see the etcd on Container Linux FAQ for more details on etcd v3 in a container on Container Linux.

Setting up etcd

Container Linux's etcd-member.service systemd unit knows how to fetch and run the current etcd v3.x container image, providing etcd v3 without requiring the binary to be present in every default OS installation.

etcd v3 startup can be configured at a new node's first boot with a Container Linux Config.

To upgrade an existing etcd v2 cluster rather than deploy a new one, start with the etcd v2 to v3 upgrade doc.

A Container Linux Config can be used to set any etcd option, like in this example:

etcd:
  name:                        my-etcd-1
  listen_client_urls:          https://10.240.0.1:2379
  advertise_client_urls:       https://10.240.0.1:2379
  listen_peer_urls:            https://10.240.0.1:2380
  initial_advertise_peer_urls: https://10.240.0.1:2380
  initial_cluster:             my-etcd-1=https://10.240.0.1:2380,my-etcd-2=https://10.240.0.2:2380,my-etcd-3=https://10.240.0.3:2380
  initial_cluster_token:       my-etcd-token
  initial_cluster_state:       new

Reading and writing to etcd

The HTTP-based API is easy to use. This guide will show both etcdctl and curl examples.

From a Container Linux machine, set a key message with value Hello:

$ etcdctl set /message Hello
Hello
$ curl -X PUT http://127.0.0.1:2379/v2/keys/message -d value="Hello"
{"action":"set","node":{"key":"/message","value":"Hello","modifiedIndex":4,"createdIndex":4}}

Read the value of message back:

$ etcdctl get /message
Hello
$ curl http://127.0.0.1:2379/v2/keys/message
{"action":"get","node":{"key":"/message","value":"Hello","modifiedIndex":4,"createdIndex":4}}

If you followed a guide to set up more than one Container Linux machine, you can SSH into another machine and can retrieve this same value.

To delete the key run:

$ etcdctl rm /message

$ curl -X DELETE http://127.0.0.1:2379/v2/keys/message
{"action":"delete","node":{"key":"/message","modifiedIndex":19,"createdIndex":4}}

Reading and writing from inside a container

To read and write to etcd from within a container you must use the IP address assigned to the docker0 interface on the Container Linux host. From the host, run ip address show to find this address. It's normally 172.17.0.1.

To read from etcd, replace 127.0.0.1 when running curl in the container:

$ curl http://172.17.0.1:2379/v2/keys/
{"action":"get","node":{"key":"/","dir":true,"nodes":[{"key":"/coreos.com","dir":true,"modifiedIndex":4,"createdIndex":4}]}}

You can also fetch the docker0 IP programmatically:

ETCD_ENDPOINT="$(ifconfig docker0 | awk '/\<inet\>/ { print $2}'):2379"

Proxy example

Let's pretend we're setting up a service that consists of a few containers that are behind a proxy container. We can use etcd to announce these containers when they start by creating a directory, having each container write a key within that directory and have the proxy watch the entire directory. We're going to skip creating the containers here but the docker guide is a good place to start for that.

Create the directory

Directories are automatically created when a key is placed inside. Let's call our directory foo-service and create a key with information about a container:

$ etcdctl mkdir /foo-service
Cannot print key [/foo-service: Is a directory]
$ etcdctl set /foo-service/container1 localhost:1111
localhost:1111
$ curl -X PUT http://127.0.0.1:2379/v2/keys/foo-service/container1 -d value="localhost:1111"
{"action":"set","node":{"key":"/foo-service/container1","value":"localhost:1111","modifiedIndex":17,"createdIndex":17}}

Read the foo-service directory to see the entry:

$ etcdctl ls /foo-service
/foo-service/container1
$ curl http://127.0.0.1:2379/v2/keys/foo-service
{"action":"get","node":{"key":"/foo-service","dir":true,"nodes":[{"key":"/foo-service/container1","value":"localhost:1111","modifiedIndex":17,"createdIndex":17}],"modifiedIndex":17,"createdIndex":17}}

Watching the directory

Now let's try watching the foo-service directory for changes, just like our proxy would have to. First, open up another shell on a Container Linux host in the cluster. In one window, start watching the directory and in the other window, add another key container2 with the value localhost:2222 into the directory. This command shouldn't output anything until the key has changed. Many events can trigger a change, including a new, updated, deleted or expired key.

$ etcdctl watch --recursive /foo-service

$ curl http://127.0.0.1:2379/v2/keys/foo-service?wait=true\&recursive=true

In the other window, let's pretend a new container has started and announced itself to the proxy by running:

$ etcdctl set /foo-service/container2 localhost:2222
localhost:2222
$ curl -X PUT http://127.0.0.1:2379/v2/keys/foo-service/container2 -d value="localhost:2222"
{"action":"set","node":{"key":"/foo-service/container2","value":"localhost:2222","modifiedIndex":23,"createdIndex":23}}

In the first window, you should get the notification that the key has changed. In a real application, this would trigger reconfiguration.

$ etcdctl watch --recursive /foo-service
localhost:2222
$ curl http://127.0.0.1:2379/v2/keys/foo-service?wait=true\&recursive=true
{"action":"set","node":{"key":"/foo-service/container2","value":"localhost:2222","modifiedIndex":23,"createdIndex":23}}

Watching the directory and triggering an executable

Now let's try watching the foo-service directory for changes and - if there are any - run the command. In one window, start watching the directory and in the other window, add another key container3 with the value localhost:2222 into the directory. This command shouldn't trigger anything until the key has changed. The same events as in the previous example can trigger a change. The exec-watch command expects etcdctl to run continuously (for watch command you can use --forever option)

$ etcdctl exec-watch --recursive /foo-service -- sh -c 'echo "\"$ETCD_WATCH_KEY\" key was updated to \"$ETCD_WATCH_VALUE\" value by \"$ETCD_WATCH_ACTION\" action"'

In the other window, let's imagine a new container has started and announced itself to the proxy by running:

$ etcdctl set /foo-service/container3 localhost:2222
localhost:2222

In the first window, you should get the notification that the key has changed. We have used $ETCD_WATCH_* environment variables which were set by etcdctl.

$ etcdctl exec-watch --recursive /foo-service -- sh -c 'echo "\"$ETCD_WATCH_KEY\" key was updated to \"$ETCD_WATCH_VALUE\" value by \"$ETCD_WATCH_ACTION\" action"'
"/foo-service/container3" key was updated to "localhost:2222" value by "set" action

Test and set

etcd can be used as a centralized coordination service and provides TestAndSet functionality as the building block of such a service. You must provide the previous value along with your new value. If the previous value matches the current value the operation will succeed.

$ etcdctl set /message "Hi" --swap-with-value "Hello"
Hi
$ curl -X PUT http://127.0.0.1:2379/v2/keys/message?prevValue=Hello -d value=Hi
{"action":"compareAndSwap","node":{"key":"/message","value":"Hi","modifiedIndex":28,"createdIndex":27}}

TTL

You can optionally set a TTL for a key to expire in a certain number of seconds. Setting a TTL of 20 seconds:

$ etcdctl set /foo "Expiring Soon" --ttl 20
Expiring Soon

The curl response will contain an absolute timestamp of when the key will expire and a relative number of seconds until that timestamp:

$ curl -X PUT http://127.0.0.1:2379/v2/keys/foo?ttl=20 -d value=bar
{"action":"set","node":{"key":"/foo","value":"bar","expiration":"2014-02-10T19:54:49.357382223Z","ttl":20,"modifiedIndex":31,"createdIndex":31}}

If you request a key that has already expired, you will be returned a 100:

$ etcdctl get /foo
Error: 100: Key not found (/foo) [32]
$ curl http://127.0.0.1:2379/v2/keys/foo
{"errorCode":100,"message":"Key not found","cause":"/foo","index":32}

More information

etcd Overview Full etcd API Docs Projects using etcd