Proposal: Labels and Swarm

[aluzzardi]

We've been waiting for labels for a long time and they might soon come with moby/moby#9882

This will open a lot of possibilities for Swarm and will affect the way users interact with it, I thought we should start a conversation.

Metadata Storage

Right now, we rely on environment variables to store things such as constraints.

This is great as it's a form of distributed storage: For instance, instead of storing centrally in Swarm all the constraints for a specific container (which it will need later on for rebalancing), the metadata is directly embedded into the container.

The problem is that we are polluting the container environment with internal metadata.

Labels on the other hand offer a much better place to store our metadata.

Namespacing has not been figured out yet, but we could have the following labels attached to Swarm containers:

com.docker.swarm.id=<swarm internal id of the container>
com.docker.swarm.constraint=<list of constraints the container has been scheduled with>
com.docker.swarm.constraint=<list of affinities the container has been scheduled with>

Report extra information

Currently, we are mangling the container names in docker ps in order to display the node name (`/).

Instead, we could revert back to regular names and report that information as labels such as:

docker ps --label node
CONTAINER ID        NAMES  [...]  NODE
d81ed81b6f73        redis  [...]  node1
c655a865d0c6        nginx  [...]  node2
4e286948b26e        nginx  [...]  node3

Many other pieces of information could be retrieved in the same way, such as the node ID, the container constraints, etc.

Filtering

Labels could also be used for filtering:

docker ps --label node=node1`  # List all containers in node1
docker stop --label node=node2  # Stop containers on node2

Local debugging of nodes

When ssh'ing directly into a node (or running docker commands against that node's API), the swarm metadata could be displayed and used for filtering.

For instance, if we decide to have different IDs between Swarm and Docker, labels could be used to interact with the swarm containers locally:

docker ps --label com.docker.swarm.id  # Display the swarm id next to the container
docker rm -f --label com.docker.swarm.id=foo # Remove a container using the swarm id

[chanwit]

+1

[thaJeztah]

+1 (exactly how I envisioned the labels to be used)

One nit, as it stands now, the same label can only be assigned once, so I think you have a typo in your first example;

com.docker.swarm.constraint=<list of constraints the container has been scheduled with>
com.docker.swarm.constraint=<list of affinities the container has been scheduled with>

[aluzzardi]

@thaJeztah: Indeed, I don't know yet if labels will support multi values. If not, I think we could simply serialize them (using : for instance, like $PATH). Did you envision this differently?

[aluzzardi]

/cc @aanand @bfirsh

[thaJeztah]

Multiple values was originally in the proposal, but later removed, to keep things simple; allowing multiple values would also raise questions, such as; when should an existing label be overwritten when should it be appended, how to remove single value from the list of labels, etc.

(I realise some of those points are not relevant directly now, because labels will be immutable in the first implementation)

To come back to your use-case; I think there are a number of (possible) solutions;

Like you proposed; using a separator to concatenate the key/values. I don't know if this becomes a problem if (at any point), the separator could occur in values being used. Something like;

com.docker.swarm.constraint="foo=bar:baz=bang"

Swarm could store the values as serialized JSON. This would bring more flexibility, perhaps easier to parse the values, and "basic" type support. Obvious downside is that (for now) filtering always has to be done in swarm; swarm has to retrieve all containers, decode and parse the JSON.

Something like:

com.docker.swarm.constraint="{\"foo\":\"bar\",\"baz\":\"bang\"}"

A third option is to move the name of the constraint inside the name/namespace of the label. This could bring some support for filtering "server-side" in the daemon, if needed;

com.docker.swarm.constraint.foo=bar
com.docker.swarm.constraint.baz=bang

I'm just thinking out loud here, so I hope it's useful 😄

[aluzzardi]

Thanks @thaJeztah, this is useful!

The part where it gets complicated is that since we use expressions, key/values won't be enough as we need to encode the operator (==, !=, etc) as well.

We could either serialize that in json or use a separator such as:

com.docker.swarm.constrant=foo==bar:hello!=world

I really like the third option, but com.docker.swarm.constrant.hello=!=world is really awkward.

[thaJeztah]

Ah, yes, I completely overlooked that in my thinking. The node holds the value, the container the expression/conditions. Hmmm. I'll start thinking out loud again, but will come back after some more thinking :)

Basically, what's needed is a clean way to store "queries". Performance is probably not a key factor here, because each constraint/condition has to be analysed anyway, and only would be done when (re)scheduling containers, right?

Do we need to take more complex conditions into account for the future? Nested conditions?

storage=ssd (or) (storage=disk (and) raid-level=5)

Probably way beyond topic, just setting my mind to work :)

JSON could work, but needs some proper thinking, it will become quite verbose quickly. Which may not be a problem, because it's generated by swarm, but still.. a quick draft;

[
    {"key":"bar", "op":"EQ", "value":"baz"},
    {"key":"hello", "op":"NE", "value":"planet"}
]

(Which could be expanded on (e.g. Nested arrays/conditions for "AND" / "OR" constraints))

Again, this allows future expansion, but is quite verbose, so might be taking this too far.

Setting the constraints on the CLI, could take a syntax like this --constraint bar==baz --constraint hello!=planet, not taking into account the complex example mentioned earlier.

[bfirsh]

If we're namespacing labels, I really hope there's some way we don't have to type com.docker.swarm... all the time. I wonder if we can reserve swarm as a special namespace.

[thaJeztah]

@bfirsh I agree that typing the whole namespace can become cumbersome; perhaps Swarm should automatically prepend the namespace for swarm-specific usage.

[dnephin]

I think this was done? https://github.com/docker/swarm/blob/master/cluster/config.go#L76 seems to support it.

Can this issue be closed?

[vieux]

Yes, thanks @dnephin