Configuration reconciliation (aka kubectl apply)

[bgrant0607]

Forked from #1007 and #1325 .

We want to support management of services via declarative configuration.

The configuration files written by the user may need to undergo multiple transformations before submitting them to the API, as discussed in #1694, but the result of these steps should be concrete objects. These objects may need to be created, updated, and/or deleted.

On an object-by-object basis, it should be possible via kubectl (via a client library) to do the following:

1. Determine whether or not the specified object already exists
2. Get the specified object
3. Create the specified object
4. Diff the specified object with an existing object in apiserver
5. Update an existing object in apiserver with the data from the specified object
6. Delete the specified object
7. Delete all objects with deployment-specific labels (see Name+label+selector scoping for configuration composition #1698)

The configured fields should be recorded in the object via annotations, as discussed in #1178 and #1201. Only the previously and newly configured fields, labels, and annotations should be diff'ed and/or updated. Other fields from the existing object should be ignored/merged, and the set of configured fields should be updated with each update.

We ultimately do want to automate most of the orchestration/workflow involved in deployment. However, this is fairly complex, and we can't necessarily automate all of it in a domain-independent, use-case-independent manner. In general, a full deployment would likely be comprised of multiple files, each containing multiple objects. These objects may have inter-dependencies (e.g., right now services must be created before their clients are created), rolling updates require special treatment (#1353), and application-specific rollout logic may be required. Therefore, I recommend we keep the more general deployment workflow separate from the basic primitive mechanism that reconciles a single object, and ensure that the two are composable. I'll file another issue for the workflow part.

Among other things, composition means that the invoker of the tool and/or library needs to be able to distinguish noop (e.g., object already exists and matches config), success, retryable failure, and permanent failure, so the overall workflow can decide what to do after each step.

/cc @ghodss

[bgrant0607]

/cc @jasonkuhrt

[jasonkuhrt]

@bgrant0607 Thanks. I may contribute some thoughts primarily regarding use-cases at littleBits as we continue to experiment with Kubernetes/GKE. For now, it looks like my first pass at creating an issue in this space is well covered already!