Skip to content
This repository has been archived by the owner on Dec 2, 2021. It is now read-only.

Latest commit

 

History

History
395 lines (248 loc) · 58 KB

declarative-application-management.md

File metadata and controls

395 lines (248 loc) · 58 KB
Raw

Declarative application management in Kubernetes

This article was authored by Brian Grant (bgrant0607) on 8/2/2017. The original Google Doc can be found here: https://goo.gl/T66ZcD

Most users will deploy a combination of applications they build themselves, also known as bespoke applications, and common off-the-shelf (COTS) components. Bespoke applications are typically stateless application servers, whereas COTS components are typically infrastructure (and frequently stateful) systems, such as databases, key-value stores, caches, and messaging systems.

In the case of the latter, users sometimes have the choice of using hosted SaaS products that are entirely managed by the service provider and are therefore opaque, also known as blackbox services. However, they often run open-source components themselves, and must configure, deploy, scale, secure, monitor, update, and otherwise manage the lifecycles of these whitebox COTS applications.

This document proposes a unified method of managing both bespoke and off-the-shelf applications declaratively using the same tools and application operator workflow, while leveraging developer-friendly CLIs and UIs, streamlining common tasks, and avoiding common pitfalls. The approach is based on observations of several dozen configuration projects and hundreds of configured applications within Google and in the Kubernetes ecosystem, as well as quantitative analysis of Borg configurations and work on the Kubernetes system architecture, API, and command-line tool (kubectl).

The central idea is that a toolbox of composable configuration tools should manipulate configuration data in the form of declarative API resource specifications, which serve as a declarative data model, not express configuration as code or some other representation that is restrictive, non-standard, and/or difficult to manipulate.

Declarative configuration

Why the heavy emphasis on configuration in Kubernetes? Kubernetes supports declarative control by specifying users’ desired intent. The intent is carried out by asynchronous control loops, which interact through the Kubernetes API. This declarative approach is critical to the system’s self-healing, autonomic capabilities, and application updates. This approach is in contrast to manual imperative operations or flowchart-like orchestration.

This is aligned with the industry trend towards immutable infrastructure, which facilitates predictability, reversibility, repeatability, scalability, and availability. Repeatability is even more critical for containers than for VMs, because containers typically have lifetimes that are measured in days, hours, even minutes. Production container images are typically built from configurable/scripted processes and have parameters overridden by configuration rather than modifying them interactively.

What form should this configuration take in Kubernetes? The requirements are as follows:

  • Perhaps somewhat obviously, it should support bulk management operations: creation, deletion, and updates.

  • As stated above, it should be universal, usable for both bespoke and off-the-shelf applications, for most major workload categories, including stateless and stateful, and for both development and production environments. It also needs to be applicable to use cases outside application definition, such as policy configuration and component configuration.

  • It should expose the full power of Kubernetes (all CRUD APIs, API fields, API versions, and extensions), be consistent with concepts and properties presented by other tools, and should teach Kubernetes concepts and API, while providing a bridge for application developers that prefer imperative control or that need wizards and other tools to provide an onramp for beginners.

  • It should feel native to Kubernetes. There is a place for tools that work across multiple platforms but which are native to another platform and for tools that are designed to work across multiple platforms but are native to none, but such non-native solutions would increase complexity for Kubernetes users by not taking full advantage of Kubernetes-specific mechanisms and conventions.

  • It should integrate with key user tools and workflows, such as continuous deployment pipelines and application-level configuration formats, and compose with built-in and third-party API-based automation, such as admission control, autoscaling, and Operators. In order to do this, it needs to support separation of concerns by supporting multiple distinct configuration sources and preserving declarative intent while allowing automatically set attributes.

  • In particular, it should be straightforward (but not required) to manage declarative intent under version control, which is standard industry best practice and what Google does internally. Version control facilitates reproducibility, reversibility, and an audit trail. Unlike generated build artifacts, configuration is primary human-authored, or at least it is desirable to be human-readable, and it is typically changed with a human in the loop, as opposed to fully automated processes, such as autoscaling. Version control enables the use of familiar tools and processes for change control, review, and conflict resolution.

  • Users need the ability to customize off-the-shelf configurations and to instantiate multiple variants, without crossing the line into the ecosystem of configuration domain-specific languages, platform as a service, functions as a service, and so on, though users should be able to layer such tools/systems on top of the mechanism, should they choose to do so.

  • We need to develop clear conventions, examples, and mechanisms that foster structure, to help users understand how to combine Kubernetes’s flexible mechanisms in an effective manner.

Configuration customization and variant generation

The requirement that drives the most complexity in typical configuration solutions is the need to be able to customize configurations of off-the-shelf components and/or to instantiate multiple variants.

Deploying an application generally requires customization of multiple categories of configuration:

  • Frequently customized

    • Context: namespaces, names, labels, inter-component references, identity

    • Image: repository/registry (source), tag (image stream/channel), digest (specific image)

    • Application configuration, overriding default values in images: command/args, env, app config files, static data

    • Resource parameters: replicas, cpu, memory, volume sources

    • Consumed services: coordinates, credentials, and client configuration

  • Less frequently customized

    • Management parameters: probe intervals, rollout constraints, utilization targets

  • Customized per environment

    • Environmental adapters: lifecycle hooks, helper sidecars for configuration, monitoring, logging, network/auth proxies, etc

    • Infrastructure mapping: scheduling constraints, tolerations

    • Security and other operational policies: RBAC, pod security policy, network policy, image provenance requirements

  • Rarely customized

    • Application topology, which makes up the basic structure of the application: new/replaced components

In order to make an application configuration reusable, users need to be able to customize each of those categories of configuration. There are multiple approaches that could be used:

  • Fork: simple to understand; supports arbitrary changes and updates via rebasing, but hard to automate in a repeatable fashion to maintain multiple variants

  • Overlay / patch: supports composition and useful for standard transformations, such as setting organizational defaults or injecting environment-specific configuration, but can be fragile with respect to changes in the base configuration

  • Composition: useful for orthogonal concerns

    • Pull: Kubernetes provides APIs for distribution of application secrets (Secret) and configuration data (ConfigMap), and there is a proposal open to support application data as well

      • the resource identity is fixed, by the object reference, but the contents are decoupled

        • the explicit reference makes it harder to consume a continuously updated stream of such resources, and harder to generate multiple variants

      • can give the PodSpec author some degree of control over the consumption of the data, such as environment variable names and volume paths (though service accounts are at conventional locations rather than configured ones)

    • Push: facilitates separation of concerns and late binding

      • can be explicit, such as with kubectl set or HorizontalPodAutoscaler

      • can be implicit, such as with LimitRange, PodSecurityPolicy, PodPreset, initializers

        • good for attaching policies to selected resources within a scope (namespace and/or label selector)

  • Transformation: useful for common cases (e.g., names and labels)

  • Generation: useful for static decisions, like "if this is a Java app…", which can be integrated into the declarative specification

  • Automation: useful for dynamic adaptation, such as horizontal and vertical auto-scaling, improves ease of use and aids encapsulation (by not exposing those details), and can mitigate phase-ordering problems

  • Parameterization: natural for small numbers of choices the user needs to make, but there are many pitfalls, discussed below

Rather than relying upon a single approach, we should combine these techniques such that disadvantages are mitigated.

Tools used to customize configuration within Google have included:

  • Many bespoke domain-specific configuration languages (DSLs)

  • Python-based configuration DSLs (e.g., Skylark)

  • Transliterate configuration DSLs into structured data models/APIs, layered over and under existing DSLs in order to provide a form that is more amenable to automatic manipulation

  • Configuration overlay systems, override mechanisms, and template inheritance

  • Configuration generators, manipulation CLIs, IDEs, and wizards

  • Runtime config databases and spreadsheets

  • Several workflow/push/reconciliation engines

  • Autoscaling and resource-planning tools

Note that forking/branching generally isn’t viable in Google’s monorepo.

Despite many projects over the years, some of which have been very widely used, the problem is still considered to be not solved satisfactorily. Our experiences with these tools have informed this proposal, however, as well as the design of Kubernetes itself.

A non-exhaustive list of tools built by the Kubernetes community (see spreadsheet for up-to-date list), in no particular order, follows:

Additionally, a number of continuous deployment systems use their own formats and/or schemas.

The number of tools is a signal of demand for a customization solution, as well as lack of awareness of and/or dissatisfaction with existing tools. Many prefer to use the simplest tool that meets their needs. Most of these tools support customization via simple parameter substitution or a more complex configuration domain-specific language, while not adequately supporting the other customization strategies. The pitfalls of parameterization and domain-specific languages are discussed below.

Parameterization pitfalls

After simply forking (or just cut&paste), parameterization is the most commonly used customization approach. We have previously discussed requirements for parameterization mechanisms, such as explicit declaration of parameters for easy discovery, documentation, and validation (e.g., for form generation). It should also be straightforward to provide multiple sets of parameter values in support of variants and to manage them under version control, though many tools do not facilitate that.

Some existing template examples:

Parameterization solutions are easy to implement and to use at small scale, but parameterized templates tend to become complex and difficult to maintain. Syntax-oblivious macro substitution (e.g., sed, jinja, envsubst) can be fragile, and parameter substitution sites generally have to be identified manually, which is tedious and error-prone, especially for the most common use cases, such as resource name prefixing.

Additionally, performing all customization via template parameters erodes template encapsulation. Some prior configuration-language design efforts made encapsulation a non-goal due to the widespread desire of users to override arbitrary parts of configurations. If used by enough people, someone will want to override each value in a template. Parameterizing every value in a template creates an alternative API schema that contains an out-of-date subset of the full API, and when every value is a parameter, a template combined with its parameters is considerably less readable than the expanded result, and less friendly to data-manipulation scripts and tools.

Pitfalls of configuration domain-specific languages (DSLs)

Since parameterization and file imports are common features of most configuration domain-specific languages (DSLs), they inherit the pitfalls of parameterization. The complex custom syntax (and/or libraries) of more sophisticated languages also tends to be more opaque, hiding information such as application topology from humans. Users generally need to understand the input language, transformations applied, and output generated, which is more complex for users to learn. Furthermore, custom-built languages typically lack good tools for refactoring, validation, testing, debugging, etc., and hard-coded translations are hard to maintain and keep up to date. And such syntax typically isn’t friendly to tools, for example hiding information about parameters and source dependencies, and is hostile to composition with other tools, configuration sources, configuration languages, runtime automation, and so on. The configuration source must be modified in order to customize additional properties or to add additional resources, which fosters closed, monolithic, fat configuration ecosystems and obstructs separation of concerns. This is especially true of tools and libraries that don’t facilitate post-processing of their output between pre-processing the DSL and actuation of the resulting API resources.

Additionally, the more powerful languages make it easy for users to shoot themselves in their feet. For instance, it can be easy to mix computation and data. Among other problems, embedded code renders the configuration unparsable by other tools (e.g., extraction, injection, manipulation, validation, diff, interpretation, reconciliation, conversion) and clients. Such languages also make it easy to reduce boilerplate, which can be useful, but when taken to the extreme, impairs readability and maintainability. Nested/inherited templates are seductive, for those languages that enable them, but very hard to make reusable and maintainable in practice. Finally, it can be tempting to use these capabilities for many purposes, such as changing defaults or introducing new abstractions, but this can create different and surprising behavior compared to direct API usage through CLIs, libraries, UIs, etc., and create accidental pseudo-APIs rather than intentional, actual APIs. If common needs can only be addressed using the configuration language, then the configuration transformer must be invoked by most clients, as opposed to using the API directly, which is contrary to the design of Kubernetes as an API-centric system.

Such languages are powerful and can perform complex transformations, but we found that to be a mixed blessing within Google. For instance, there have been many cases where users needed to generate configuration, manipulate configuration, backport altered API field settings into templates, integrate some kind of dynamic automation with declarative configuration, and so on. All of these scenarios were painful to implement with DSL templates in the way. Templates also created new abstractions, changed API default values, and diverged from the API in other ways that disoriented new users.

A few DSLs are in use in the Kubernetes community, including Go templates (used by Helm, discussed more below), fluent DSLs, and jsonnet, which was inspired by Google’s Borg configuration language (more on its root language, GCL). Ksonnet-lib is a community project aimed at building Kubernetes-specific jsonnet libraries. Unfortunately, the examples (e.g., nginx) appear more complex than the raw Kubernetes API YAML, so while it may provide more expressive power, it is less approachable. Databricks looks like the biggest success case with jsonnet to date, and uses an approach that is admittedly more readable than ksonnet-lib, as is Kubecfg. However, they all encourage users to author and manipulate configuration code written in a DSL rather than configuration data written in a familiar and easily manipulated format, and are unnecessarily complex for most use cases.

Helm is discussed below, with package management.

In case it’s not clear from the above, I do not consider configuration schemas expressed using common data formats such as JSON and YAML (sans use of substitution syntax) to be configuration DSLs.

Configuration using REST API resource specifications

Given the pitfalls of parameterization and configuration DSLs, as mentioned at the beginning of this document, configuration tooling should manipulate configuration data, not convert configuration to code nor other marked-up syntax, and, in the case of Kubernetes, this data should primarily contain specifications of the literal Kubernetes API resources required to deploy the application in the manner desired by the user. The Kubernetes API and CLI (kubectl) were designed to support this model, and our documentation and examples use this approach.

Kubernetes’s API provides IaaS-like container-centric primitives such as Pods, Services, and Ingress, and also lifecycle controllers to support orchestration (self-healing, scaling, updates, termination) of common types of workloads, such as ReplicaSet (simple fungible/stateless app manager), Deployment (orchestrates updates of stateless apps), Job (batch), CronJob (cron), DaemonSet (cluster services), StatefulSet (stateful apps), and custom third-party controllers/operators. The workload controllers, such as Deployment, support declarative upgrades using production-grade strategies such as rolling update, so that the client doesn’t need to perform complex orchestration in the common case. (And we’re moving proven kubectl features to controllers, generally.) We also deliberately decoupled service naming/discovery and load balancing from application implementation in order to maximize deployment flexibility, which should be preserved by the configuration mechanism.

Kubectl apply was designed (original proposal) to support declarative updates without clobbering operationally and/or automatically set desired state. Properties not explicitly specified by the user are free to be changed by automated and other out-of-band mechanisms. Apply is implemented as a 3-way merge of the user’s previous configuration, the new configuration, and the live state.

We chose this simple approach of using literal API resource specifications for the following reasons:

  • KISS: It was simple and natural, given that we designed the API to support CRUD on declarative primitives, and Kubernetes uses the API representation in all scenarios where API resources need to be serialized (e.g., in persistent cluster storage).
  • It didn’t require users to learn multiple different schemas, the API and another configuration format. We believe many/most production users will eventually want to use the API, and knowledge of the API transfers to other clients and tools. It doesn’t obfuscate the API, which is relatively easy to read.
  • It automatically stays up to date with the API, automatically supports all Kubernetes resources, versions, extensions, etc., and can be automatically converted to new API versions.
  • It could share mechanisms with other clients (e.g., Swagger/OpenAPI, which is used for schema validation), which are now supported in several languages: Go, Python, Java, …
  • Declarative configuration is only one interface to the system. There are also CLIs (e.g., kubectl), UIs (e.g., dashboard), mobile apps, chat bots, controllers, admission controllers, Operators, deployment pipelines, etc. Those clients will (and should) target the API. The user will need to interact with the system in terms of the API in these other scenarios.
  • The API serves as a well defined intermediate representation, pre- and post-creation, with a documented deprecation policy. Tools, libraries, controllers, UI wizards, etc. can be built on top, leaving room for exploration and innovation within the community. Example API-based transformations include:
    • Overlay application: kubectl patch
    • Generic resource tooling: kubectl label, kubectl annotate
    • Common-case tooling: kubectl set image, kubectl set resources
    • Dynamic pod transformations: LimitRange, PodSecurityPolicy, PodPreset
    • Admission controllers and initializers
    • API-based controllers, higher-level APIs, and controllers driven by custom resources
    • Automation: horizontal and vertical pod autoscaling
  • It is inherently composable: just add more resource manifests, in the same file or another file. No embedded imports required.

Of course, there are downsides to the approach:

  • Users need to learn some API schema details, though we believe operators will want to learn them, anyway.
  • The API schema does contain a fair bit of boilerplate, though it could be auto-generated and generally increases clarity.
  • The API introduces a significant number of concepts, though they exist for good reasons.
  • The API has no direct representation of common generation steps (e.g., generation of ConfigMap or Secret resources from source data), though these can be described in a declarative format using API conventions, as we do with component configuration in Kubernetes.
  • It is harder to fix warts in the API than to paper over them. Fixing "bugs" may break compatibility (e.g., as with changing the default imagePullPolicy). However, the API is versioned, so it is not impossible, and fixing the API benefits all clients, tools, UIs, etc.
  • JSON is cumbersome and some users find YAML to be error-prone to write. It would also be nice to support a less error-prone data syntax than YAML, such as Relaxed JSON, HJson, HCL, StrictYAML, or YAML2. However, one major disadvantage would be the lack of library support in multiple languages. HCL also wouldn’t directly map to our API schema due to our avoidance of maps. Perhaps there are there YAML conventions that could result in less error-prone specifications.

What needs to be improved?

While the basic mechanisms for this approach are in place, a number of common use cases could be made easier. Most user complaints are around discovering what features exist (especially annotations), documentation of and examples using those features, generating/finding skeleton resource specifications (including boilerplate and commonly needed features), formatting and validation of resource specifications, and determining appropriate cpu and memory resource requests and limits. Specific user scenarios are discussed below.

Bespoke application deployment

Deployment of bespoke applications involves multiple steps:

  1. Build the container image
  2. Generate and/or modify Kubernetes API resource specifications to use the new image
  3. Reconcile those resources with a Kubernetes cluster

Step 1, building the image, is out of scope for Kubernetes. Step 3 is covered by kubectl apply. Some tools in the ecosystem, such as Draft, combine the 3 steps.

Kubectl contains "generator" commands, such as kubectl run, expose, various create commands, to create commonly needed Kubernetes resource configurations. However, they also don’t help users understand current best practices and conventions, such as proper label and annotation usage. This is partly a matter of updating them and partly one of making the generated resources suitable for consumption by new users. Options supporting declarative output, such as dry run, local, export, etc., don’t currently produce clean, readable, reusable resource specifications (example). We should clean them up.

Openshift provides a tool, oc new-app, that can pull source-code templates, detect application types and create Kubernetes resources for applications from source and from container images. podex was built to extract basic information from an image to facilitate creation of default Kubernetes resources, but hasn’t been kept up to date. Similar resource generation tools would be useful for getting started, and even just validating that the image really exists would reduce user error.

For updating the image in an existing deployment, kubectl set image works both on the live state and locally. However, we should make the image optional in controllers so that the image could be updated independently of kubectl apply, if desired. And, we need to automate image tag-to-digest translation (original issue), which is the approach we’d expect users to use in production, as opposed to just immediately re-pulling the new image and restarting all existing containers simultaneously. We should keep the original tag in an imageStream annotation, which could eventually become a field.

Continuous deployment

In addition to PaaSes, such as Openshift and Deis Workflow, numerous continuous deployment systems have been integrated with Kubernetes, such as Google Container Builder, Jenkins, Gitlab, Wercker, Drone, Kit, Bitbucket Pipelines, Codeship, Shippable, SemaphoreCI, Appscode, Kontinuous, ContinuousPipe, CodeFresh, CloudMunch, Distelli, AppLariat, Weave Flux, and Argo. Developers usually favor simplicity, whereas operators have more requirements, such as multi-stage deployment pipelines, deployment environment management (e.g., staging and production), and canary analysis. In either case, users need to be able to deploy both updated images and configuration updates, ideally using the same workflow. Weave Flux and Kube-applier support unified continuous deployment of this style. In other CD systems a unified flow may be achievable by making the image deployment step perform a local kubectl set image (or equivalent) and commit the change to the configuration, and then use another build/deployment trigger on the configuration repository to invoke kubectl apply --prune.

Migrating from Docker Compose

Some developers like Docker’s Compose format as a simplified all-in-one configuration schema, or are at least already familiar with it. Kubernetes supports the format using the Kompose tool, which provides an easy migration path for these developers by translating the format to Kubernetes resource specifications.

The Compose format, even with extensions (e.g., replica counts, pod groupings, controller types), is inherently much more limited in expressivity than Kubernetes-native resource specifications, so users would not want to use it forever in production. But it provides a useful onramp, without introducing yet another schema to the community. We could potentially increase usage by including it in a client-tool release bundle.

Reconciliation of multiple resources and multiple files

Most applications require multiple Kubernetes resources. Although kubectl supports multiple resources in a single file, most users store the resource specifications using one resource per file, for a number of reasons:

  • It was the approach used by all of our early application-stack examples
  • It provides more control by making it easier to specify which resources to operate on
  • It’s inherently composable -- just add more files

The control issue should be addressed by adding support to select resources to mutate by label selector, name, and resource types, which has been planned from the beginning but hasn’t yet been fully implemented. However, we should also expand and improve kubectl’s support for input from multiple files.

Declarative updates

Kubectl apply (and strategic merge patch, upon which apply is built) has a number of bugs and shortcomings, which we are fixing, since it is the underpinning of many things (declarative configuration, add-on management, controller diffs). Eventually we need true API support for apply so that clients can simply PUT their resource manifests and it can be used as the fundamental primitive for declarative updates for all clients. One of the trickier issues we should address with apply is how to handle controller selector changes. We are likely to forbid changes for now, as we do with resource name changes.

Kubectl should also operate on resources in an intelligent order when presented with multiple resources. While we’ve tried to avoid creation-order dependencies, they do exist in a few places, such as with namespaces, custom resource definitions, and ownerReferences.

ConfigMap and Secret updates

We need a declarative syntax for regenerating Secrets and ConfigMaps from their source files that could be used with apply, and provide easier ways to roll out new ConfigMaps and garbage collect unneeded ones. This could be embedded in a manifest file, which we need for "package" metadata (see Addon manager proposal and Helm chart.yaml). There also needs to be an easier way to generate names of the new resources and to update references to ConfigMaps and Secrets, such as in env and volumes. This could be done via new kubectl set commands, but users primarily need the “stream” update model, as with images.

Determining success/failure

The declarative, asynchronous control-loop-based approach makes it more challenging for the user to determine whether the change they made succeeded or failed, or the system is still converging towards the new desired state. Enough status information needs to be reported such that progress and problems are visible to controllers watching the status, and the status needs to be reported in a consistent enough way that a general-purpose mechanism can be built that works for arbitrary API types following Kubernetes API conventions. Third-party attempts to monitor the status generally are not implemented correctly, since Kubernetes’s extensible API model requires exposing distributed-system effects to clients. This complexity can be seen all over our end-to-end tests, which have been made robust over many thousands of executions. Definitely authors of individual application configurations should not be forced to figure out how to implement such checks, as they currently do in Helm charts (--wait, test).

Configuration customization

The strategy for customization involves the following main approaches:

  1. Fork or simply copy the resource specifications, and then locally modify them, imperatively, declaratively, or manually, in order to reuse off-the-shelf configuration. To facilitate these modifications, we should:
    • Automate common customizations, especially name prefixing and label injection (including selectors, pod template labels, and object references), which would address the most common substitutions in existing templates
    • Fix rough edges for local mutation via kubectl get --export and kubectl set (--dry-run, --local, -o yaml), and enable kubectl to directly update files on disk
    • Build fork/branch management tooling for common workflows, such as branch creation, cherrypicking (e.g., to copy configuration changes from a staging to production branch), rebasing, etc., perhaps as a plugin to kubectl.
    • Build/improve structural diff, conflict detection, validation (e.g., kubeval, ConfigMap element properties), and comprehension tools
  2. Resource overlays, for instantiating multiple variants. Kubectl patch already works locally using strategic merge patch, so the overlays have the same structure as the base resources. The main feature needed to facilitate that is automatic pairing of overlays with the resources they should patch.

Fork provides one-time customization, which is the most common case. Overlay patches provide deploy-time customization. These techniques can be combined with dynamic customization (PodPreset, other admission controllers, third-party controllers, etc.) and run-time customization (initContainers and entrypoint.sh scripts inside containers).

Benefits of these approaches:

  • Easier for app developers and operators to build initial configurations (no special template syntax)
  • Compatible with existing project tooling and conventions, and easy to read since it doesn’t obfuscate the API and doesn’t force users to learn a new way to configure their applications
  • Supports best practices
  • Handles cases the original configuration author didn’t envision
  • Handles cases where original author changes things that break existing users
  • Supports composition by adding resources: secrets, configmaps, autoscaling
  • Supports injection of operational concerns, such as node affinity/anti-affinity and tolerations
  • Supports selection among alternatives, and multiple simultaneous versions
  • Supports canaries and multi-cluster deployment
  • Usable for add-on management, by avoiding obstacles that Helm has, and should eliminate the need for the EnsureExists behavior

What about parameterization?

An area where more investigation is needed is explicit inline parameter substitution, which, while overused and should be rendered unnecessary by the capabilities described above, is frequently requested and has been reinvented many times by the community.

A simple parameterization approach derived from Openshift’s design was approved because it was constrained in functionality and solved other problems (e.g., instantiation of resource variants by other controllers, project templates in Openshift). That proposal explains some of the reasoning behind the design tradeoffs, as well as the use cases. Work started, but was abandoned, though there is an independent client-based implementation. However, the Template resource wrapped the resource specifications in another object, which is suboptimal, since transformations would then need to be able to deal with standalone resources, Lists of resources, and Templates, or would need to be applied post-instantiation, and it couldn’t be represented using multiple files, as users prefer.

What is more problematic is that our client libraries, schema validators, yaml/json parsers/decoders, initializers, and protobuf encodings all require that all specified fields have valid values, so parameters cannot currently be left in non-string (e.g., int, bool) fields in actual resources. Additionally, the API server requires at least complete/final resource names to be specified, and strategic merge also requires all merge keys to be specified. Therefore, some amount of pre-instantiation (though not necessarily client-side) transformation is necessary to create valid resources, and we may want to explicitly store the output, or the fields should just contain the default values initially. Parameterized fields could be automatically converted to patches to produce valid resources. Such a transformation could be made reversible, unlike traditional substitution approaches, since the patches could be preserved (e.g., using annotations). The Template API supported the declaration of parameter names, display names, descriptions, default values, required/optional, and types (string, int, bool, base64), and both string and raw json substitutions. If we were to update that specification, we could use the same mechanism for both parameter validation and ConfigMap validation, so that the same mechanism could be used for env substitution and substitution of values of other fields. As mentioned in the env validation issue, we should consider a subset of JSON schema, which we’ll probably use for CRD. The only unsupported attribute appears to be the display name, which is non-critical. Base64 could be represented using media. That could be useful as a common parameter schema to facilitate parameter discovery and documentation that is independent of the substitution syntax and mechanism (example from Deployment Manager).

Without parameters how would we support a click-to-deploy experience? People who are kicking the tires, have undemanding use cases, are learning, etc. are unlikely to know what customization they want to perform initially, if they even need any. The main information users need to provide is the name prefix they want to apply. Otherwise, choosing among a few alternatives would suit their needs better than parameters. The overlay approach should support that pretty well. Beyond that, I suggest kicking users over to a Kubernetes-specific configuration wizard or schema-aware IDE, and/or support a fork workflow.

The other application-definition use cases mentioned in the Template proposal are achievable without parameterization, as well.

What about application configuration generation?

A number of legacy applications have configuration mechanisms that couple application options and information about the deployment environment. In such cases, a ConfigMap containing the configuration data is not sufficient, since the runtime information (e.g., identities, secrets, service addresses) must be incorporated. There are a number of tools used for this purpose outside Kubernetes. However, in Kubernetes, they would have to be run as Pod initContainers, sidecar containers, or container entrypoint.sh init scripts. As this is only a need of some legacy applications, we should not complicate Kubernetes itself to solve it. Instead, we should be prepared to recommend a third-party tool, or provide one, and ensure the downward API provides the information it would need.

What about package management and Helm?

Helm, KPM, App Registry, Kubepack, and DCOS (for Mesos) bundle whitebox off-the-shelf application configurations into packages. However, unlike traditional artifact repositories, which store and serve generated build artifacts, configurations are primarily human-authored. As mentioned above, it is industry best practice to manage such configurations using version control systems, and Helm package repositories are backed by source code repositories. (Example: MariaDB.)

Advantages of packages:

  1. Package formats add structure to raw Kubernetes primitives, which are deliberately flexible and freeform
    • Starter resource specifications that illustrate API schema and best practices
    • Labels for application topology (e.g., app, role, tier, track, env) -- similar to the goals of Label Schema
    • File organization and manifest (list of files), to make it easier for users to navigate larger collections of application specifications, to reduce the need for tooling to search for information, and to facilitate segregation of resources from other artifacts (e.g., container sources)
    • Application metadata: name, authors, description, icon, version, source(s), etc.
    • Application lifecycle operations: build, test, debug, up, upgrade, down, etc.
  2. Package registries/repositories facilitate discovery of off-the-shelf applications and of their dependencies
    • Scattered source repos are hard to find
    • Ideally it would be possible to map the format type to a container containing the tool that understands the format.

Helm is probably the most-used configuration tool other than kubectl, many application charts have been developed (as with the Openshift template library), and there is an ecosystem growing around it (e.g., chartify, helmfile, landscaper, draughtsman, chartmuseum). Helm’s users like the familiar analogy to package management and the structure that it provides. However, while Helm is useful and is the most comprehensive tool, it isn’t suitable for all use cases, such as add-on management. The biggest obstacle is that its non-Kubernetes-compatible API and DSL syntax push it out of Kubernetes proper into the Kubernetes ecosystem. And, as much as Helm is targeting only Kubernetes, it takes little advantage of that. Additionally, scenarios we’d like to support better include chart authoring (prefer simpler syntax and more straightforward management under version control), operational customization (e.g., via scripting, forking, or patching/injection), deployment pipelines (e.g., canaries), multi-cluster / multi-environment deployment, and multi-tenancy.

Helm provides functionality covering several areas:

  • Package conventions: metadata (e.g., name, version, descriptions, icons; Openshift has something similar), labels, file organization
  • Package bundling, unbundling, and hosting
  • Package discovery: search and browse
  • Dependency management
  • Application lifecycle management framework: build, install, uninstall, upgrade, test, etc.
    • a non-container-centric example of that would be ElasticBox
  • Kubernetes drivers for creation, update, deletion, etc.
  • Template expansion / schema transformation
  • (It’s currently lacking a formal parameter schema.)

It's useful for Helm to provide an integrated framework, but the independent functions could be decoupled, and re-bundled into multiple separate tools:

  • Package management -- search, browse, bundle, push, and pull of off-the-shelf application packages and their dependencies.
  • Application lifecycle management -- install, delete, upgrade, rollback -- and pre- and post- hooks for each of those lifecycle transitions, and success/failure tests.
  • Configuration customization via parameter substitution, aka template expansion, aka rendering.

That would enable the package-like structure and conventions to be used with raw declarative management via kubectl or other tool that linked in its business logic, for the lifecycle management to be used without the template expansion, and the template expansion to be used in declarative workflows without the lifecycle management. Support for both client-only and server-side operation and migration from grpc to Kubernetes API extension mechanisms would further expand the addressable use cases.

(Newer proposal, presented at the Helm Summit.)

What about the service broker?

The Open Service Broker API provides a standardized way to provision and bind to blackbox services. It enables late binding of clients to service providers and enables usage of higher-level application services (e.g., caches, databases, messaging systems, object stores) portably, mitigating lock-in and facilitating hybrid and multi-cloud usage of these services, extending the portability of cloud-native applications running on Kubernetes. The service broker is not intended to be a solution for whitebox applications that require any level of management by the user. That degree of abstraction/encapsulation requires full automation, essentially creating a software appliance (cf. autonomic computing): autoscaling, auto-repair, auto-update, automatic monitoring / logging / alerting integration, etc. Operators, initializers, autoscalers, and other automation may eventually achieve this, and we need to for cluster add-ons and other self-hosted components, but the typical off-the-shelf application template doesn’t achieve that.

What about configurations with high cyclomatic complexity or massive numbers of variants?

Consider more automation, such as autoscaling, self-configuration, etc. to reduce the amount of explicit configuration necessary. One could also write a program in some widely used conventional programming language to generate the resource specifications. It’s more likely to have IDE support, test frameworks, documentation generators, etc. than a DSL. Better yet, create composable transformations, applying the Unix Philosophy. In any case, don’t look for a silver bullet to solve all configuration-related problems. Decouple solutions instead.

What about providing an intentionally restrictive simplified, tailored developer experience to streamline a specific use case, environment, workflow, etc.?

This is essentially a DIY PaaS. Write a configuration generator, either client-side or using CRDs (example). The effort involved to document the format, validate it, test it, etc. is similar to building a new API, but I could imagine someone eventually building a SDK to make that easier.

What about more sophisticated deployment orchestration?

Deployment pipelines, canary deployments, blue-green deployments, dependency-based orchestration, event-driven orchestrations, and workflow-driven orchestration should be able to use the building blocks discussed in this document. AppController and Smith are examples of tools built by the community.

What about UI wizards, IDE integration, application frameworks, etc.?

Representing configuration using the literal API types should facilitate programmatic manipulation of the configuration via user-friendly tools, such as UI wizards (e.g., dashboard and many CD tools, such as Puppet Pipelines) and IDEs (e.g., VSCode, IntelliJ), as well as configuration generation and manipulation by application frameworks (e.g., Spring Cloud).