CUE as the core of Infrastructure as Code

[cueckoo]

Originally opened by @cedricgc in cuelang/cue#431

CUE as a general data configuration language has potential wherever humans define data. The use case we see the most interest though, is with CUE as the language to define software infrastructure (Infrastructure as Code). With the rise of cloud computing, organizations now have to contend with much of their infrastructure as abstract resources represented as data. Kubernetes extends this idea to the application layer with its own resource model. The end result is that cloud-native software organization has a large scale, complex configuration problem as described in my previous essay. CUE has specific properties that are well suited for solving this new challenge:

CUE is designed for large scale configuration

CUE deliberately opts for the graph unification model used in computational linguistics instead of the traditional inheritance model. A language definition can be thought of as a massive configuration spanning hundreds of thousands of lines of code. In the future, we will look at software infrastructure in the same way: a complex, tightly interconnected graph of resources that describes an organization's entire computing environment. In that scenario, the ability to understand a configuration worked on by engineers across a whole company and to safely change a value that modifies thousands of objects in a configuration goes from nice-to-have to essential.

CUE supports first-class code generation and automation

A design goal of CUE is to have code that is straightfoward for humans to write, but is also simple for machines to generate. An important facet of configuration is bridging the gap between data written in many different formats by both humans and machines. This means that CUE can integrate with existing tools and workflows naturally while other tools would have to build complex custom solutions. For example, CUE can generate Kubernetes definitions from Go code and OpenAPI schemas and immediately work with resources directly or build higher level libraries.

CUE's scripting layer is also a critical component for integrating with IaC tools. CUE the language can be the best interface for developers to define data, but scripting is the glue that lets us build full automation workflows. For example, with scripting we can create a workflow that exports CUE configurations as JSON/YAML then push to Kuberenetes through kubectl. A similar flow would export Terraform compatible JSON and pass the rest to Terraform to do the heavy lifting. We have the ability to pull data into CUE through code generation and can push data out through automation. The combination of both allows CUE the language to focus on what it is best at without trying to compete with existing tools.

CUE integrates well with Go, the language of cloud tooling

Go is the language of choice for infrastructure tools such as Kubernetes and Terraform. CUE is also implemented in and exposes a rich client API in Go. This leads to opportunities for tight integration with IaC tools. This ranges from the straightforward, like generating CUE libraries for even third party Terraform plugins (written in Go) to leveraging data constraint features in Kubernetes natively in controllers for policy enforcement, etc. CUE shares many of the same design philosophies as Go and can be natural fit for Go developers who need a good way to handle data constraint problems.

Next Steps

Infrastructure as Code is not something that is core to the CUE language itself, but has the opportunity to be a killer use case that drives adoption and grows the community. CUE offers a great interface to describing complex data for humans and can be the layer of automation that is missing in developer workflows. For CUE, having such a wide reaching use case (anybody using cloud computing or declarative infrastructure can benefit from this) can drive investment in CUE tooling and libraries that are necessary in the long term for healthy adoption.

I want this thread to be a hub point for discussion about we as a community approach this space and techniques we can use, no matter what the particular implementation looks like. There are many ways to tackle this problem and would benefit from early collaboration and knowledge sharing.

[cueckoo]

Original reply by @cedricgc in cuelang/cue#431 (comment)

A more incremental approach would be to double down heavily on existing successful IaC tools, like Terraform and use CUE features to layer value on top. As I described above, CUE is well suited to wrapping existing tools by generating compliant JSON structures and having Terraform handle the rest. I think CUE would be a more powerful option than HCL at scale and we could also build automation to handle code and tool tasks like code linting and state file management. We can add value without having to compete with a tool directly.

[cueckoo]

Original reply by @cedricgc in cuelang/cue#431 (comment)

The idea would be similar for Pulumi, but I think it would look more like CUE as one of the first class supported languages that Pulumi would use as a frontend interface. This would be more reliant and CUE having a stronger value proposition with tooling and libraries to be picked over general purpose languages supported by Pulumi.

[cueckoo]

Original reply by @verdverm in cuelang/cue#431 (comment)

There is a huge use case around a configuration that feeds into both TF / hyper-cloud and Helm / k8s. Then there are all the other tools in the same spaces.

I do like the idea about adding a component to Pulumi and others, rather than saying use this not that. Spot on strategy here. I've been thinking about this with a number of other technologies. The general idea I am pitching to clients is replace their existing yaml/json config with Cue and then just generate the format a particular program wants.

The other thing about rewriting devops tooling as an industry is that there is legacy / entrenched usage. I think this is a great idea in the long run, there is definitely a better UX out there than most of what is provided today. Cue offers a great opportunity to build something that can make all parties happy and support a wide class of backends from an increasingly abstract design space. [ this is the hypothesis going into our product strategy anyways ;]

[cueckoo]

Original reply by @cedricgc in cuelang/cue#431 (comment)

I've been working on essay that describes what a next generation IaC tool would look like. Instead of waiting for the completed essay, I thought I would post some initial notes and get some early feedback and thoughts:

• Next Generation Terraform
  • Current IaC tools (Terraform, Pulumi) combine key functionalities into single offering
    • Owning the developer workflow is an executional moat
      • Creates the opportunities for enterprise features (Terraform Atlas, Pulumi Stacks)
  • K8s controllers can instead manage cloud resources
    • Crossplane for high-fidelity representation of cloud resources
    • Control loops naturally do state diffing and convergence
    • Improvements from current practice
      • Loops are continuous reconcilliation instead of singular operations
        • Self-healing
        • Migrating resource is built-in (full lifecycle over time)
      • Open controllers means arbitrary components can be plugged into the workflow
        • Open Policy Agent to validate infrastructure configuration (policy as code)
        • Security analysis of resource graph
      • Can take advantage of K8s resource model
        • Access to K8s types and labels for managing infra resources too
        • Model the full stack from cloud provider down to application workloads
          • Avoids configuration turtle-stacking
      • Kubernetes designed to work with configuration touched by both humans and machines
        • Allows for multiple sources of inputs for a configuration
        • Gradual migration of data that is managed by humans to managed by systems
          • TODO: Think about this more, could have wider implications
  • Unbundling diff and convergence from UX opens IaC to be usurped by newer UX approaches
    • Possible existential risk
      • Easier to replace enterprise features with K8s components
        • eg. Pulumi Crossguard -> Open Policy Agent
    • IaC could adapt in response instead
      • Terraform and Pulumi building support for defining K8s resources
      • IaC companies have deeper understanding of product space
      • Single toolchain end to end
        • Easier to integrate components
        • Can change faster
    • Opportunity for CUE to be the UX interface for cloud configuration
      • Rising complexity to manage cloud configuration
        • Tie back to Configuration Complexity Curse
      • CUE specific advantages
        • Designed to handle high scale configuration problems
          • Avoids inheritance
        • Designed for code generation and automation
          • Critical for covering K8s + third party API surface
  • Components of IaC
    • User interface (Configuration or Programming Language SDKs)
      • Future
        • CUE
        • Crossplane
        • Challenges
          • Code generation
          • Tool Maturity
    • DAG dependency analysis
      • Future
        • GitOps (?)
        • TODO: Research methods for defining dependencies between K8s resources
          • Native or third party projects
          • Static analysis of resource definitions
        • Challenges
          • Lack of clear tool here
          • May need native K8s support for resource dependencies
    • Infrastructure Diff
      • Future
        • K8s Controllers
        • GitOps
    • Convergence
      • Future
        • K8s Controllers
        • Crossplane
        • Challenges
          • Need to wait for supported controller to use cloud feature
          • Impedance mismatch between cloud and K8s control planes

[cueckoo]

Original reply by @138over in cuelang/cue#431 (comment)

Some quick thoughts

• authoring for DAGs and Flows which are types/values too.
• java/maven world has a concept of an archetype. and archetype maps to a lifecycle, phases, goals. this is a powerful concept, another type/value candidate related to first bullet. can standardize on DAGs, Flows, Lifecycles, and apply variants, and exceptions
• authoring for variants and exceptions
• relocatable, it does not matter where the target is: cloud, service, colo, laptop, device
• peer-to-peer, what git (DVCS) was really designed for... opposite of github, etc...

[cueckoo]

Original reply by @verdverm in cuelang/cue#431 (comment)

@cedricgc just awesome! thank you for spending your time on this

[cueckoo]

Original reply by @jlongtine in cuelang/cue#431 (comment)

@cedricgc There's some really good stuff in here. I'll try to take some time to digest and respond soon (might be after my vacation next week, though).

We've got a pretty slick tool that we've been using at Gloo (gloo.us) for CloudFormation, and it gives tf apply-like semantics but using CFN as the state store, which gives you all the niceties of rollback and such. It's hot.

And I think we're likely to work on a tool for generating Kubernetes manifests from CUE that will then be utilized by kubectl or ArgoCD, etc. for application to a cluster.

This space is pretty exciting, honestly. Lots of good happening here.

Talk soon!

[cueckoo]

Original reply by @cedricgc in cuelang/cue#431 (comment)

I like the idea of using CUE to create a nicer overlay for CFN so you get state management for free. That is pretty clever. I would say the main advantage for Terraform is that we could generate CUE for the 3rd party plugins as well, but the need depends really on the organization. It could also ultimately be covered by K8s CRDs which I was hinting towards in the outline.

[cueckoo]

Original reply by @cedricgc in cuelang/cue#431 (comment)

I was pointed to the Kubernetesx project which has some good lessons for what we are trying to do. I like the focus on composition of resources: it closely follows how CUE can build types without inheritance. One thing I found interesting is that they can build encapsulated methods on those resources to do scoped transformations. For example:

const deployment = new kx.Deployment(...);
const service = deployment.createService();

Here we can define a Deployment and the resulting object has a method to build a Service. This encapsulates logic and provides an ergonomic API that is not available from raw Kubernetes. One of the tradeoffs of the Kubernetes Resource Model is that there is very little encapsulation for the underlying resources. It may be possible to create that encapsulation at the interface level (CUE) instead.