content: supercharging buildkite dynamic pipelines

This change adds a guide that takes the reader through the process of
making a static & known-good Buildkite pipeline dynamic.

It uses the steps contained in a Buildkite blog post to assemble the
known-good starting point and place the reader in a state where the
guide can inform them, without having to deal with too many branching
paths and possibilities.

It uses directories and CUE package names that are compatible with the
simpler static pipeline import guide proposed in
cue-labs#23. The two guides haven't been explicitly
designed to work together, apart from those naming concerns - but
nothing else stands out as being obviously incompatible.

Signed-off-by: Jonathan Matthews <github@hello.jonathanmatthews.com>

WIP_buildkite_dynamic_pipelines/README.md

@@ -0,0 +1,298 @@
+# Supercharging Buildkite dynamic pipelines with CUE
+<sup>by [Jonathan Matthews](https://jonathanmatthews.com/)</sup>
+
+This guide demonstrates how to use CUE to generate dynamic pipelines for
+[the Bring-Your-Own-Compute CI service **Buildkite**](https://buildkite.com).
+
+This guide should be read alongside Buildkite's blog post
+"[The power of Dynamic Pipelines](https://buildkite.com/blog/how-to-build-ci-cd-pipelines-dynamically)",
+as this guide builds on the setup described in the blog post.
+
+Start by reading this guide's [introduction](#introduction) section.
+
+## Introduction
+
+Buildkite offers a way to define CI pipelines as they're initiated and as
+they're executing, which allows the pipeline's steps to be varied dynamically.
+This allows the process that emits the specific pipeline steps to take the
+pipeline's execution context into account.
+
+For example: the Buildkite blog post contains a setup guide (that you'll follow
+shortly) which finishes by creating a dynamic pipeline that includes a deploy
+step *only if it's the master branch*.
+
+Using **this** guide, you'll first follow the Buildkite blog post's instructions
+until the dynamic pipeline needs to be created, and then switch back to this
+guide to create the pipeline using CUE.
+
+### Prerequisites
+
+To use this guide, first make sure:
+
+- you have
+  [CUE installed](https://alpha.cuelang.org/docs/introduction/installation/)
+  locally. This allows you to run `cue` commands
+- you have
+  [`git` installed](https://git-scm.com/downloads)
+  locally. This allow you to make changes to your fork of the Buildkite example
+  repository
+- you have access to a git hosting account. It's probably easiest to use a
+  GitHub account, but any publically accessible git service will work
+- you have access to a Buildkite account. If you don't have one, the Buildkite
+  blog post shows you where to sign up for their Free plan
+
+## Start here
+
+Begin by reading
+[the Buildkite blog post on dynamic pipelines](https://buildkite.com/blog/how-to-build-ci-cd-pipelines-dynamically)
+and follow its instructions *up to but not including* the section titled
+"Including custom steps". **The blog post gives you a choice of a Bash or
+a Powershell example project: please select the Bash example.**
+
+When you reach the blog post's sentence `We don’t yet have a "dynamic" build
+pipeline, or a pipeline that runs a script`, switch back to this guide and
+continue reading from here.
+
+| :grey_exclamation: Info :grey_exclamation: |
+|:------------------------------------------ |
+| If you want to start with one of your own pipelines, and not the pipeline created by following the Buildkite blog post's instructions, then that's fine. Make sure that your pipeline starts in a green state, and you have the ability to change the underlying git repository's contents. You'll have to make adjustments as you follow this guide to adapt its steps to the specifics of your pipeline and repository layout. Continue following this guide at the "[Add some files to the repository](#add-some-files-to-the-repository)" step.
+
+## Continue here after reading the Buildkite blog post
+
+By following the Buildkite blog post's instructions before this section, you
+have successfully executed a static example pipeline and are now ready to make
+the pipeline dynamic.
+
+### Prepare to make changes to the pipeline's contents
+
+#### :arrow_right: Fork the Buildkite example repository
+
+As part of the blog post's instructions, you created a pipeline using the
+Buildkite
+[bash-example repository](https://github.com/buildkite/bash-example.git).
+
+Fork that repository under your own git hosting account and clone the forked
+repository onto your local machine, so that you can make changes to its
+contents. All references to "repository", in this guide, now refer to your
+forked repository.
+
+#### :arrow_right: Update the pipeline to reference your fork
+
+In the Buildkite web UI, find the settings page for the pipeline you created.
+Open the "GitHub" settings tab.
+
+In the "Repository Settings" section, change the "Repository" field so that it
+contains the cloneable URL of your forked repository and click "Save
+Repository".
+
+#### :arrow_right: Test the pipeline's new settings
+
+In the Buildkite web UI, create a new build for the pipeline. Check that it
+still completes successfully, and goes green.
+
+If the updated pipeline doesn't go green you'll need to diagnose and fix this
+before continuing.
+
+### Add some files to the repository
+
+#### :arrow_right: Create a location for your CUE files
+
+At the top-level of your repository, create a directory to hold your Buildkite
+CUE files:
+
+:computer: `terminal`
+```sh
+cd $(git rev-parse --show-toplevel)   # make sure we're sitting at the repository root
+mkdir -p internal/ci/buildkite/
+```
+
+| :grey_exclamation: Info :grey_exclamation: |
+|:------------------------------------------ |
+| You *can* change the `buildkite` directory's *location*, but **don't change its name**.<br>If you change its location, you'll have to adapt some commands to match the new location, as you follow the rest of this guide.
+
+#### :arrow_right: Create CUE tool
+
+:floppy_disk: `internal/ci/buildkite/dynamic_tool.cue`
+```CUE
+package pipeline
+
+import (
+	"encoding/yaml"
+	"tool/os"
+	"tool/cli"
+)
+
+ENV: command.emit_dynamic_steps.env
+dynamic_steps: [...]
+
+command: emit_dynamic_steps: {
+	env:  os.Environ
+	emit: cli.Print & {
+		text: yaml.Marshal({steps: dynamic_steps})
+	}
+}
+```
+
+#### :arrow_right: Create dynamic steps
+
+:floppy_disk: `internal/ci/buildkite/dynamic.steps.cue`
+```
+package pipeline
+
+ENV: {
+	BUILDKITE_AGENT_ENDPOINT: *"test value" | _
+	BUILDKITE_BRANCH:         *"test value" | _
+}
+dynamic_steps: [
+	{command: "echo dynamic step 1"},
+	{command: "echo dynamic step 2"},
+	{command: "echo The agent endpoint in use is \(ENV.BUILDKITE_AGENT_ENDPOINT)"},
+	{command: "echo dynamic step 3"},
+	if ENV.BUILDKITE_BRANCH == "main" {"wait"},
+	if ENV.BUILDKITE_BRANCH == "main" {
+		{
+			command: "echo Deploy!"
+			label:   ":rocket:"
+		}
+	},
+]
+```
+
+These steps are (of course!) just examples. Replace them with steps that meet
+your pipeline's requirements. You can use any CUE features and structure,
+including conditionally using the contents and values of the context-specific
+`ENV` struct, so long as the `dynamic_steps` field ultimately contains an
+ordered list of your steps.
+
+#### :arrow_right: Create CUE schema
+
+:floppy_disk: `internal/ci/buildkite/dynamic.schema.cue`
+```
+package pipeline
+
+ENV: close({[string]: string})
+#Step: _
+dynamic_steps: [...#Step]
+```
+
+|  :grey_exclamation: Info :grey_exclamation: |
+|:------------------------------------------- |
+| It would be great if we could use [Buildkite's authoritative pipeline schema](https://github.com/buildkite/pipeline-schema) here. Unfortunately, CUE's JSONSchema support can't currently import it. This is being tracked in CUE Issues [#2698](https://github.com/cue-lang/cue/issues/2698) and [#2699](https://github.com/cue-lang/cue/issues/2699), and this guide should be updated once the schema is useable.
+
+#### :arrow_right: Create CUE policy
+
+:floppy_disk: `internal/ci/buildkite/dynamic.policy.cue`
+```
+package pipeline
+
+import "encoding/json"
+
+ENV: _
+
+#branchesPermittedToDeploy: "main" | "staging" | "qa"
+
+// if the branch driving the pipeline *isn't* contained in
+// #branchesPermittedToDeploy then insist that all steps with commands *don't*
+// include the word "Deploy" in their command string.
+if json.Marshal(ENV.BUILDKITE_BRANCH & #branchesPermittedToDeploy) == _|_ {
+	#Step: command?: !~"Deploy"
+}
+```
+
+NB This is an *example* policy, using the branch name that's being built to
+decide if the policy applies or not.
+
+**You will need to change this policy to match your pipeline requirements**
+before using this pipeline on a real project!
+
+### Update the pipeline to use CUE
+
+#### :arrow_right: Install CUE on the Buildkite agent's machine
+
+*This guide doesn't yet extend to exposing CUE as a Buildkite plugin, or
+installing it at the start of each pipeline run. Instead, it currently assumes
+that the `cue` binary is available on each machine that runs the Buildkite
+agent.*
+
+[Install CUE](https://alpha.cuelang.org/docs/introduction/installation/) on
+each machine that runs the Buildkite agent and which might pick up this
+pipeline's jobs. Make sure `cue` is available in the Buildkite agent's `PATH`.
+
+#### :arrow_right: Update the pipeline's definition
+
+In the Buildkite web UI, update the pipeline's settings so that its first step
+runs this command:
+
+```
+cue cmd emit_dynamic_steps ./internal/ci/buildkite:pipeline | buildkite-agent pipeline upload
+```
+
+If you followed the Buildkite blog post's instructions, then your pipeline will
+probably have been created in Buildkite's "Legacy Steps" mode. If so, use the
+web UI's editor to change the "Commands to run" field to:
+
+```
+cue cmd emit_dynamic_steps ./internal/ci/buildkite:pipeline | buildkite-agent pipeline upload
+```
+
+Alternatively, if your pipeline is set up in Buildkite's "YAML Steps" mode, you
+can place the following in the "Steps" YAML editor pane, overwriting the steps
+that are already there:
+
+```yaml
+steps:
+- command: cue cmd emit_dynamic_steps ./internal/ci/buildkite:pipeline | buildkite-agent pipeline upload
+```
+
+| :exclamation: WARNING :exclamation: |
+|:----------------------------------- |
+If you are adapting an existing, working, pipeline, there may already be additional YAML keys present other than `steps`.<br>**Do not change these**.
+
+#### :arrow_right: Publish your changes
+
+Use `git` to commit and push the files you've added to the repository. For
+example:
+
+:computer: `terminal`
+```sh
+git add internal/ci/buildkite
+git commit -m "ci: make pipeline dynamic with CUE"
+git push
+```
+
+#### :arrow_right: Test your new pipeline
+
+In the Buildkite web UI, create a new build for the pipeline. Check that it
+still completes successfully, and goes green.
+
+If it doesn't go green, double-check that you copied and set the initial
+pipeline command correctly, in this guide's step headed
+"[:arrow_right: Update the pipeline's definition](#arrow_right-update-the-pipelines-definition)".
+
+You can also use the following command in your local checkout of the repository
+in order to check the contents of the steps which will be passed back to
+Buildkite for execution:
+
+:computer: `terminal`
+```sh
+cue cmd emit_dynamic_steps ./internal/ci/buildkite:pipeline
+```
+
+If your CUE is using any of
+[Buildkite's environment variables](https://buildkite.com/docs/pipelines/environment-variables#buildkite-environment-variables)
+to make decisions about the steps to output then you can set them up
+temporarily, locally, as shown in this example:
+
+:computer: `terminal`
+```sh
+BUILDKITE_BRANCH="main" cue cmd emit_dynamic_steps ./internal/ci/buildkite:pipeline
+```
+
+## Conclusion
+
+Congratulations! You've converted a Buildkite pipeline from being static to
+being defined and driven by CUE - potentially using information that's only
+available at the time of execution to build the pipeline dynamically.
+
+Your use of CUE will increase the safety with which you make pipeline changes,
+by providing a scalable and effective framework for managing complex pipelines.