Dynamic Buildkite Pipelines with CUE #21

jpluscplusm · 2023-11-17T13:59:47Z

Buildkite is a CI SaaS with a (current) focus on bring-your-own-compute.

A recent post on their blog is titled "The power of Dynamic Pipelines", and it explains why they believe using more than static pipelines is a good thing.

However, almost all of the blog post is concerned with the "why" (dynamic pipelines are good) and the "how" (to get started with Buildkite from scratch) - the "what should dynamic pipeline generation look like?" question is left almost untouched.

CUE can help here. It's well situated at exactly the intersection of what Buildkite users need, and what Buildkite themselves are un-opinionated about.

A guide to replacing the blog's (to be very fair to the author!) uncomplicated shell script example with a CUE equivalent that improves on every conceivable axis would be a great cue-by-example guide.

Here's a new piece of content, showing how to manage a set of Buildkite pipeline files in CUE instead of YAML. Because Buildkite (in its non-dynamic-pipeline mode) still needs to see a YAML file serialised in the repo, the guide includes a CUE _tool that turns the CUE back into YAML on demand. It also includes a schema for the pipeline's representation, but this schema is very trivial, and not hugely useful - it could do with improving. I wasn't able to use the upstream schema (https://github.com/buildkite/pipeline-schema) because of issues now tracked as cue-lang/cue#2698 and cue-lang/cue#2699. Closes cue-labs#21

Here's a new cue-by-example, showing how to manage a set of Buildkite pipeline files in CUE instead of YAML. Because Buildkite (in its non-dynamic-pipeline mode) still needs to see a YAML file serialised in the repo, the guide includes a CUE _tool that turns the CUE back into YAML on demand. It also includes a schema for the pipeline's representation, but this schema is very trivial, and not hugely useful - it could do with improving. I wasn't able to use the upstream schema (https://github.com/buildkite/pipeline-schema) because of issues now tracked as cue-lang/cue#2698 and cue-lang/cue#2699. Closes cue-labs#21

Here's a new cue-by-example, showing how to manage a set of Buildkite pipeline files in CUE instead of YAML. Because Buildkite (in its non-dynamic-pipeline mode) still needs to see a YAML file serialised in the repo, the guide includes a CUE _tool that turns the CUE back into YAML on demand. It also includes a schema for the pipeline's representation, but this schema is very trivial, and not hugely useful - it could do with improving. I wasn't able to use the upstream schema (https://github.com/buildkite/pipeline-schema) because of issues now tracked as cue-lang/cue#2698 and cue-lang/cue#2699. Closes cue-labs#21 Signed-off-by: Jonathan Matthews <github@hello.jonathanmatthews.com>

Here's a new cue-by-example, showing how to manage a set of Buildkite pipeline files in CUE instead of YAML. Because Buildkite (in its non-dynamic-pipeline mode) still needs to see a YAML file serialised in the repo, the guide includes a CUE _tool that turns the CUE back into YAML on demand. It also includes a schema for the pipeline's representation, but this schema is very trivial, and not hugely useful - it could do with improving. I wasn't able to use the upstream schema (https://github.com/buildkite/pipeline-schema) because of issues now tracked as cue-lang/cue#2698 and cue-lang/cue#2699. This is related to cue-labs#21 but doesn't /close/ it, as that issue tracks a cue-by-example for Buildkite's "dynamic" pipeline mode, which this guide doesn't address. Signed-off-by: Jonathan Matthews <github@hello.jonathanmatthews.com>

This change adds a guide that takes the reader through the process of making a static & known-good Buildkite pipeline dynamic. It uses the steps contained in a Buildkite blog post to assemble the known-good starting point and place the reader in a state where the guide can inform them, without having to deal with too many branching paths and possibilities. It uses directories and CUE package names that are compatible with the simpler static pipeline import guide proposed in cue-labs#23. The two guides haven't been explicitly designed to work together, apart from those naming concerns - but nothing else stands out as being obviously incompatible. Closes cue-labs#21 Signed-off-by: Jonathan Matthews <github@hello.jonathanmatthews.com>

Here's a new cue-by-example, showing how to manage a set of Buildkite pipeline files in CUE instead of YAML. Because Buildkite (in its non-dynamic-pipeline mode) still needs to see a YAML file serialised in the repo, the guide includes a CUE _tool that turns the CUE back into YAML on demand. It also includes a schema for the pipeline's representation, but this schema is very trivial, and not hugely useful - it could do with improving. I wasn't able to use the upstream schema (https://github.com/buildkite/pipeline-schema) because of issues now tracked as cue-lang/cue#2698 and cue-lang/cue#2699. This is related to cue-labs#21 but doesn't /close/ it, as that issue tracks a cue-by-example for Buildkite's "dynamic" pipeline mode, which this guide doesn't address. Signed-off-by: Jonathan Matthews <github@hello.jonathanmatthews.com>

This change adds a guide that takes the reader through the process of making a static & known-good Buildkite pipeline dynamic. It uses the steps contained in a Buildkite blog post to assemble the known-good starting point and place the reader in a state where the guide can inform them, without having to deal with too many branching paths and possibilities. It uses directories and CUE package names that are compatible with the simpler static pipeline import guide proposed in cue-labs#23. The two guides haven't been explicitly designed to work together, apart from those naming concerns - but nothing else stands out as being obviously incompatible. Closes cue-labs#21 Signed-off-by: Jonathan Matthews <github@hello.jonathanmatthews.com>

Here's a new cue-by-example, showing how to manage a set of Buildkite pipeline files in CUE instead of YAML. Because Buildkite (in its non-dynamic-pipeline mode) still needs to see a YAML file serialised in the repo, the guide includes a CUE _tool that turns the CUE back into YAML on demand. It also includes a schema for the pipeline's representation, but this schema is very trivial, and not hugely useful - it could do with improving. I wasn't able to use the upstream schema (https://github.com/buildkite/pipeline-schema) because of issues now tracked as cue-lang/cue#2698 and cue-lang/cue#2699. This is related to #21 but doesn't /close/ it, as that issue tracks a cue-by-example for Buildkite's "dynamic" pipeline mode, which this guide doesn't address. Signed-off-by: Jonathan Matthews <github@hello.jonathanmatthews.com>

This change adds a guide that takes the reader through the process of making a static & known-good Buildkite pipeline dynamic. It uses the steps contained in a Buildkite blog post to assemble the known-good starting point and place the reader in a state where the guide can inform them, without having to deal with too many branching paths and possibilities. It uses directories and CUE package names that are compatible with the simpler static pipeline import guide proposed in cue-labs#23. The two guides haven't been explicitly designed to work together, apart from those naming concerns - but nothing else stands out as being obviously incompatible. Closes cue-labs#21 Signed-off-by: Jonathan Matthews <github@hello.jonathanmatthews.com>

jpluscplusm self-assigned this Nov 20, 2023

jpluscplusm added the content idea Suggestion for a piece of content label Nov 20, 2023

myitcv closed this as completed in f13b670 Nov 29, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Dynamic Buildkite Pipelines with CUE #21

Dynamic Buildkite Pipelines with CUE #21

jpluscplusm commented Nov 17, 2023

Dynamic Buildkite Pipelines with CUE #21

Dynamic Buildkite Pipelines with CUE #21

Comments

jpluscplusm commented Nov 17, 2023