Alternative to YAML for defining workflows

[krzema12]

Problem statement

Currently the only way of defining a workflow is using YAML. It has its drawbacks: lack of typing, no way to extract a common piece of logic, no way to programatically generate a workflow, indent-sensitivity, no auto-complete regarding e.g. actions' inputs, and more. While it works fine with simple workflows, it becomes unmaintainable with more complex ones.

To address it, I came up with an idea to use a general-purpose language to describe the workflows. To leverage the current Actions' API, it could be anything that can produce a YAML. I'm the creator and co-maintainer of GitHub Actions Kotlin DSL library (repo, docs). It works by having Kotlin scripts using the DSL to generate the YAML, and during job runtime the script and the YAML are compared to ensure they are in sync. An example workflow looks like this:

@file:DependsOn("it.krzeminski:github-actions-kotlin-dsl:0.15.0")

import it.krzeminski.githubactions.actions.actions.CheckoutV3
import it.krzeminski.githubactions.actions.actions.SetupJavaV3
import it.krzeminski.githubactions.actions.actions.SetupJavaV3.Distribution.Adopt
import it.krzeminski.githubactions.actions.gradle.GradleBuildActionV2
import it.krzeminski.githubactions.domain.RunnerType.UbuntuLatest
import it.krzeminski.githubactions.domain.triggers.Cron
import it.krzeminski.githubactions.domain.triggers.PullRequest
import it.krzeminski.githubactions.domain.triggers.Push
import it.krzeminski.githubactions.domain.triggers.Schedule
import it.krzeminski.githubactions.dsl.workflow
import java.nio.file.Paths

val checkIfWrappersUpToDateWorkflow = workflow(
    name = "Check if wrappers up to date",
    on = listOf(
        Push(branches = listOf("main")),
        PullRequest(),
        Schedule(triggers = listOf(Cron(hour = "1", minute = "0"))),
    ),
    sourceFile = Paths.get(".github/workflows/_GenerateWorkflows.main.kts"),
    targetFile = Paths.get(".github/workflows/check-if-wrappers-up-to-date.yaml"),
) {
    job(
        id = "check",
        runsOn = UbuntuLatest,
    ) {
        uses(CheckoutV3())
        uses(
            name = "Set up JDK",
            action = SetupJavaV3(
                javaVersion = "11",
                distribution = Adopt,
            )
        )
        uses(
            name = "Generate wrappers",
            action = GradleBuildActionV2(
                arguments = ":wrapper-generator:run",
            )
        )
        run(
            name = "Fail if there are any changes in the generated wrappers or their list in the docs",
            command = "git diff --exit-code library/src/gen/ docs/supported-actions.md"
        )
    }
}

(source: https://github.com/krzema12/github-actions-kotlin-dsl/blob/fa02380f2b3c7a684ff52831a75f7fabf92284c1/.github/workflows/check-if-wrappers-up-to-date.main.kts)

The DSL library generates the following YAML:

# This file was generated using Kotlin DSL (.github/workflows/_GenerateWorkflows.main.kts).
# If you want to modify the workflow, please change the Kotlin file and regenerate this YAML file.
# Generated with https://github.com/krzema12/github-actions-kotlin-dsl

name: Check if wrappers up to date

on:
  push:
    branches:
      - 'main'
  pull_request:
  schedule:
   - cron: '0 1 * * *'

jobs:
  "check_yaml_consistency":
    runs-on: "ubuntu-latest"
    steps:
      - id: step-0
        name: Check out
        uses: actions/checkout@v3
      - id: step-1
        name: Execute script
        run: rm '.github/workflows/check-if-wrappers-up-to-date.yaml' && '.github/workflows/_GenerateWorkflows.main.kts'
      - id: step-2
        name: Consistency check
        run: git diff --exit-code '.github/workflows/check-if-wrappers-up-to-date.yaml'
  "check":
    runs-on: "ubuntu-latest"
    needs:
      - "check_yaml_consistency"
    steps:
      - id: step-0
        uses: actions/checkout@v3
      - id: step-1
        name: Set up JDK
        uses: actions/setup-java@v3
        with:
          java-version: 11
          distribution: adopt
      - id: step-2
        name: Generate wrappers
        uses: gradle/gradle-build-action@v2
        with:
          arguments: :wrapper-generator:run
      - id: step-3
        name: Fail if there are any changes in the generated wrappers or their list in the docs
        run: git diff --exit-code library/src/gen/ docs/supported-actions.md

(source: https://github.com/krzema12/github-actions-kotlin-dsl/blob/fa02380f2b3c7a684ff52831a75f7fabf92284c1/.github/workflows/check-if-wrappers-up-to-date.yaml)

It does fix several issues that one can experience with YAML, but it's not comfortable to integrate with GitHub. Having to maintain both Kotlin scripts and YAMLs introduces friction and complicates the library. It's like one would have to maintain both an application written in Java and the resulting bytecode next to it.

Analogical DSLs could exist in Python, Ruby, JS/TS or whatever language one can think of and find most comfortable. The possibilities this approach opens finally bring fun and flexibility into creating the workflows. Compare it with e.g. Jenkins which adopted Groovy as their pipelines description language.

Proposed approach

From the users' perspective, they wouldn't probably want to be aware there's some YAML under the hood. That's why, to enable GitHub detect scripts that are workflow generators, GitHub could look at .workflow. part in the name, plus the fact that the file is executable. If the conditions are met, treat it as a workflow generator i.e. an executable file that produces the YAML to the standard output:

Now there's a question when the workflow generator script should be executed to produce the YAML. I propose having a separate job created implicitly by GitHub that other jobs would depend on. Regarding workers usage quota, it would count as any other job ran by the user. The job would print out the resulting YAML for debugging purposes:

An alternative approach would be to have a single entry point for generating YAMLs, e.g. .github/workflows/init.sh, where the users could produce the YAMLs to files:

#!/usr/bin/env sh
kotlin build.main.kts
kotlin update-gradle-wrapper.main.kts
# any technology can be used here
node generate-wrappers.js
echo "YAML wrappers updated"

The advantage is that there's no convention on the naming of individual workflow files (.workflow. inside is not necessary), which can be understood as a good or bad thing. The exact API here would also depend on when the workflows are now accessed by GitHub, e.g. to evaluate the triggers.

The bottom line is that such generic API makes it very easy to use virtually any programming language - even if not a full-fledged DSL, then a simple script that would e.g. generate repeated piece of YAML. The API is backward-compatible as the standard YAMLs would still be supported.

Please let me know what you think, and what could be a feasible solution from your point of view.

[jmfayard]

Yes please!

GitHub actions is an incredibly useful service, but it's being hold back by the poor developer experience of writing YAML workflows.
Most people resort do copy-paste programming where they find someone's work doing something hopefully similar to what they want to do, but if they have to do anything custom, they will waste hours parsing the huge documentation, the weirdness of YAML, the very slow edit-compile-run feedback loop.

We need something like a typescript compiler well integrated in the IDE that outputs JavaScript, which is what libraries like github-actions-kotlin-dsl deliver.

And if GitHub makes the small step proposed by krzema12, we can imagine similar libraries popping up in other modern programming languages.

[wardellbagby]

Wow, I tried to solve this problem in a very similar way using Typescript instead of Kotlin.

But huge +1 to making these generators work like first class citizens!

https://github.com/wardellbagby/gh-workflow-gen

[krzema12]

Friendly ping - dear GitHub, is it something you would be interested to introduce?

[airtower-luna]

That said, we do understand the difficulty of writing syntactically correct YAML, and plan to work on experiences (e.g. maybe editor extensions) that will help people do this.

In my opinion the error messages workflows produce for incorrect YAML syntax should be a priority. A thing I frequently see people ask about is something like "systax error on line N", but that line itself is fine – the error is in the object (list/mapping, even nested) starting on that line. I personally think YAML is a great choice, but it's easy to see how such error messages confuse people.

[krzema12]

@juliandunn thanks for your response. Just to clarify, this discussion's aim isn't to request creation of a first-party DSL, but rather to provide convenience API for third parties to provide DSLs and run them in a equally easy way as the YAML-based workflows. YAML could still live under the hood.

[jmfayard]

@juliandunn yes, that could be as simple as GitHub recognizing the convention of having a script at .github/workflows/init.sh is a convention that the user is using a third party DSL to regenerate her YAMLs

[radekn]

@juliandunn I would argue that the current YAML-based configuration actually is a DSL that just sits on top of YAML syntax, and that writing syntactically correct YAML is probably the least important of the problems here (although not trivial either).
The larger problem is that this DSL has some fairly frustrating limitations, like arbitrary restrictions on what you're allowed to abstract and reuse. To give some examples:

• you can extract some repeated steps into reusable workflows and actions, but there seems to be no way to extract just constants that are used in more than one place/file
• there is no sane way of passing complex (array or object) inputs to custom workflows and actions - this makes reusable actions much less useful and developer-friendly than they could be
• you can use functions in expressions, but the built-in ones don't even include basic string manipulation, and you can't define your own functions either

The point is that instead of trying to create a less crippled YAML-based DSL, you might as well let people use the power of a proper programming language (ideally, of their own choice). If done right, it could mean less work for you and less frustration for the users.

EDIT:
I understand that sometimes restricted power is a feature that can prevent problems - for example, doing anything nondeterministic in your workflow generation script, either intentionally or accidentally, would likely be a bad thing. In that case, maybe at least allowing people to use a language like Nix or Dhall, both of which have been designed to be safe in that respect, would be a good middle ground?

[jmfayard]

@juliandunn now that we have clarified that we are not asking to get rid of YAML, but just for a simple mecanism to execute something like github-workflows-kt that will produce a valid YAML workflow, what do you think of the proposal?

[yehonadav-feel]

+1

[PrzemyslawSwiderski]

+1

[lkfortuna]

Hey there, to provide some more feedback from the Actions team, this is a really cool idea, and appreciate the hard work and thought that went into this!

Just to set expectations, we do not have plans to implement this request at this time. It is unlikely we will work on this in the next year. Support for DSL is not on the priority list right now unfortunately.

[krzema12]

Hi @lkfortuna, thanks for the update. Too bad, but I get that you may have other priorities for now.
Would you consider another related feature request: https://github.com/orgs/community/discussions/32054? If GitHub provided a first-party way of specifying the types of inputs and outputs, it would greatly improve type-safety of using the actions, not only via the DSL I maintain, but also I can imagine GitHub performing some checks while editing the YAML or executing the action.

[krzema12]

CircleCI has a feature that lets one achieve something close to what is described in this thread. It's called dynamic configuration: https://circleci.com/docs/dynamic-config/. It still requires some YAML to be present, but then it can generate an arbitrary YAML on the fly and say "this is the actual pipeline" by using a feature called pipeline continuation. Pretty powerful.

[remnantkevin]

Buildkite provides another example of dynamically building pipelines. You can build YAML pipeline steps using whatever language or tools you like, and then upload these steps as the pipeline using buildkite-agent pipeline upload.

At my work, we currently use bash and Python scripts to build the steps, but you could use something more type safe if you wanted.

Dynamically building pipelines is both powerful and potentially confusing because you can't (as easily) point to a specific step in a specific pipeline file and say "that's the exact step that was run".

Buildkite docs:

• https://buildkite.com/docs/pipelines/create-your-own
• https://buildkite.com/docs/pipelines/defining-steps#step-defaults-pipeline-dot-yml-file

[krzema12]

I think it's also possible for GitLab CI: https://infinitelambda.com/dynamic-pipeline-generation-gitlab/