How to do dynamic "uses" for reusable workflow

[guilhermeblanco]

Select Topic Area

Question

Body

My use case:

• Complex pipeline that must be run for each of our several thousand customers
• Centralized into a single pipeline, and each customer repository calls a reusable workflow that sits in a different repository
• Pipeline repository has a branch per environment, with different infrastructure rollouts (ie. development uses an internal load balancer with only one certificate, staging uses an external load balancer, production uses CDN and external load balancer)
• Development happens on the development branch, and infrastructure enhancements get promoted to other environments via merge.
• Pipeline branches are different from each other (it's not the same pipeline)
• Multiple environments per customer

I have researched previous discussions here and all I could find was a centralized, single-branch solution, and using different workflows (ie. customer-deploy-development.yaml, customer-deploy-staging.yaml, etc). While this would solve the pipeline call issue, it would then significantly complicate the pipeline for infrastructure rollout (each branch can be different), bordering unmaintainable.

Another approach would be separating each environment into its own file, tracking a single branch per workflow file. This means we'd have to go over thousands of customer repositories creating the same file, with only the @<environment> differently from each other. This seems counterproductive to us.

Anyway, to solve our use case scenario, we are attempting to do the following:

name: customer-deploy

on:
  push:
    branches:
      - development
      - staging
      - production

jobs:
  deploy:
    uses: "octocat/pipelines/.github/workflows/customer-deploy.yaml@${{ github.ref_name }}"
    with:
      environment_name: ${{ github.ref_name }}
      repository_name: ${{ github.repository }}
      repository_ref_name: ${{ github.ref_name }}
      repository_commit_hash: ${{ github.sha }}
    secrets: inherit

However, we experience this error:

[Invalid workflow file: .github/workflows/site-deploy.yaml#L12](https://github.com/octocat/customer123/actions/runs/0000000000/workflow)
invalid value workflow reference: version cannot have whitespace: ${{ github.ref_name }}

We have attempted multiple variations of this, without success. Even build another job that only computes the string and use it, like this:

...

jobs:
  metadata-pipeline:
    runs-on: self-hosted
    env:
      PIPELINE: "octocat/pipelines/.github/workflows/customer-deploy.yaml"
      RELEASE: ${{ github.ref_name }}
    outputs:
      pipeline: ${{ steps.release.outputs.pipeline }}
    steps:
      - name: Extract release pipeline
        id: pipeline
        run: |
          echo "pipeline=${PIPELINE}@${RELEASE##*/} >> $GITHUB_OUTPUT

  deploy:
    uses: ${{ needs.metadata-pipeline.outputs.pipeline }}
    needs:
      - metadata-pipeline
    with:
      environment_name: ${{ github.ref_name }}
      repository_name: ${{ github.repository }}
      repository_ref_name: ${{ github.ref_name }}
      repository_commit_hash: ${{ github.sha }}
    secrets: inherit

Which also gives me:

[Invalid workflow file: .github/workflows/site-deploy.yaml#L27](https://github.com/octocat/customer123/actions/runs/0000000000/workflow)
invalid value workflow reference: no version specified

How do we achieve this?

[airtower-luna]

How do we achieve this?

Unfortunately the answer is "you don't". The value for uses must be a literal. I'm not aware of any location in the docs that explicitly says so, but I've seen GitHub staff confirm it on this forum, with the rationale that it's needed to enforce restrictions on which actions/workflows can be called.

[ChristopherHX]

Do you have a finite amount of possible environments? If no, you can skip reading this.

I would classify this as a workaround, like already mentioned expressions are forbidden in uses.
Create a proxy workflow octocat/pipelines/.github/workflows/customer-deploy.yaml@main, which references each of the possible environments. You only need to update it if you modify inputs and outputs.

on: workflow_call
# TODO add some inputs if you really need them, you example inputs could be removed without loss of information.
jobs:
  development:
    if: github.ref_name == 'development'
    uses: "octocat/pipelines/.github/workflows/customer-deploy.yaml@development"
    with:
      environment_name: ${{ github.ref_name }}
      repository_name: ${{ github.repository }}
      repository_ref_name: ${{ github.ref_name }}
      repository_commit_hash: ${{ github.sha }}
    secrets: inherit
  staging:
    if: github.ref_name == 'staging'
    uses: "octocat/pipelines/.github/workflows/customer-deploy.yaml@staging"
    with:
      environment_name: ${{ github.ref_name }}
      repository_name: ${{ github.repository }}
      repository_ref_name: ${{ github.ref_name }}
      repository_commit_hash: ${{ github.sha }}
    secrets: inherit
  production:
    if: github.ref_name == 'production'
    uses: "octocat/pipelines/.github/workflows/customer-deploy.yaml@production"
    with:
      environment_name: ${{ github.ref_name }}
      repository_name: ${{ github.repository }}
      repository_ref_name: ${{ github.ref_name }}
      repository_commit_hash: ${{ github.sha }}
    secrets: inherit

All consumers just have this file

name: customer-deploy

on:
  push:
    branches:
      - development
      - staging
      - production

jobs:
  deploy:
    uses: "octocat/pipelines/.github/workflows/customer-deploy.yaml@main"
    secrets: inherit

You may take special care about required status checks, the different environments use different job names and skipped jobs are counting as success.

[guilhermeblanco]

Thanks @airtower-luna and @ChristopherHX for the answers, it is greatly appreciated.
I'm accepting @ChristopherHX as the proper answer.

[robert-vrabel-db]

I have a different requirement that requires the same requested solution. We are in the processing of migrating workflows between orgs. The workflow in the new org will continue to be developed but the old workflow is frozen. We planned on storing the org name in a variable that the caller workflow would interrogate to call the right reusable workflow.

prepare:
    name: Prepare environment
    needs: setup
    uses: ${{ needs.setup.outputs.org_name }}/.github/workflows/prepare.yml@main

In this case, the setup job gets the value org value and expects the prepare job to use it to call the right workflow. Subsequent jobs require the output of the prepare step. e.g.

      github_plan_environment: ${{ needs.prepare.outputs.github_plan_environment }}
      github_runner_group: ${{ needs.prepare.outputs.github_runner_group }}

So using the solution suggested above doesn't really work since each job must have a unique name (e.g. development, staging, production). Is the only solution to essentially duplicate the entire caller workflow with unique job names that are conditional based off of the value of the org name?

[paulogodinhoaq]

Sad that this not a possibility, I understand the potential for injecting a foreign workflow, but we could at least be allowed to call workflows that are from the same repo or org.

[AmitArie]

any new?

[FidelusAleksander]

I have a different use case, where I have a separate repository containing reusable workflows and reusable actions. The repository is semanticly versioned. Let's call that the "reusable repository"

Some of the reusable workflows use the actions in that same repository :
Reusable repository .github/workflows/some-workflow.yaml
uses: "reusable_repository/.github/actions/some-action@${{ github.ref_name }}

And in a different repository that is calling the ** reusable_repository/.github/workflows/some-workflow.yaml@v3.5.2** I would like the reusable action "some-action" to be called from the same ref_name

[ChristopherHX]

The problem is even worse: ${{ github.ref_name }} is not the reference to the reusable workflow ref, while
${{ github.job_workflow_ref }} and ${{ github.job_workflow_sha }} are missing and requires us to request a openid connect token (which in turn doesn't work in pull_request events) to be able to use local actions as workaround.
See https://github.com/orgs/community/discussions/31054

[Piedone]

We have the same use case, and AFAIK (in contrast to referencing actions/workflows in the same repository where they're executed from) in this case you have to specify the full path, including the organization and repo name. I.e. in "reusable repository" you have to specify the full {owner}/{repo}/.github/{actions or workflows}/{filename}@{ref}, not just ./.github/{actions or workflows}/{filename}, if the workflow or action containing this line is called from a different repo.

In our own such "reusable repository" we've created a workflow that warns you if you forget to adjust the references, see here: https://github.com/Lombiq/GitHub-Actions/blob/dev/.github/workflows/validate-this-gha-refs.yml

[soumyamahunt]

While dynamic uses is not directly supported, you can achieve what you want by nesting the remote reusable workflow inside another reusable workflow that is path based. Just before running your path based workflow job is run, you can create the workflow.yml file with required data and have it take advantage of pseudo dynamic uses. For your example you can add a prepare job that prepares your dynamic workflow file:

prepare:
  runs-on: ubuntu-latest
  env:
    DYNAMIC_WORKFLOW: .github/workflows/dynamic.yml
  steps:
  - name: Setup dynamic workflow
    uses: actions/github-script@v6
    with:
      script: |
        const path = require('path');
        const fs = require('fs/promises');
        const workflow = {
          name: "Dynamic workflow",
          on: {
            workflow_call: {}
          },
          jobs: {
            actual-deploy: {
              uses: "octocat/pipelines/.github/workflows/customer-deploy.yaml@${{ github.ref_name }}",
              with: {
                environment_name: "${{ github.ref_name }}",
                repository_name: "${{ github.repository }}",
                repository_ref_name: "${{ github.ref_name }}",
                repository_commit_hash: "${{ github.sha }}"
              },
              secrets: "inherit"
            }
          }
        };
        await fs.writeFile(${{ env.DYNAMIC_WORKFLOW }}, JSON.stringify(workflow));

and then invoke the file in the next job, note that this job must start after prepare job is finished:

deploy:
  needs: prepare
  uses: ".github/workflows/dynamic.yml"
  secrets: inherit

Note that there is limitation on level of nesting as described here, if you are not exceeding that you can use this approach.

[wmiller112]

This seems like a great workaround but I can't get it working. I get an error that the uses file doesnt exist from the second job, and if I put a placeholder workflow with the same name, it just uses that placeholder file and not the generated workflow from the prepare job. I could see how it would be possible for composite actions (as demonstrated by this action) since the files persist within the job

[1oglop1]

@soumyamahunt you may be misunderstanding how GitHub actions work
please see the documentation
https://docs.github.com/en/actions/learn-github-actions/understanding-github-actions

Each job will run inside its own virtual machine runner, or inside a container

Therefore what you have described cannot work because

1. workflow jobs do not share the same storage, so it would require to commit a file .github/workflows/dynamic.yml
2. your prepare job is missing checkout action - hence there's "no storage"
3. there's a syntax error actual-deploy is not valid JS "actual-deploy"

https://docs.github.com/en/actions/using-workflows/reusing-workflows

[prein]

If triggering using workflow_dispatch instead of workflow_call would work for your case, then consider this workaround:

  deploy:
    runs-on: ubuntu-latest
    steps:
      - name: Dispatch workflow
        uses: actions/github-script@v6
        with:
          github-token: ${{ github.token }}
          script: |
            await github.rest.actions.createWorkflowDispatch({
              owner: 'octocat',
              repo: 'pipelines',
              workflow_id: 'customer-deploy.yaml',
              ref: '${{ github.ref_name }}'
            })

[GiorgosT97]

What i do not understand from not having uses keyword be dynamic is how to test a reusable workflow.

If i have on my MAIN repo an option to choose a version as an input from my Workflows repo, I could check a PR to see if it's working or compatibility with previous versions etc.

Now it seems I can achieve something like this, only with reusable actions where you can parametrize uses (?)

[gavinclarkeuk]

Been struggling with the same issue, it gets worse when you consider nested calls to composite actions. I think the suggestion above could work if you did all your testing on a named branch (e.g. test on main, then release to version tag), but there is no getting away from the fact that the duplication would makes it massively error prone.