Reusable workflow/actions name and references

[alonbl]

Hello,

Via trail and error I found out that the workflow engine does not resolve expression in uses statement.

I expected the following to work:

on:
  workflow_call:
    inputs:
      ref: '@v1'
jobs:
  job1:
    runs-on: ubuntu-latest
    steps:
      - uses: repo/action@${{ inputs.ref }}

What I would like to achieve is for production to use the formal tag, while debugging, testing and CI the workflow itself I can use a pre-release reference.

I would like to have a single repository that contains my reusable workflows and reusable actions for use of my repositories. I need somehow for the reusable workflows to know what reference used to fetch them in order to call the actions and nested workflows. I found no way to do that, so I tried to explicitly the reference this as parameter, but this does not work either.

The setup:

   consuming_repository                                            reusable_repository

                                  ---execute reusable RW1 ---> 
                                                                    RW1 ----> RW2
                                                                        ----> RA1
                                                                        ----> RA2

I thought of solving this by using git submodules, however, it is impossible to mix local and reusable as github does not support nested paths for workflows:

invalid value workflow reference: workflows must be defined at the top level of the .github/workflows/ directory

Any ideas how to implement nested reusable in separate repository correctly?

Thanks,
Alon

[ChristopherHX]

Any ideas how to implement nested reusable in separate repository correctly?

You can hack around this limitation by writing an local action to disk and tell the runner to execute it. I made this action to solve an different limitation, but this works too https://github.com/ChristopherHX/conditional/blob/main/action.yml.

on:
  workflow_call:
    inputs:
      ref: 'v1'
jobs:
  job1:
    runs-on: ubuntu-latest
    steps:
    - uses: ChristopherHX/conditional@b4a9649204f81002ec9a4ef7d4bf7d6b2ab7fa55
      with:
        step: |
          uses: repo/action@${{ inputs.ref }}
          with:
            arg: ${{ tojson(inputs.ref) }}

The whole step property is a string, therefore no error is reported. It is easy to create bugs with step inputs

Feel free to copy this little action to your own repo or modify it.

I need somehow for the reusable workflows to know what reference used to fetch

Yes it is possible to get the repository, ref and name of the reusable workflow, see job_workflow_ref claim https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/about-security-hardening-with-openid-connect. However you need to grant the reusable workflow the oidc permission and parse the claims of the jwt. I can try to lookup my sample if you are interested.

It would be so much easier if github exposes job_workflow_ref as ${{ github.job_workflow_ref }}, but I'm dreaming here.

[alonbl]

Thanks @ChristopherHX !

This is one ugly hack!!! It does solves the reusable action with the ability to "dynamically" generate the action based on a template.

Do you have an idea how to do so with callable workflow? Do you recommend a job which creates a workflow file and job that needs that job in order to call to the locally generated workflow? This will add another step of a runner which takes time and resources. Do you have any other idea?

Thank you very much!

[ChristopherHX]

Not everything is possible in a usable way, but Good Luck if you still want to continue.

Do you have an idea how to do so with callable workflow?

No, GitHub seems to parse the whole workflow incl. all referenced reusable / callable workflow before any job runs. So you can't generate any callable workflow within the same workflow run.

Do you recommend a job which creates a workflow file and job that needs that job in order to call to the locally generated workflow?

You probably don't want to do that...
Never tried that, maybe you can commit the workflow file in a new branch and trigger a workflow run with workflow_dispatch / repository_dispatch.

I wrote an GitHub Actions Emulator, which can execute your generated workflow on an GitHub Runner or locally. But this would be only an ugly composite action with multiple jobs on one machine without GitHub Support

[alonbl]

Thanks @ChristopherHX, it is very helpful.

I understand that reusable workflow/actions exists theoretically, but practically using a real reusable scenario is impossible. It is like these who implement the feature do not actually use them, nor understanding the use-case.

Do you have any idea where can I open/find feature request for these to explain GitHub product managers the practical requirements of reusability?

Thanks,
Alon

[ChristopherHX]

Based on my knowlege this is the place where you can open/find feature request. Don't expect fast responses, however some feature request got actually implemented.

[ChristopherHX]

Alternatively use oidc, clone your local actions and use ./path/to/action ( pre steps are not supported ) instead of using reponame and version

Example

caller workflow

name: Test reusable
on:
  push:
jobs:
  _:
    uses: ChristopherHX/newcomposite-sample/.github/workflows/main.yml@local-actions-with-reusable

ChristopherHX/newcomposite-sample/.github/workflows/main.yml@local-actions-with-reusable:

on:
  workflow_call:
jobs:
  test:
    runs-on: ubuntu-latest
    permissions:
      id-token: write # required to get an oidc token
      contents: read # optional permission to allow cloning the caller repository, not shure what happens if the reusable is in an internal repo
    # permissions: write-all # if you don't want to restrict permissions
    steps:
    - uses: ChristopherHX/oidc@45e798774435c73c1bbf62e2df0ba60f95cdb457
      id: oidc
    - uses: actions/checkout@v3 # checks out caller workflow
    - uses: actions/checkout@v3 # checks out called workflow
      with:
        path: .github/actions
        repository: ${{ steps.oidc.outputs.job_workflow_repo_name_and_owner }}
        ref: ${{ steps.oidc.outputs.job_workflow_repo_ref }}
    - run: ls
    - uses: ./.github/actions/dowork
  # nested_workflow:
  #   uses: ./.github/workflows/second.yml # This is resoved in the same repository and version as the current reusable, I tested this, but couldn't find any documentation about this.
 

ChristopherHX/oidc@45e7987:

const core = require("@actions/core");
const jwt = require("jsonwebtoken");
(async function() {
    try {
        var id_token = await core.getIDToken();
        var decoded = jwt.decode(id_token , {complete: true});
        for(var key in decoded.payload) {
            core.setOutput(key, decoded.payload[key]);
        }
        if(decoded.payload.job_workflow_ref) {
            var nameAndRef = decoded.payload.job_workflow_ref.split("@", 2);
            if(nameAndRef.length === 2) {
                var nameAndPath = nameAndRef[0].split("/");
                if(nameAndPath.length >= 3) {
                    core.setOutput("job_workflow_repo_owner", nameAndPath[0]);
                    core.setOutput("job_workflow_repo_name", nameAndPath[1]);
                    core.setOutput("job_workflow_repo_name_and_owner", nameAndPath[0] + "/" + nameAndPath[1]);
                    core.setOutput("job_workflow_repo_path", nameAndPath.splice(2).join("/"));
                }
                core.setOutput("job_workflow_repo_ref", nameAndRef[1]);
            }
        }
    } catch(ex) {
        core.error("Failed to get odic token, did you forget the `permissions: { id-token: write }` permission in your caller workflow, alternatively `permissions: write-all` works too: " + ex.toString());
        process.exit(1);
    }
})()

dowork is a folder in the called workflow https://github.com/ChristopherHX/newcomposite-sample/tree/local-actions-with-reusable and contains an action next to the reusable workflow

See https://github.com/ChristopherHX/Buildbot/runs/8060745717?check_suite_focus=true for an example workflow run.

If this solves your problem then please mark as an answer

Edit You don't have to specify id-token permission on the calling job / workflow, this simplifies usage a lot. I thought this would be permission elevation, but I guess this a weird default handling.

[alonbl]

Hi @ChristopherHX ,

You are a master in github hacks! Thank you so much.

The hint that relative nested workflow is working from the repo of the calling workflow solved the re-usability of the workflow, I will use it even if not documented as I have no choice.

I tried to use environment instead of using the oidc great hack, however, the environment is not inherited by the nested workflow... this is unexpected as well... so I end up with using the oidc as you suggested.

Thank you so much for the valuable help.

I am afraid that if I mark this as resolved, github personal will not be aware of a problem.
Your hacks are great, but it should not be this difficult to use the environment.

Summary:

1. Each workflow/action should be able to obtain information of where it was accessed from so that it may use relative components.
2. It should be possible to use expression in uses clause to build the name of target dynamically.
3. env or generic user context should be inherited by nested workflows so that there will be no requirement to explicitly specify variables to pass into nested workflows/actions.
4. secrets should be accessible to actions, similar to workflows, including secret inheritance, I opened a bug for this.
5. Support accessing workflows/actions from private repositories.
6. Probably support of lazy resolution of workflows, so that a logic can be applied to build the end-result including expression support.

Thanks,
Alon

[ChristopherHX]

I agree, since I failed to use my workaround for the pull_request event trigger without enabling unsecure write and secrets access to all pull_request's from forks.

I don't get why the workflow ref is not in the github context.

[ChristopherHX]

GitHub finally added github.workflow_ref/github.workflow_sha, it is now straight forward to use local actions of the same version as the reusable workflow. This does also work with pull_request events from forks.
If we would have split in expressions, then we don't need a parse step.

Invalid Example

on:
  workflow_call:
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v3 # checks out caller workflow
    - uses: actions/github-script@v6 # Extract nameAndOwner of github.workflow_ref
      id: parseData
      with:
        script: |
          var nameAndRef = process.env.WORKFLOW_REF.split("@", 2);
          if(nameAndRef.length === 2) {
              var nameAndPath = nameAndRef[0].split("/");
              if(nameAndPath.length >= 2) {
                  core.setOutput("nameAndOwner", nameAndPath[0] + "/" + nameAndPath[1]);
                  core.setOutput("path", nameAndPath.splice(2).join("/"));
              }
              core.setOutput("ref", nameAndRef[1]);
          }
      env:
        WORKFLOW_REF: ${{ github.workflow_ref }}
    - uses: actions/checkout@v3 # checks out called workflow
      with:
        path: .github/actions
        repository: ${{ steps.parseData.outputs.nameAndOwner }}
        ref: ${{ github.workflow_sha }}
    - uses: ./.github/actions/dowork # dowork is a folder in the workflow repo

We need github.job_workflow_ref / github.job_workflow_sha.

[roulettedares]

Despite the docs at https://docs.github.com/en/actions/learn-github-actions/contexts#github-context, the property github.job_workflow_sha is still not set in a job using a reusable workflow. I've tried both self-hosted and github-hosted runners. halp!

[yogeshlonkar]

Even if the github.job_workflow_sha value is fixed it still leaves additional steps in each job of reusable workflow, cloning repository and if it's private/internal repository you need PAT to clone it.

I hope GitHub adds the dynamic uses: feature, I would love to see something like below

- uses: org/repo/path@commitish # no expression support
- uses:
    name: org/repo # it's fine if this field does not support expression for security reasons
    path: path-to-directory-containing-actions-yaml # but this should support expression
    ref: commitish # so should this

I don't want to write additional logic for cloning and definitely don't want to use PAT to clone internal repositories.

My current workaround is to have placeholders in reusable workflow. The placeholders are replaced at the time of tagging

reusable-workflow.yaml

- uses: org/repo/path-to-action@{reusable_workflow_version}
- uses: org/repo/path-to-other-action@{reusable_workflow_version}

I've a workflow that replaces these placeholders with simple sed command and creates, pushes commit then creates a tag from that commit

sed -i 's/{reusable_workflow_version}/${{ inputs.version }}/g' reusable-*
git tag -a -m "Release from ${{ github.sha }}" ${{ inputs.version }}

The drawback is tagging needs additional steps, and this only works for reusable workflows that have actions from same repository. There is issue of tags from detached HEAD.

I hope GitHub provides solution for this problem soon

[TWiStErRob]

Nice one @yogeshlonkar! What's your workflow for PRs? How do you test them? Do you run the same script when you start working on a branch, and then revert at the end?

[yogeshlonkar]

We have separate test workflows for each action in the repository. But for reusable workflows we have separate repository where we simply use dev tag from reusable repository. This is very cumbersome but works for now.

[gmazzo]

We need github.job_workflow_ref / github.job_workflow_sha.

Just to confirm, these github.job_workflow_ref and github.job_workflow_sha does not exist right?

I'm working on a workflow that needs to gather a file that it's in the same branch/tag/commit that the called workflow. I couldn't find a context value that holds that reference. Is it possible?
github.workflow_sha points to the commit hash of the caller workflow, but I need the hash of the called one 🤔

[TWiStErRob]

@gmazzo I think the OICD is a workaround for that:
https://github.com/orgs/community/discussions/31054#discussioncomment-3495110

[neilime]

Any chance having github.job_workflow_ref and github.job_workflow_sha without using OIDC workaround?

[neilime]

Any news on this topic ?

[noamgreen]

look like action is stuck in 1999 ....

[neilime]

Yes it's a shame and a nuisance

[ChristopherHX]

Shure actions is stuck on this topic and a lot of others as well

Somewhere and lost track of the source I read the rest api can now be used as partial replacement.
The same person made an action using that afaik.

Hmm using multiple workflows can create a chaos using this...

https://docs.github.com/en/rest/actions/workflow-runs?apiVersion=2022-11-28#get-a-workflow-run-attempt--code-samples

The referenced_workflows is the interesting part.

What we need are ${{ github.token }}, github.run_attempt and ${{ github.run_id }} then do rest request and parse json

All in all a buggy mess, I couldn't tell which reusable workflow entry is correct.