Support for sharing secrets between jobs

[danielmarbach]

actions/runner#1498 (comment)

To give you some more insights. Normally we would want to do everything we can as part of one job. Yet in this specific case we are setting a server cluster that requires a few nodes to work. That is an expensive operations and we are working against a limited set of resources. By having the setup of the cluster in the same matrix build job we would then setup the cluster per matrix which would quickly lead to resource exhaustion.

To accomodate that we have created a setup job that creates the cluster. The matrix builds wait for the setup job to be completed. Then there is also a cleanup job that runs after all the matrix builds have run or things have failed. Both the matrix jobs as well as the cleanup need information about how and where to access the cluster in order to be able to connect to it and eventually destroy it again after the run. We want to avoid having this information to be leaked. Hence we were hoping to "just mask the dynamic secrets" and then share them with jobs.

I can understand though the design and architectural reasons why that is not allowed (or supported). It just means for any such a scenario you are basically forced to reinvent the wheel like we did.

here is the encryption step we ended up using

https://github.com/Particular/NServiceBus.RavenDB/blob/74f4ee2c1ad17327e8da523667a2b9f8aa323df0/.github/workflows/ci.yml#L125-L129

here the artifact upload

https://github.com/Particular/NServiceBus.RavenDB/blob/74f4ee2c1ad17327e8da523667a2b9f8aa323df0/.github/workflows/ci.yml#L130-L137

then the explicit download, decrypt steps

https://github.com/Particular/NServiceBus.RavenDB/blob/74f4ee2c1ad17327e8da523667a2b9f8aa323df0/.github/workflows/ci.yml#L168-L184

[ethomson]

Thanks @danielmarbach. I appreciate the helpful example - this isn't something that we support today but your use case makes a lot of sense and is something that we'll consider in the future. I've added this to the backlog.

Thanks again for the helpful feedback.

[KevinMSampson]

To add to this we should have a way to target jobs that can use secrets. I don't want to have someone print or use a secret from a called workflow job in a caller workflow step afterwards.

[danielmarbach]

Agreed

[mike-kaminski]

Could this be handled by having the setup job run an API operation to either create a temporary repo secret using an easily constructed naming convention, or simply update a statically assigned secret each run that can be leveraged by subsequent jobs? You could add the temporary secret's ID as an output that the consuming job(s) reference for the value (the secret name/ID) and then utilizes as appropriate? It'd be less "clunky" and could also be used for other workflows, or if it's an organization level secret instead, be used by other repos in the org. Perhaps as a method to share that temporary infrastructure?

[jnahmias]

I've run into this exact requirement myself, see https://github.com/orgs/community/discussions/29880 for more details on my use-case.
@ethomson - any update on where this is on the roadmap? Having a way to share ephemeral random tokens within a workflow would be really useful.

[JeffreyArt1]

Any updates on this?

[danielmarbach]

FYI in the meantime we moved away from using multiple jobs towards having custom actions with setup and teardown. I will try to update the links in this issue to point to the previous version

[mikocot]

this is not really a solution or even a workaround, the reason to divide workflow in jobs is not only for readability (which could be indeed achieved by compositw actions instead) but to have independent blocks that can be run or not, that can fail or not and therefore control the flow, and finally to also run matrices.

If you have a matrix that is going to trigger 10 runs, you don't want, or often cannot, run all the previous actions that led to it in each of those runs.

[danielmarbach]

That is true. My FYI was never intended to hint or say what we did was a solution or workaround. I just wanted to give a heads up our reality has changed. The problem as you rightfully pointed out still exists.

[jonashartwig]

Hi, we at Telia Company have a similar requirement that grows out of company directives. They require easy modular reusable workflows and proper secret handling. For secrets Vault is used.

So here is what we struggle with:

We have a re-usable workflow that deploys any helm chart from either the git repo or a remote helm repo. For lots of use cases that re-usable workflow must be provided with secrets that are external to that workflow. So what we tried to do is this:

name: Build and deploy

on:
  push:
    branches:
      - "*"
    tags:
      - "v*"

jobs:
  build:
    uses: telia-company/devx-cicd-workflows-and-actions/.github/workflows/build-helm.yml@main
    with:
      squad: "popsicle"
      chartValueFiles: values/sit.yaml
    # these secrets are ok, and can be loaded inside the re-usable workflow, they are always the same
    secrets:
      repositoryUsername: ${{ secrets.ARCUS_JFROG_WRITER_USERNAME }}
      repositoryPassword: ${{ secrets.ARCUS_JFROG_WRITER_PASSWORD }}
  vault:
    runs-on: ["telia-managed"]
    steps:
      - name: Import Secrets
        uses: hashicorp/vault-action@v2.4.0
        id: vault
        with:
          url: https://some.url
          namespace: Hid100007639
          # these must be on the repository anyway
          token: ${{ secrets.ARCUS_VAULT_TOKEN }}
          # here starts the problem, these secrets must come into the next job
          secrets: |
            kv/data/github secret1;
            kv/data/github secret2
  deploy-sit:
    needs: [ "vault" ]
    uses: telia-company/devx-cicd-workflows-and-actions/.github/workflows/deploy-helm.yml@main
    with:
      squad: popsicle
      namespace: infra
      chartValueFiles: values/sit.yaml
      releaseName: arcus
    secrets:
      # here is the problem, this is not easy possible, outputs are rightfully redacted as this are secrets and we do not have another way
      chartValues: |
        secret1: ${{ needs.vault.secrets.secret1 }}
        secret2: ${{ needs.vault.secrets.secret2 }}
      k8sConfig: this should also come from vault

What we want is a way to simply and safe share secrets between jobs in one run. It would be great if a workflow like outputs could get secrets like feature so those are available through needs context. Or any other solution of course :)

regards

[romain-cambonie]

@jonashartwig
https://nitratine.net/blog/post/how-to-pass-secrets-between-runners-in-github-actions/

TL;DR; you need to encrypt / decrypt the secret.

Using this article as base I successfully passed secrets between jobs.

An example in a workflow that pass encrypted AWS secrets between jobs : https://github.com/romain-cambonie/serenity-workflow-dev/blob/master/.github/workflows/_add-aws-credentials-to-workspace.terraform.reusable.yml

[jonashartwig]

Hi, thanks. We have read that article multiple times and considered doing it that way. However we think the setup and maintenance of this is too much.

[h-r-k-matsumoto]

I faced same issue when using vault-action.
I resolved it by distributing the secrets of Vault to secret in a repository like below.
hashicorp/vault-action@main...h-r-k-matsumoto:vault-action:feature/gh-secrets

I know this is not desirable.
It cause to persist secret unnecessarily.
However, it is possible to centrally manage secret with Vault.

[tvquizphd]

Hi @Alleged627. Are you directing your comments to @danielmarbach ? This is a world-wide discussion.

I'm running into the same issue on my own project. GitHub Actions allows workflows with multiple jobs.
It seems that jobs cannot share "secrets" among themselves.
So, any info shared between jobs is publicly visible.

The main workaround now is to use encryption before sharing info between jobs.

[kesireddymanikanta1]

secrets need to encoded in first job and then later decoded in another job

[rdhar]

BACKGROUND CONTEXT

Although there are now docs on masking and passing secrets between jobs or workflows (as @romain-cambonie linked), it didn't meet requirements like:

• Storing and retrieving from a secret store seems a tad over-the-top (as @jonashartwig mentioned).
  • The same goes for encrypted artifact upload/download.
• For ephemeral, temporary OIDC tokens, it's not feasible to dynamically store/retrieve those secrets (as @jnahmias raised).

PROBLEM STATEMENTS

With these factors in mind, the main blockers are:

• Per docs, to prevent accidental disclosures, GitHub Actions redacts any configured secrets as well as common encodings of those values, like Base64.
• For dynamically-generated secrets, those have to remain masked from the caller workflow to the reusable workflow to avoid exposing their their values in the workflow logs.

PROPOSED SOLUTION

1. Before going to output secrets from the caller workflow, Base64 encode the values twice.
   1. This enables GitHub Actions to output the secrets without exposing them.
2. Per docs, pass the encoded values to the reusable workflow as secret inputs.
   1. This enables the reusable workflow accept and treat the encoded values as secrets.
3. In the reusable workflow, decode the values from Base64 twice before masking them.
   1. This ensures that the secrets remain masked in the workflow logs throughout.
   2. They can now be populated as environment variables for use in the subsequent steps of the reusable workflow.

WORKING EXAMPLE

These methods are sourced from DevSecTop/TF-via-PR (permalink) repository, which hosts a reusable workflow to run Terraform commands via PR comments, like a CLI.

As a bonus, any number of secrets can be securely passed into the reusable workflow to be used as environment variables, for example. The repository also contains recent GitHub Actions workflow runs to verify that the secrets remain masked throughout.

# caller-workflow.yml

jobs:
  credentials:
    runs-on: ubuntu-latest
    outputs:
      CREDENTIAL1: ${{ steps.credentials.outputs.CREDENTIAL1 }}
      CREDENTIAL2: ${{ steps.credentials.outputs.CREDENTIAL2 }}
    steps:
      - name: Output encoded credentials
        id: credentials
        env:
          CREDENTIAL1: ${{ secrets.CREDENTIAL1 }}
          CREDENTIAL2: ${{ secrets.CREDENTIAL2 }}
        run: |
          echo "CREDENTIAL1=$(echo $CREDENTIAL1 | base64 -w0 | base64 -w0)" >> $GITHUB_OUTPUT
          echo "CREDENTIAL2=$(echo $CREDENTIAL2 | base64 -w0 | base64 -w0)" >> $GITHUB_OUTPUT

  reusable-workflow:
    needs: credentials
    uses: reusable-workflow.yml
    secrets:
      env_vars: |
        CREDENTIAL1=${{ needs.credentials.outputs.CREDENTIAL1 }}
        CREDENTIAL2=${{ needs.credentials.outputs.CREDENTIAL2 }}

# reusable-workflow.yml

on:
  workflow_call:
    secrets:
      env_vars:
        required: true
jobs:
  parse-credentials:
    runs-on: ubuntu-latest
    env:
      env_vars: ${{ secrets.env_vars }}
    steps:
      - name: Decode credentials as environment variables
        run: |
          for i in $env_vars; do
            i=$(echo $i | sed 's/=.*//g')=$(echo ${i#*=} | base64 -di | base64 -di)
            echo ::add-mask::${i#*=}
            printf '%s\n' "$i" >> $GITHUB_ENV
          done
      - name: Validate credentials
        run: |
          # Secrets are now available as masked environment variable.
          echo $CREDENTIAL1 # or ${{ env.CREDENTIAL1 }}
          echo $CREDENTIAL2 # or ${{ env.CREDENTIAL2 }}

Where:

• base64 -w0: Ensures that the encoded values are outputted in a single line, regardless of the length of the input.
• base64 -di: Decodes the values from Base64, ignoring any newlines from echo'd strings.
• echo ${i#*=}: Removes the key of the key-value pair using parameter expansion, leaving only the value.
• sed 's/=.*//g': Removes the value of the key-value pair, leaving only the key.
• echo ::add-mask::${i#*=}: Masks the value of the key-value pair without outputting.

FLAWS

• If the workflow is (re-)run in debug mode, then secrets from credentials job specifically are output in the workflow log.
• The reusable workflow needs to be configured with the steps under parse-credentials job.

[shinebayar-g]

Hey @rdhar thanks for letting me know the missed detail. You are right. Job output is not logged. It just says Set output <name>. For the reusable workflow, input is received as secrets: thus it's hidden from the job input. It's a great workaround when you have control of the reusable workflow.

[shinebayar-g]

Hey @rdhar , have you tried running the credentials job with Enable debug logging checked? I think that might show outputs.

[rdhar]

Hey @shinebayar-g, you're spot-on, (re-)running the workflow in debug mode does output secrets in the credentials job, where it's isolated. And some level of control is required over the reusable workflow in order to parse the encoded credentials from the caller workflow.

I've now listed both of these flaws in the original post to ensure users are made aware.

[shinebayar-g]

Yeah, gotta make sure you're not accidentally leaking secrets in the debug run. I haven't found a way to check if the workflow is running with Enable debug logging enabled. (ACTIONS_RUNNER_DEBUG variable mentioned on docs is apparently always empty unless you put some secret on the repo) If you ever happen to find that please let me know :)

[awgeorge]

Try ${{ runner.debug }}

[shinebayar-g]

Outputs containing secrets are redacted on the runner and not sent to GitHub Actions.

https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idoutputs

This makes reusable workflows useless when we want to download some secrets from somewhere and pass it to reusable workflows.
Whole encrypt/decrypt thing doesn't work when we want to use someone else's reusable workflow which we cannot control.

[ChrisPage-AT]

It would be really fantastic if Github supported secrets in "outputs" variables for jobs. I wouldn't care if it's a separate section named "secret outputs" or keep things in outputs and have the ability to add a "secret: true" flag. In workflows with multiple jobs, it would be really nice to only have to make one call to secrets providers and then be able to use that secret throughout the whole workflow via output.

Separate sections example:

jobs:
  get-workflow-values:
    runs-on: linux-small
    name: Get Workflow Values
    outputs:
     notSecretValue: ${{ steps.get-not-secret-value.outputs.value }}
    secret-outputs:
     secretValue: ${{ steps.get-secret-value.outputs.value }}

Flag example:

jobs:
  get-workflow-values:
    runs-on: linux-small
    name: Get Workflow Values
    outputs:
     notSecretValue: ${{ steps.get-not-secret-value.outputs.value }}
     secretValue: 
      description: "This is a secret value for things"
      value: ${{ steps.get-secret-value.outputs.value }}
      secret: true