[Bug] GitHub Actions does not set job output because it wrongly detects a secret

[behnazh-w]

We are using the build outputs to pass the hash of artifacts between two jobs, which is necessary for creating SLSA provenances.

GitHub Actions is not setting the output with the error message:

Skip output 'hashes' since it may contain secret.

This is clearly a false positive as the hashes of the artifacts would have no secrets. Is there any workaround to pass this value as the output here?

[ossanna16]

Hi there @behnazh-w and welcome to our community! Thank you for asking a great question 🙂
While you are waiting for a reply to your question, here are a few things you can do:

To get started, introduce yourself in our official introduction thread
To connect with other new users, check out this Discussion to Meet Your GitHub Community Team, further your GitHub knowledge by working through our GitHub Skill 🆙 exercises, and participate in our weekly Fun Friday chats

[steiza]

Hey @behnazh-w - I'm keen on helping ensure GitHub Actions works for SLSA provenance generation.

I tried to reproduce this, but wasn't able to, so I'm not sure exactly what's going on. That said, it looks like we're already using workflow artifacts, so maybe we could use that mechanism (instead of output) to share the hash between jobs in the workflow?

Note also that set-output is scheduled for deprecation, so maybe another reason to try a different path.

Here was my attempt to reproduce this, which again, didn't work:

$ cat test.yml
name: Test
on: 
  workflow_dispatch:

jobs:
  first:
    outputs:
      hashes: ${{ steps.hash.outputs.hashes }}
    runs-on: ubuntu-latest
    steps:
      - name: Generate output hash
        id: hash
        run: echo "::set-output name=hashes::$(echo 'hi' | sha256sum | base64 -w0)"
  second:
    needs: [first]
    uses: ./.github/workflows/other_workflow.yml@main
    with:
      hashes: "${{ needs.first.outputs.hashes }}"

$ cat other_workflow.yml 
name: Other workflow
on:
  workflow_call:
    inputs:
      hashes:
        description: "hashes"
        required: true
        type: string

jobs:
  echo:
    runs-on: ubuntu-latest
    steps:
      - name: Echo hashes
        run: "echo ${{ inputs.hashes }}"

[behnazh-w]

@steiza Thanks for helping out.

I tried to reproduce this, but wasn't able to, so I'm not sure exactly what's going on.

Same here. I cannot reproduce the problem in a private clone of this repository. However, this problem is consistently happening for various Micronaut projects that are created from the same template repo, where the SLSA provenance generation is integrated. My guess is that the GitHub Actions security analysis that monitors job outputs for secrets has a bug, which results in this false positive.

That said, it looks like we're already using workflow artifacts, so maybe we could use that mechanism (instead of output) to share the hash between jobs in the workflow?

The slsa-github-generator GitHub Actions need the job output to pass the hashes between jobs. That's because a workflow artifact can be updated (replaced) by a different potentially malicious job triggered by the same workflow. So if we store the hashes in a workflow artifact we can't guarantee the integrity of the build artifacts. But unfortunately GitHub is not working as expected and does not set the output correctly. So the slsa-github-generator requires a feature from GitHub that is not reliable.

Note also that set-output is scheduled for deprecation, so maybe another reason to try a different path.

We should still be able to use the GITHUB_OUTPUT environment files, right?

[steiza]

Variables for GitHub Actions have shipped today, so you can move non-secret values out of secrets and into variables.

For more information see https://docs.github.com/en/actions/learn-github-actions/variables.

[laurentsimon]

I found another corner case https://github.com/laurentsimon/slsa-delegate-project/actions/runs/4008385408/jobs/6882482821
I have no secrets used, but the scanning code thinks a job output has secrets, and it matches the brackets of the JSON toJson(steps.mystep.outputs):

OUTPUTS: ***
  "some-output": "tool-defined-some-output",
  "other-output": "tool-defined-other-output"
***

Any idea about root cause?

[ChristopherHX]

I have no secrets used, but the scanning code thinks a job output has secrets

All accessible secrets are masked, even those you don't use in your workflow file.
I think the docs make me believe that you have to use the secret, but that's proven wrong
To reduce secrets the runner would mask use reusable workflows and only map the secrets you really need.
E.g. You might have an org or repo secret like this, now all { and } are masked.

{
  "key": "value"
}

You could minify all json secrets into a single line, then you shouldn't see { or } masked.
As descibed above reusable workflows can make selected secrets inaccessible and therefore avoid masking { or }

[laurentsimon]

I was using reusable workflow with no secrets passed.

[ChristopherHX]

Can you share your diag logs? (rerun jobs has a check box to enable debug logs, then download log archive of the rerun)

You find all secret names sent to the runner in it with secrets masked in the worker log.

Edit
I forked your repository run ci and the result was no masking, therefore I would need your diag log to understand this phenomen. Secret ACTIONS_RUNNER_DEBUG=true enables uploading of diag logs

All in all I'm curious what happend, diag logs allows me to replay the job locally using a debugger.

[y-chen]

I am having the same issue while exporting as job output the first 7 chars of github.sha and a random generated UUID using uuidgen.

The masking is completely random. Sometime one is skipped and sometime the other.

Have you figured out a solution?

[laurentsimon]

This may be the long-term solution github/roadmap#575.

[srinadhbh]

The issue still persists w/ github.com. Is there a work around?

[louwers]

This is very annoying. Outputs should not be ignored if there is a potential secret, because build outputs can be IDs and the detection meachanism is flakey.

[MeCRO-DEV]

Any solution yet? I'm having the same issue when passing Azure resource group name to the next job, and GitHub will not set the output because it was treated as a secret. But in fact, it is just a regular variable.

[MeCRO-DEV]

to the next job, and GitHub will not set the output because it was treated as a secret. But in fact, it is just a regular variable

[MeCRO-DEV]

Any solution yet? I'm having the same issue when passing Azure resource group name to the next job, and GitHub will not set the output because it was treated as a secret. But in fact, it is just a regular variable.

[maeghan-porter]

FWIW I am also having this issue with an AWS ARN. It is considering the Account ID in the ARN a secret and so ignoring it as job output. Very annoying.

[erwinv]

I am also having the same issue with an AWS ECR Image URI. I am trying to separate the (1) build-and-push-to-ECR job from the (2) deploy-ECR-image-to-App-Runner job. Job 2 needs the Image URI from Job 1.

[devin-purple]

I'm seeing this as well.

[petrgazarov]

I was able to do this by setting mask-aws-account-id: 'false' on the aws-actions/configure-aws-credentials@v2 action. That action automatically adds a mask to the account ID. GitHub Actions then detects that the account id in the image URL matches the masked value and treats the image URL as a secret. Not masking the account ID solves it.

- name: Configure AWS credentials
  uses: aws-actions/configure-aws-credentials@v2
  with:
    aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
    aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
    aws-region: us-east-1
    mask-aws-account-id: 'false' # add this

[y-chen]

I am using artifact as workaround for the moment.

I created 2 actions:

name: Upload artifact
description: Upload an artifact

inputs:
  key:
    description: The artifact key
    required: true
  value:
    description: The artifact value
    required: true

runs:
  using: "composite"
  steps:
    - name: Write artifact
      run: echo ${{ inputs.value }} >> ${{ inputs.key }}
      shell: bash

    - name: Upload artifact
      uses: actions/upload-artifact@v3
      with:
        name: ${{ inputs.key }}
        path: ${{ inputs.key }}
        retention-days: 5

name: Output artifact
description: Output artifact

inputs:
  key:
    description: The artifact key
    required: true

outputs:
  value:
    value: ${{ steps.read.outputs.value }}
    description: "The read value"

runs:
  using: "composite"
  steps:
    - name: Download artifact
      uses: actions/download-artifact@v3
      with:
        name: ${{ inputs.key }}

    - name: Output artifact
      id: read
      run: |
        FILE=${{ inputs.key }}
        VALUE=$(cat $FILE)
        echo "Value is ${VALUE}"
        echo "value=${VALUE}" >> $GITHUB_OUTPUT
      shell: bash

And use them like:

      # From one job
      - name: Upload current commit
        uses: ./.github/actions/upload-artifact
        with:
          key: current-commit
          value: ${{ steps.current-commit.outputs.hash }}

      # From another job
      - name: Current commit
        id: current-commit
        uses: ./.github/actions/output-artifact
        with:
          key: current-commit

      # Then I can read the value in another step
      - name: Print
        run: echo ${{ steps.current-commit.outputs.value }}

[benjaminCassard]

I used artifacts as a workaround of this problem but now i'm reaching quota limitation of artifacts (even if retention is 1 day). Same as https://github.com/orgs/community/discussions/37942#discussioncomment-6434547 I use job outputs for standard variables and not secrets.
This issue should definitely be fixed...

[Megachill]

I truly hope MS addresses this! It is annoying beyond belief... Leave it to US to make a decision what is secret and what isn't. We do not need hand holding when it comes to secrets management. Another reason to switch back to gitlab... This is frustrating as hell.🤦‍♂️

[tpoxa]

outputs:
      main_image: ${{ env.MAIN_IMAGE }}
steps:
 -  run: |
            echo "MAIN_IMAGE=$REGISTRY/$PROJECT/$REPOSITORY/main:$TAG_NAME" >> "$GITHUB_ENV"

none of the variables are derived from secrets, but still, main_image output is suppressed

[talmdal]

but does the resulting output string happen to contain a substring that matches one of the secrets that is used in the job. It seems from my limited experience that it is doing something like

for each secret:
   if (outputString.contains(secretValue) 
       issue warning
       exit

[FecoDoo]

Same issue here, have been debugging for a stupid amount of time. It seems like some outputs will be filtered but others may not. It is still a black box and no one knows how GitHub decides whether an output is sensitive or not.

Here's my code snippet, where output p is skipped as GitHub considered it may contain secret, while email is not, though both variables are masked in the same fashion.

jobs:
  credentials:
    runs-on: ubuntu-latest
    outputs:
      email: ${{ steps.base64.outputs.email_b64 }}
      p: ${{ steps.base64.outputs.p_b64 }}
    steps:
      - uses: hashicorp/vault-action@v2
        with:
          ......
          secrets: |
            repo email | EMAIL;
            repo password | PASSWORD;
      - name: base64
        id: base64
        run: |
          echo "email_b64=$(echo "${{ steps.credential.outputs.EMAIL }}" | base64 -)" >> "$GITHUB_OUTPUT"
          echo "p_b64=$(echo "${{ steps.credential.outputs.PASSWORD }}" | base64 -)" >> "$GITHUB_OUTPUT"

[FecoDoo]

Updates: This solution from rdhar seems to be good work-around, which works fine in my case. Check out this discussion for details.

[talmdal]

Is anything being done to resolve this?

I ran across this when creating a S3 resource path /foo_ops/github/artifacts/foo_project/file.zip.

There is also a secret being used in the job with the value foo

I then get the erroneous warning of "Skip output 'artifact-name' since it may contain secret." The name is not emitted as output and the rest of the workflow fails.

Other jobs in the same workflow that don't reference the secret with the value 'foo' have no issues.

The github processor needs to have better understanding of the context and not just match by value.

[gorankarlic]

Having the same issue, value of environment variable MACHINE is changed from e2-medium to e2-medium*** - not just in the log output, but even when used in simple a run action (i.e. bash script) - this certainly does not make sense at all.

Follow up: this was caused by using a raw JSON as a secret - encoding the JSON with Base64 resolved the issue: https://docs.github.com/en/actions/security-guides/using-secrets-in-github-actions#storing-base64-binary-blobs-as-secrets

[abseht]

🤕 Same issue for

    environment:
      name: ${{ inputs.target_env }}
      url: ${{ inputs.target_env == 'production' && 'https://github.com/${{ github.repository }}/releases/tags/${{ env.TAG }}' || inputs.target_env == 'preview' && '${{ github.event.repository.deployments_url }}/preview' || 'https://github.com/${{ github.repository }}/commit/${{ env.TAG }}' }}

Get in the Complete Job step

Evaluate and set environment url
Warning: Skip setting environment url as environment 'development' may contain secret.
Cleaning up orphan processes

[thanford7]

Same issue:

Evaluate and set job outputs
Warning: Skip output 'ecs_cluster_name' since it may contain secret.

The value of ecs_cluster_name is shortify_social-cluster. I marked the output in terraform as not sensitive, but Github still persists in marking it as a secret.

[Megachill]

I know this may not be of help here - but Github has been slacking in terms of addressing this and letting us explicitly set things, so we moved ALL our pipes to gitlab where we are in control at least. You could map however and then re-use with a bit of creativity in your pipe...

To having to do these kinds of workarounds because of "Daddy" issues and "we know better than you do!" is frankly annoying af.