Audit workflows codecov #398
Comments
|
From gitter discussion, it's possible that we don't really need to worry if |
|
I wrote a quick script to collect all the actions and secrets in any workflow in any repo in an org on github to get an overview of what third party actions might have access to if they are compromised: results for jupyterhubWe can use that to audit whether we should do some sha pinning on some dependencies and/or reimplementing some that don't offer us a whole lot. |
|
Does anyone know if all secrets are independent, or if some are shared across multiple repos? If there's a significant number it might be worth switching them to an organisation secret https://docs.github.com/en/actions/reference/encrypted-secrets#creating-encrypted-secrets-for-an-organization |
|
Most secrets (certainly PyPI) are scoped per repo so don't make a lot of sense to share. Should probably be the same with deploy keys for e.g. chart push, but I'm not sure if we do that right now. Are our Docker push secrets single-use tokens with repo scopes? That might make the most sense to share if they are the same |
|
Maybe @hamelsmu can answer this: if no secrets appear in a workflow with a compromised action, is there any way for that action to reach secrets e.g. in other workflows on the same repo? Or is the scope of access for an action strictly what's in the workflow file? That seems to be right. |
|
It depends who is running the workflow and what the permissions you have on these things are. It's a really confusing subject. But by default, pull requests from forks don't have access to your secrets. |
|
@hamelsmu thanks! This issue is about the codecov vulnerability where an action was compromised, which means it ran from our main branches as well. So if the workflow had secrets in it, they definitely would have been compromised. I think we are comfortable with the fact that PR workflows don't have access to things, but are wondering if actions have access to anything when run from the main branch on the origin repo even if no secrets appear in the workflow. I'm assuming not, but trying to find stronger confirmation than I have been able to so far. |
|
In particular, this text is what's tripping me up:
Which seems to suggest that a compromised action has access to $GITHUB_TOKEN and all secrets, unconditionally. |
|
@minrk my best guess interpretation is that there are ways for all steps executing as part of a job, including third party actions, to break free from their isolation (no proper isolation) and potentially acquire the GITHUB_TOKEN or equivalent permissions, so even though its not directly made available, it should be considered directly available from a security standpoint. |
|
I think the preceding paragraph has some important context:
This sounds like a compromised action can access the state created by previous steps, and can manipulate the state of the system causing subsequent actions to behave differently. For example it might modify an output artifact such as a script which is passed to another job, thereby accessing the secrets in that other job. To me it implies the secret still needs to be inserted into the workflow at some point rather than all secrets being accessible, but it would be nice to get a definitive statement from GitHub. |
|
@manics yeah, that's my understanding as well. I just wish they weren't self-contradictory and why I think it's weird to say that it "would have access to all secrets configured on your repository" without qualification, since that doesn't seem to be true. I've at least confirmed that compromised actions repos have access to $GITHUB_TOKEN via Interestingly, when I tried a proof of concept creating a workflow to exfiltrate secrets with this, it was rejected without workflows permission:
Frustratingly, there doesn't appear to be a way to prevent $GITHUB_TOKEN from having write access, which makes third-party actions much less attractive across the board. I really wish you had to opt-in to write permissions in workflows, but not even having opt-out seems wild. The only thing I can think of is to run our tests only on PRs and never from our own repos, and only run deployment workflows with extremely limited action dependencies from our own branches. Also the bar for using any third-party actions should be pretty high, I guess. |
|
Alternatively we could fork third party actions from individuals in smaller orgs, do a full code review, and for Javascript actions rebuild |
|
Another option could be to revive the idea I had of building a container image that contains all the tools we need to, e.g., deploy mybinder.org and run our deployment in that environment rather than relying on third-party actions to install tools for us (extra benefit is speed since the installation steps are run at |
|
I’ll see if someone from the Actions team can look at this issue (or at
the very least ill share this issue with the team and see if anyone has any
ideas)
…On Fri, Apr 16, 2021 at 7:23 AM Sarah Gibson ***@***.***> wrote:
*Another option* could be to revive the idea I had of building a
container image that contains all the tools we need to, e.g., deploy
mybinder.org and run our deployment in that environment rather than
relying on third-party actions to install tools for us (extra benefit is
speed)
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#398 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AALKJEVNCJG7YSXCOQDKFIDTJBB6PANCNFSM4276ORPA>
.
|
|
If we're pinning all actions we could consider pinning all software dependencies (including transient dependencies) too. |
I WAS COMPLETELY WRONG.The problem is much worse than I had anticipated Re: GITHUB_TOKEN Feedback from some knowledgeable folks at GitHub
So um, I don't know what else to say. Will try to find out more information about this. My new understanding is that compromised third party third party Actions can trivially access your secrets You might be wondering what is the point of this YAML syntax The answer is the compromised Action can still grab |
|
If your workflow does not explicitly reference any of your secrets, then none of them are sent to the runner. The GITHUB_TOKEN can't read secrets from your repo. However, it can do other things like push code as outlined in the documentation. |
That is correct. If you do not reference any secrets in the workflow then none will be available. |
|
Thanks @chrispat so it seems that a downstream compromised workflow step could access a secret if it is passed to an earlier step (because that's what triggers the secret to be sent to the runner)? Or is each step self contained somehow? |
|
Steps are not isolated from each other, they all run in the same workspace. A job is the isolation level in Actions. |
|
@chrispat one more question. Since the secrets are available in memory to downstream steps within a job, what is your recommendation on the scope of what needs to be audited? For example I can think of the following things - (1) All third party actions you use (2) Any software dependencies you use in your workflow (3) Any dependencies that Actions itself uses or vendors into the environment. |
|
Thanks @chrispat and @hamelsmu! Good to have confirmation that secrets aren't directly available. We have small workflows with credentials for publishing releases and larger workflows with none, which makes our audit surface easier. That mainly leaves the write access with $GITHUB_TOKEN that's left to deal with. What is the best way to have a workflow that cannot have any write permissions? For instance, in our affected test workflows, we don't use any secrets, so that's fine. I thought that was enough to be safe, but didn't understand that all actions appear to unconditionally have write access to your repo. |
|
We are shipping a feature next week whtat will let you configure the permissions for the GITHUB_TOKEN secret and let you set the default permissions to |
|
This changelog? https://github.blog/changelog/ |
|
This recent blog has been on my to-read list as well https://github.blog/2021-04-13-implementing-least-privilege-for-secrets-in-github-actions/ |
|
Awesome! I took the opportunity to apply the updated restricted default permissions to the jupyterhub org (and ipython, jupyter, jupyter-server). Any actions that need write permissions ought to be opt-in now |
|
I've checked the GitHub Events archive for the affected timeframe, and the repos which used the codecov action never had a push event triggered by GHA. So I think we can close this issue for now, but it's still probably prudent to do audits (and maybe pin shas) for all of our deploy/publish/release actions |
|
Another follow-up: Codecov updated their report indicating that they are confident the only compromise was exfiltration of
* note: that commit looks like it was made by @consideRatio because there is no record in the commit that it was made by GHA (should be fixed by jupyterhub/zero-to-jupyterhub-k8s#2153). The events archive, however, has an 'actor' field that cannot be spoofed, unlike git committer/author. |
codecov revealed a security issue where their bash uploader was compromised.
Anyone using the bash uploader and validating the checksum wasn't affected, but the official codecov/codecv-action didn't (and still doesn't). We don't use the bash uploader directly, but this search suggests we used the codecov action in the following repos:
GitHub Actions docs state that every action has access to every bit of information available to the workflow, meaning we should assume that every bit of information available anywhere in a workflow with the codecov-action has been compromised.
So we should audit the affected workflows of each of these repos and either:
Having multiple workflows seems to help here: As I understand it, secrets are not available unless they are explicitly added to the workflow somewhere (they are technically accessible to all actions if they are used anywhere in the workflow). I think we mostly don't use any credentials in our test workflows. And this reveals that we should keep it that way!
It also suggests to me that we should avoid using third party actions in our release workflows or any workflows with any secrets anywhere, unless they do a lot, and be pretty careful to pin actions in release workflows with at least full tags, and possibly shas.
The text was updated successfully, but these errors were encountered: