Add and expose job/trigger context as a JWT

[ojohnny]

Clear and concise description of the problem

As a security-concerned devops engineer¹ I want to be able to provision short-lived SSH certificates² from a Trusted SSH Certificate Authority based on contextual information (who triggered the build, which project, which job) so that I can follow security best practices (e.g. Principle of least privilege, ability to revoke access etc) when configuring servers.

¹Hypothetical user, I'm just a paranoid developer/selfhoster.

²Other data may also be of interest to retrieve from the same system, e.g. user-based sudo password may be needed in cases where admins have their own accounts on the servers being configured. This appear to be a "best practice" at least regarding tracability/logging, and each admin may need to have their own SSH certificate/sudo password pair. Perhaps SSH certificates allows for some tracability even in shared account settings, but I haven't dabbled too much in this area.

Suggested solution

I should start by stating that the functionality of being a Trusted SSH Certificate Authority could/would/should be handled by an external system (i.e. out of scope for Woodpecker), and one would most likely use something like Hashicorp Vault's Signed SSH Certificates in this situation.

I don't really expect an "integration" to this particuar tool to be made for Woodpecker, but rather expose "good enough" credentials so that we at least can use their CLI tools to sign an SSH key manually inside a job. This would first require one to be able to authenticate via their API.

Requirements

• Expose a JWT (e.g. as an environment variable) containing relevant context (perhaps the builtin environment variables + information about the "sender" of the Webhook)
  • This obviously needs to be signed using a key, possibly the one corresponding to /api/signature/public-key introduced by Use asym key to sign webhooks #916?
  • One would then configure Hashicorp Vault to accept JWT's using that public key ("static key" in their documentation), which I think is sufficient for a first implementation.
  • This is very similar to how Gitlab does it, and one could take inspiration from what their CI_JOB_JWT contains and how their documentation suggests how to configure Vault policies.
• "Triggerer/sender" information needs to be retrieved from the webhook (most forges seem to have this information in their webhooks?) and verified using signatures in webhooks.
  • Github uses a X-Hub-Signature / X-Hub-Signature256 header in the webhook request.
    • I don't believe Woodpecker is currently setting up the webhook secret right now, so Github is not currently sending this header to Woodpecker?
  • Gitlab uses a X-Gitlab-Token header in the webhook request.
    • Woodpecker appears to already set the token needed for Gitlab to send this header.
  • Gitea uses both X-Gitea-Signature and X-Hub-Signature headers in their webhook requests.
    • Woodpecker appears to already set the secret needed for Gitea to send this header.
  • (verification of webhooks could be a separate issue, but this feature should not be deployed without it)

Alternative

• There is a separate issue regarding using Hashicorp Vault as a storage engine (Hashicorp vault as secret storage integration #929), but I'm unsure if that would handle the "other" API's provided by Hashicorp Vault, such as Signed SSH Certificates. Probably not? And if it did, then I suppose that integration would need to have access to similar information anyway?
• If there was a way to have "per-user secrets", then one could conceivably store separate credentials to the Vault for each user who may trigger the pipeline. But that feels a bit more like a hack (more steps to setup for the user) and I don't believe Woodpecker has this functionality already.
• I'm don't think this could be handled by creating a plugin? (the webhook headers/body is not exposed to plugins from what I have read so far?)

Additional context

• Gitlabs documentation for authenticating with Hashicorp Vault.
• We need to consider the life cycle of the JWT
  • Should the TTL (time-to-live) be configurable, if so, how? Global configuration?
    • At least Gitlab seems to have a default value of 1h, should we use this as a default value?
  • If we create it when the webhook was received, then if the job gets queued then the JWT may have become expired before the job is run (if the TTL is "too low")
  • Could we create a JWT just before each job is being run? If so, then we might be able to shorten the TTL a bit should we want to.
• I'd be happy to help out with this/create a PR if this is an acceptable feature. I would need to familiarize myself with the codebase first, however.

Validations

• Read the Contributing Guidelines.
• Read the docs.
• Check that there isn't already an issue that request the same feature to avoid creating a duplicate.

[anbraten]

First of all thanks for writing this super detailed issue with all your thoughts and ideas. I recently read about authenticating CI jobs against external and internal (Woodpecker itself) pass systems.

Linking #368 for the authentication of jobs against Woodpecker itself.

As far as I can tell OpenID seems to be a pretty popular keyword in this context. The idea would be to add an OpenID identity provider to or next to the Woodpecker server and pass OpenID tokens to the pipeline jobs. Those tokens could then be checked by the third parties you are trying to access.

Some related links from other CI systems:

• https://circleci.com/blog/openid-connect-identity-tokens/
• https://docs.gitlab.com/ee/integration/openid_connect_provider.html
• https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/configuring-openid-connect-in-cloud-providers

[ojohnny]

I thought about using some kind of token for this at first, but then I noticed that it might not be possible for some forges to expose a token like that (at least if I understand go-gitea/gitea#20394 correctly), so then I thought that verifying webhooks seems more "universal" and readily available, and at least still could allow for some nifty things to get done.

I am also not sure how one would actually use such a token to verify against Hashicorp Vault, as their OpenID Connect auth method appears to require user interaction via browser?

[anbraten]

This is how it should work with Github and Vault: https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/configuring-openid-connect-in-hashicorp-vault

[ojohnny]

The "Github Actions" flow seems to be similar to the one I suggested, but with the addition that they use "OIDC Discovery" for JWT verification instead of "Static keys". If I interpret their documentation correctly, that is.

They still generate a JWT with job contextual information, which is similar to what Gitlab does and what I suggested, but from my understanding they also have implemented a discovery endpoint via the tokens iss (which for them is located at https://token.actions.githubusercontent.com).

If I understand you correctly, you are proposing that we should also implement OpenID Connect Discovery in addition to the JWT?

In summary, the differences I'm seeing are:

• Github Actions (likely) uses the "OIDC Discovery" mechanism to verify the token as opposed to "Static keys" in Hashicorp Vault.
  • Is this a requirement to have implemented an OIDC Discovery endpoint in Woodpecker? Woodpecker would then basically become an OpenID provider, in a way.
  • My opinion is still that using static keys "is sufficient for a first implementation", but having verification via OIDC Discovery would be very nice to have. I'm just worried that it would become "too big" for a single feature in order to "get something out there".
  • I guess the mechanism of verifying webhooks from the (external) forges is still needed in order to securely "proxy" the authentication of the user via the Woodpecker OIDC provider in any case? 🤔
• Github Actions appears to have a "builtin action" hashicorp/vault-action. I suppose this similar to what Woodpecker calls a "plugin"?
  • I believe such a plugin would be "nice to have", and can most likely use whatever is implemented for this ticket in order to do so. But I believe it to be a separate feature request.
  • By doing this they "don't need" to expose the token via a environment variable. But if one were to implement such a plugin, surely the plugin would need it?

[sloonz]

If it can be of any help, I have implemented this a while ago on Drone : https://github.com/sloonz/drone-env-signed and https://github.com/sloonz/drone-ssh-ca.

That’s the thing that currently prevents me from migrating to Woodpecker.

[loicsikidi]

Hi,

Any news about this topic (eg. roadmap)?

I think that this feature is relatively important from a security point of view because the latter brings Web Identity Federation capability which helps to secure the supply chain (non-exhaustive list):

• Allow to authenticate to cloud providers with short-time credentials
• Allow to sign artefact using Sigstore

As expressed in a previous post this feature is provided by a lot of CI tools.

Vendor
Bitbucket
CircleCI
Github
Gitlab

[6543]

There is no eta. If a dev pick it up and it got reviewed nd merged, it's done