Implement repo level, user level and global environment variables support

[dvjn]

What?

• Add support for variables at user, org, repo and admin level which will be exposed to the pipelines.
• This is similar to secrets, but they will not be limited to specific events and plugins and will not be masked.

Why?

• There are use cases when we need to set some org level or repo level variables, which need not be a secret.
• A workaround is to use secrets, but then it masks the variable from all the output which is annoying sometimes.

[dvjn]

Hi @6543 @anbraten @qwerty287 I have done initial UI changes for this, please review the proposal and screenshots and let me know if I should continue working on this and implement the rest.

Screenshots:

Repo Settings:

User Settings:

Admin Settings:

Add Variable Screen:

[anbraten]

As variables are not secret anyways there was a suggestion somewhere to have a special config file like .woodpecker/config.yaml in the repo which can be used for such settings. This would be quite handy as it would also allow to use those variables in the cli exec command as well and allows easy copies of that config to other woodpecker instances etc. Should those variables then be shown in the UI as well as read-only or should only the config file be the source of them? What do you think?

[qwerty287]

should only the config file be the source of them

I'd prefer this way, but it's not that easy to have a good way to implement this at org/global level.

[zc-devs]

JFYI, how it is implemented in Gitea.

[dvjn]

As variables are not secret anyways there was a suggestion somewhere to have a special config file like .woodpecker/config.yaml in the repo which can be used for such settings.

I think this is not as useful as it seems, since you can already use variables in a workflow file. This config.yaml approach is mainly beneficial when you need these variables in multiple workflows within the same repository.
However, the most important use-cases of this PR, in my opinion, are: first, the ability to manage global variables using the admin UI, and second, the capability to share variables across repositories under an organization or user.

FYI, the solution I'm proposing is heavily inspired by Gitea.

[dvjn]

@anbraten @qwerty287 @zc-devs
Do let me know your final thoughts on this and let me know if I should move forward with implementing it.

[dvjn]

I can also add agent variables to address #3311

[zc-devs]

your final thoughts

I like the idea.

if I should move forward with implementing it

I am not a right guy :) Let's hear from maintainers.

I can also add agent variables

That would be great!

[qwerty287]

Yes, the way you proposed would be fine for me.

However, you write "Fixes #3661" in the desc which is not correct (I believe there's an issue about your thing too). #3661 is about workflow-level vars and this should definitely be defined in the yaml.

[dvjn]

However, you write "Fixes #3661" in the desc which is not correct (I believe there's an issue about your thing too). #3661 is about workflow-level vars and this should definitely be defined in the yaml.

My bad, will remove that issue from the description.

[itsTurnip]

I'm really interested in this PR. Keep moving! Thanks for your work.

[dvjn]

I have completed the code changes in the server and web frontend. This means the feature is now in a fully usable state.
The docs and cli changes are still pending. Planning to do those after an initial round of code review.

It would be great if someone could review the code.

Summary of Changes:

• Duplicated code for secrets (removing events and images).
• Added support for from_variable and variables (similar to from_secret and secrets) in pipeline config.
• Added new table in database variables, with migrations.
• Copied some secrets test cases and used for variables.

[lafriks]

Wasn't there an idea to drop secrets section from step? Maybe let's do not add variables section and keep only one way to use them with from_variable?

[dvjn]

Wasn't there an idea to drop secrets section from step?

I am not aware of it. If it is just an idea without a proper discussion, maybe let's not act on that?

Maybe let's do not add variables section and keep only one way to use them with from_variable?

I have kept feature parity with secrets, to not cause confusion. If we want to remove it, wouldn't it be better to remove both secrets and variables together in a separate PR.

Just my thoughts, let me know if we still want to remove it.

[zc-devs]

If I set some variable

and run pipeline

skip_clone: true
steps:
  glob-vars:
    image: alpine
    commands:
      - echo $$COMPANY

what would I see in steps (UI) logs?

---

Maybe add build_pr_images label?

[dvjn]

what would I see in steps (UI) logs?

It would be empty as you haven't explicitly added those variables in your step either by using variables or from_variable.

[zc-devs]

As this is not a secret value, it doesn't make sense to restrict this values by a step (to me). Like you mentioned

they will not be limited to specific events and plugins and will not be masked

So, I would expect it to be printed without explicitly added those variables in your step.
In case if I want to map "global" var to different name in a step, I can use from_variable.

[dvjn]

@zc-devs

That is a good point. That would make this feature much easier to use.

Although, I feel that this kind of implicit behaviour might cause some confusion. It won't always be obvious where an environment variable is coming from.

Also, it could be confusing for a user who has push permissions on a repo, but cannot change the repo settings. This kind of user won't be able to view where the variables are automatically getting set from.

[zc-devs]

much easier to use

That is what I meant 👍

It won't always be obvious where an environment variable is coming from.
user won't be able to view where the variables are automatically getting set from.

From Server or Agent 😄 I can say that about all those variables, that comes from Agent/Server :) (example)

[dvjn]

I understand your point, but I'm still afraid of adding this new implicit behavior.

Can we get another opinion if that's okay?

[dvjn]

Hi maintainers, it would be great to get a review from y'all.

[lafriks]

I don't think setting them explicitly is a good idea as that could have unforeseen consequences that would be not so easy to track down both from usability and security perspective

[lafriks]

Wasn't there an idea to drop secrets section from step?

I am not aware of it. If it is just an idea without a proper discussion, maybe let's not act on that?

Maybe let's do not add variables section and keep only one way to use them with from_variable?

I have kept feature parity with secrets, to not cause confusion. If we want to remove it, wouldn't it be better to remove both secrets and variables together in a separate PR.

Just my thoughts, let me know if we still want to remove it.

Personally I don't see benefit of having multiple ways of using same feature, just adds confusion.

Also if we add it and later on remove we will have to again create more breaking changes that can be prevented without adding that in the first place

[lafriks]

Also would be confusing if both variables and secrets would be used by the same name on what takes preference etc.

I only briefly skimmed through file changes but imho this introduces back security vulnerability with using variables section with plugins

[dvjn]

Thank you for the comments.

So from what I understood, I'll just keep one way to use variables, i.e., using from_variable, and remove support for variables in pipeline.

Then I'll complete the Cli and Docs changes for the final review.

[dvjn]

@lafriks I have removed the support for using variables, and completed the go client, cli and doc changes.

This PR is ready to be reviewed and merged.

[woodpecker-bot]

Deploying preview to https://woodpecker-ci-woodpecker-pr-3723.surge.sh

[dvjn]

Hi maintainers, I have finished working on this PR. This is ready for review.

[6543]

before reading more into this pull code and issue, tow important questions:

• why did you touch the pipeline/* code at all? I thought we already allow to have environment viariables *ok its the frontend parse :)
• if variables dont have a prefix, all currently steps counted as plugins wo have variables set should not be contued as plugins! ok it does not all fine too

[6543]

we switched back to case sensitive for secrets ... so we should stay consistent

[dvjn]

I can still see name = strings.ToLower(name) in the getSecretValue even after merging main, should I still proceed to remove them from variables?

[6543]

let me have a look at the last pulls in this topic ...

[dvjn]

Any updates on this?

[codecov]

Codecov Report

Attention: Patch coverage is 5.63025% with 1123 lines in your changes missing coverage. Please review.

Project coverage is 26.02%. Comparing base (3c8204a) to head (856ff67).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
server/services/variable/mocks/service.go	5.22%	247 Missing and 7 partials ⚠️
server/store/mocks/store.go	0.00%	190 Missing ⚠️
server/api/org_variable.go	0.00%	98 Missing ⚠️
server/api/repo_variable.go	0.00%	83 Missing ⚠️
server/api/global_variable.go	0.00%	72 Missing ⚠️
server/services/variable/db.go	33.82%	44 Missing and 1 partial ⚠️
server/store/datastore/variable.go	0.00%	43 Missing ⚠️
cli/variable/variable_list.go	0.00%	40 Missing ⚠️
cli/variable/variable_info.go	0.00%	39 Missing ⚠️
cli/variable/variable.go	0.00%	31 Missing ⚠️
... and 17 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3723      +/-   ##
==========================================
- Coverage   26.96%   26.02%   -0.95%     
==========================================
  Files         394      409      +15     
  Lines       27416    28587    +1171     
==========================================
+ Hits         7393     7439      +46     
- Misses      19323    20437    +1114     
- Partials      700      711      +11     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.