An internal CI/CD system compatible with Github actions workflow yaml syntax, action yaml syntax and most action plugins

[lunny]

Internal CI/CD system

I propose to implement an internal CI/CD system that is compatible with Github workflows yaml syntax, action yaml syntax, and most action plugins. And this is in fact not only a CI/CD system, but you can also choose to not check out the code for some events for example, when an issue is posted, you want to reply to them, then you don't need to check out the code. You can just reply to that issue. So the sub-system name is botsactions, if you have a better name than that, please let me know.

Pros

• Migrated repositories from Github could easily reuse the actions workflow files
• Plugins could be copied from https://github.com/actions or https://gitea.com/actions to your self-hosted Gitea instances or any git service repository address
• Transparent for the CI/CD process. You can check out all workflow files and plugin codes you are using.
• Bring version management for your CI/CD scripts and plugins
• Customize your image. Github's image for ubuntu-latest is very big but Gitea now will use node:16-buster as the default image which is about 150MB and of course you could change it yourself.

Cons

• Images are maybe very big for those small devices and have to be changed when you have new dependencies.
• Many Gitea users are familiar with Drone/Woodpecker
• More feedbacks are welcome

Reference documentation

At the moment, we lack documents, but if you want to know more details about what Github Actions looks like, please visit https://docs.github.com/en/actions.

Names chosen

Different from Github's related sub-system's name, we call the plugins action or actions. One action in fact is a repository with a file named action.yml in the root, the repository is hosted in any git service server which could be gitea.com, your self-hosted Gitea instance, and even github.com. There is a default configuration but we should also support absolute git URL as a plugin repository.

Task summary

There are many things need to do

• Secrets management Secrets storage with SecretKey encrypted #22142
• An event listener and task scheduler Implement actions #21937
• A stable communication protocol between Gitea and runners https://gitea.com/gitea/actions-proto-def
• A runner based on docker https://gitea.com/gitea/act_runner
• A UI that will show tasks and logs Implement actions #21937
• Some core actions are now in https://gitea.com/actions which currently are mirrors but may be changed if there are incompatible between Github API v3 and Gitea API v1
• Support absolute plugin address like uses: https://github.com/peaceiris/actions-hugo@v2 https://gitea.com/gitea/act/pulls/14
• Artifact protocol compatible with Github's, some reports could be displayed in UI directly i.e. Golang standard coverage file Implement actions artifacts #22738
• Service support https://gitea.com/gitea/act/pulls/5
• Cron support chore(actions): support cron schedule task #26655
• Cache action https://gitea.com/gitea/act_runner/pulls/25
• The default plugin programming language is Javascript/Typescript, but we need to support more programming languages
  • Python https://gitea.com/gitea/act_runner/issues/44
  • Golang
  • Rust
  • PHP
  • More
• Check tab in the pull request to list results on the pull request page so that we don't need to go to actions page
• Composite workflow, a workflow file could reference another workflow as a step.

Although we will have an internal CI/CD system Gitea will still be open to integrating external systems and will provide more features to make the integration smooth. The below tasks do not belong to the internal CI/CD system but are very important for integrating the external CI/CD system

• A standard protocol to get tasks and get logs back https://gitea.com/gitea/actions-proto-def

[a1012112796]

tips about api for ci tools to repot detail message:

1. lint messages, can show them like code comment (but can marked as wrong by manager)
   the stype of data that provide by api

{
    commit: xxxxxxx
    path: "xxxxx/xxxxx",
    line: -1,
    type:  bug/ security/ codesmell/ info,
    detail: xxxxxxx,
   provider: xxxxx,
}

example in gitee

2. code coverage data like codecov

{
    commit: xxxxxxx
    path: "xxxxx/xxxxx",
    group: golang,    // only data in same group will be calculate together
   provider: xxxxx,
   start_line: xx,
   start_cols: xx,
   end_line: xx,
  end_cols: xx,
  matched: 0 -- not, .>1 -- yes
}

do some calculate and show some usefull data on ui.

3. detal log in ci steps, should show them in an individual page like gh actions.
   then user can design a sample ci tools which not need it's own ui :)
   style:

{
    commit: xxxxxxx
    step_name: aaa,
    parent_step_name: bbb,
   previous_step_name: ccc,
    log: xxxxx, or link to log data
   status: pending or runing or success ....
}

example on gh

[Be-ing]

Support github actions files and allow external agent to execute the action via Gitea's API and returned result as above results.

Can you clarify what this will entail? Would this mean GitHub Actions could run on a github.com mirror of a repo and the results would appear on a Gitea server?

[lunny]

That means Github actions could be running on Gitea(with this CI feature) servers(actually on user's agent, Gitea will only a coordinator and result saver).
Then mirrors/migrations of repositories from github/github enterprise could run the ci directly and no(or small) modification needed.

[a1012112796]

see

• https://docs.github.com/en/rest/reference/checks
• https://docs.github.com/en/rest/reference/code-scanning

[moqmar]

Would this be implemented by explicitly checking for a .github directory and sending those repos to the agent automatically on push using something like a global webhook? Because I think it would be great to not limit this to just GitHub Actions, but rather have a "global webhook" for pushes to an agent that can then decide what to do with that push - either using the GitHub Actions definitions, or for example running other CI implementations.

[lunny]

@moqmar Global webhooks have been implemented. The issue is to be compatible with actions for those migrations from github.

[ChristopherHX]

I have been working on an external webhook endpoint for github actions self-hosted runners to use with my gitea instance or locally on any pc. This is basically a fork of the runner to be able to reuse their code for a missing aspnetcore server. During my tests I was able to use gitea almost like a ghes instance, some actions like actions/checkout@v2 have had minor incompatibilities with authentication.

I would be glad to see endpoints for CI logs and have an integrated UI. Currently I have a basic react UI to display my logs, added as an external tab to gitea.

Experimental Usage
Download and extract https://github.com/ChristopherHX/runner.server/releases/latest for your system

Setup Gitea as an ghes endpoint, file bin/appsettings.json, change the url from http://localhost:5000 to a different http url / port

  "Runner.Server": {
    "GitServerUrl": "https://gitea:3042",
    "GitApiServerUrl": "https://gitea:3042/api/v1",
    "GitGraphQlServerUrl": null,
    "GITHUB_TOKEN": "Add your personal access token here",
    "ActionDownloadUrls": [
      {
        "TarballUrl": "https://gitea:3042/{0}/archive/{1}.tar.gz",
        "ZipballUrl": "https://gitea:3042/{0}/archive/{1}.zip"
      }
    ]
  }

Add <url of Runner.Server>/runner/server/_apis/v1/Message (http://localhost:5000/runner/server/_apis/v1/Message) as a webhook (select gitea) for your repo, to enable actions.

Setup https and use openid connect with gitea (Optional for Security, only working if the github actions server is using https)

• Create an oauth2 app (User settings)
• You will need to add the clientid and secret into bin/appsettings.json
• You will need a pem certificate pair (cert.pem (only a single certificate will work, no cert chain) , key.pem)

"Kestrel": {
    "Endpoints": {
      "HttpsFromPem": {
        "Url": "https://*:5001",
        "Certificate": {
          "Path": "./cert.pem",
          "KeyPath": "./key.pem"
        }
      }
    }
  },
  "ClientId": "ClientId of your Oauth app",
  "ClientSecret": "Client secret of your Oauth app",
  "Authority": "https://gitea:3042",

Add <url of Runner.Server>/signin-oidc (https://localhost:5001/signin-oidc) as redirect url for the OAuth app in gitea.

Start bin/Runner.Server(.exe)
Open <url of Runner.Server> (http://localhost:5000, https://localhost:5001) to see if a job was received

Configure self-hosted runners, https://github.com/ChristopherHX/runner.server#advanced-usage

EDIT 2021/10/26
I added a ( temporary ) public test instance of runner.server connected to try.gitea.io https://try.gitea.io/ChristopherHX/actions-on-gitea/commit/7ac850a3a96a7c60417c1306f50d49ea08d2e7d7

Press rerun workflow to see in progress job runs
EDIT 2021/02/02
Fix openid sample config to actually have working https with pem

[techknowlogick]

@ChristopherHX whoa! That's amazing!!

[Bobby99as]

This seems nice

[kbingham]

For this to work, the config cannot be inside .github or .gitea folder as that will cause conflict with original repository. Instead, we can have in a custom branch called .gitea-actions-branch which only contains the yaml config file for 'gitea action'.

Being able to store the configuration in a custom branch would help a lot for adding automation to open source projects that don't want to have specific CI files added to the core repo. The linux kernel for instance.

[lunny]

For this to work, the config cannot be inside .github or .gitea folder as that will cause conflict with original repository. Instead, we can have in a custom branch called .gitea-actions-branch which only contains the yaml config file for 'gitea action'.

Being able to store the configuration in a custom branch would help a lot for adding automation to open source projects that don't want to have specific CI files added to the core repo. The linux kernel for instance.

Maybe we can have some options:

1. .github/workflows or .gitea/workflows
2. <repo_name>.actions.git like <repo_name>.wiki.git
3. A setting of the repository to store multiple workflow files.
4. A global workflows feature, which you can create a workflow to includes files and repositories.

[siegenthalerroger]

I actually really like the idea of having a seperate repo analogous to the wiki repository for this. Would be interesting to explore other functionality that could be implemented in this way.

[kbingham]

Yup now I've been exploring more, I can see a separated repo for the workflows could be a very helpful way of doing this too. Then multiple repos (mirrors, forks, or separate repos for separate developers) can point to common workflow actions.

[dsseng]

Maybe after artifact upload is done, basic code intelligence (not as powerful as SourceGraph, but just goto definitions/references + browse file symbols) could be implemented. After finishing builds/tests the action might do something like rust-analyzer lsif . and upload it as a special artifact Gitea will build navigation from. This approach gives the most detailed code navigation, just as in IDE using the same language server as used to generate LSIF. See https://lsif.dev/ for more details regarding the format.

Didn't open a new issue because of dependency on artifacts

[prologic]

Are there any docs on built-in actions in 1.19.0? Even just a quick bullet point here in this thread? 🙏

[prologic]

Answering my own question, getting this up and running with Gitea v1.19.0 (from upstream Docker images) was pretty straight forward:

1. Add the following to your app.ini config (since if you're running an older version, it won't be there by default and disable by default?):

[actions]
ENABLED = true

2. Restart Gitea
3. Go to Site Administration -> Runners
4. Create new Runner and Copy the token
5. Setup and register a Runner

$ git clone https://gitea.com/gitea/act_runner.git
$ cd act_runner
$ make build
$ ./act_runner register
# follow the instructions
$ ./act_runner daemon

That's it! 🥳 Now just add workflows in .gitea/workflows/build.yml which use the same syntax as Github actions? (I guess so since the runner being used is the act_runner).

[prologic]

Happy to continue beta testing this (I also run an older Drone CI setup too!) and provide feedback.

One thing that springs to mind (which I'm hoping one of the developers can answer):

• How do I set-up Gitea + actions in a way where I can spawn runners on demand? (like Github does)

Basically, instead of having a static pool of runners (however I choose to run them), it would be nice to have a webhook (globally) that I can listen on and spawn a new runner on demand (say in a virtual machine or similar). Thanks! 🙏

[aminify]

Apparently in version 19.0.1 we should put something different in the app.ini file. Can someone help with that?
I put the below in it and actions weren't enabled. I used docker image by the way... I don't know if that's the source of issue. Really poor documentation so far...

[repo.actions]
ENABLED = true

[jimafisk]

@aminify I think it's just:

[actions]
ENABLED = true

https://docs.gitea.io/en-us/config-cheat-sheet/#actions-actions

[delvh]

No, the cheatsheet is still true.
You simply misunderstood something here:
the key DEFAULT_REPO_UNITS/DEFAULT_FORK_REPO_UNITS in the [repository] section used the value actions instead of repo.actions.
The section to enable actions is [actions].

[lunny]

All units configuration should use repo.actions, but the ini section is [actions].

[RynoM]

• How do I set-up Gitea + actions in a way where I can spawn runners on demand? (like Github does)

This would be nice! I've been tinkering with using it to deploy/restart my docker services. Would be nice if it could just do dynamic generation / access from outside for runner tokens. That way the container is more ephemeral.

Also its probably been said before, but an environment variable for enabling actions would be nice, instead of having to update app.ini.

[aminify]

I have setup the runner and everything and I have a docker build command to run in my action. Unfortunately I get /var/run/act/workflow/1: line 2: docker: command not found. Anyone knows the best way to enable docker inside the runner?

[panekj]

I have setup the runner and everything and I have a docker build command to run in my action. Unfortunately I get /var/run/act/workflow/1: line 2: docker: command not found. Anyone knows the best way to enable docker inside the runner?

You need to install it or use a docker image runner that has it installed

[aminify]

thanks @panekj
for others facing the same issue. I managed to fix this docker: command not found issue using ubuntu-latest:docker://catthehacker/ubuntu:act-22.04 as label argument for act-runner (in register phase)

[sorenisanerd]

Another missing part of Gitea's implementation of Github Actions is workflow commands in general, but most critically the add-mask command. My logs are riddled with secrets that anyone can see and steal.

[prologic]

Another missing part of Gitea's implementation of Github Actions is workflow commands in general, but most critically the add-mask command. My logs are riddled with secrets that anyone can see and steal.

Wait?! Why are you logging secrets in the first place? 🤔

[vquie]

Another missing part of Gitea's implementation of Github Actions is workflow commands in general, but most critically the add-mask command. My logs are riddled with secrets that anyone can see and steal.

I can confirm this. I use the 1password action to retrieve passwords. They are logged in all their beauty. Their code says add-mask though.

[JakobDev]

Wait?! Why are you logging secrets in the first place?

There are Programs who print API Keys for debug reasons with no Way to disable it. It also possible, that this may be leaked in a error message when something crashes, so it's generally a good Idea to mask this, just in case.

[kbingham]

Yup now I've been exploring more, I can see a separated repo for the workflows could be a very helpful way of doing this too. Then multiple repos (mirrors, forks, or separate repos for separate developers) can point to common workflow actions.

FWIW - Today I discovered gitlab can do this and it's all documented here:

• https://docs.gitlab.com/ee/ci/pipelines/settings.html#specify-a-custom-cicd-configuration-file
• https://gitlab.com/gitlab-org/gitlab/-/issues/14376

[prologic]

Wait?! Why are you logging secrets in the first place?

There are Programs who print API Keys for debug reasons with no Way to disable it. It also possible, that this may be leaked in a error message when something crashes, so it's generally a good Idea to mask this, just in case.

Those programs should be loaded up on a rocket and shot into sun! 😆 talk about security law practice 🤦‍♂️

[delvh]

I wonder how much more we want to abuse this one issue?
As far as I'm concerned the basic functionality exists, and the remains should be extracted into separate issues.
I don't think it's good that this issue is still open despite the main functionality being implemented already.
Issues should focus on one specific thing, otherwise they'll be pretty much never closed.