Investigate GitHub Action runners on our infrastructure

[mmarchini]

Forking discussion from #2205 to keep that issue

---

@jkleinsc:

@rvagg I think moving over to GitHub actions could be a great way to reduce the maintenance load on the build WG as it is infrastructure you do not have to maintain. The self hosted runners would have to still be maintained, but I'm pretty sure the agent that runs on the self host runners is self updating. The only caveat that might be an issue is that GitHub doesn't recommend using self hosted runners on public repos: help.github.com/en/actions/hosting-your-own-runners/about-self-hosted-runners#self-hosted-runner-security-with-public-repositories. I do wonder though if docker containers were used if it would mitigate the security issue.

---

@mmarchini:

I'm interested in seeing a POC of a runner so we can compare maintenance burden with Jenkins. From a configuration perspective, setting up Actions jobs seems a lot more straightforward than Jenkins.

OTOH, we can't move things permanently from Jenkins to Actions right now as we first need to make sure we can cover the minimum compiler and operating system versions on Actions. Even if we can cover, there are several platforms not supported by Actions (FreeBSD, SmartOS, AIX, PPC, LinuxOne). With that being said, the runner is open sourced, so adding support for these platforms is possible (of course, I don't expect the WG to do it).

Another thing to consider is that our Release workflow relies on Jenkins today, so even if we decide to add runners for the supported platforms, we need to keep the Jenkins jobs to make sure Release still works. Having duplicated machines (one running Jenkins, one running Actions) seems like a bigger maintenance burden though.

Other small issues like lack of custom parametrized runs on Actions also prevents us from moving away from Jenkins (stress tests, for example).

I'm still hopeful for GitHub Actions, but any move in that direction means a bigger change on how we operate and deserves it's own Strategic Initiative. Which is why I'm interested in seeing how a POC will play out, we might find out it's not worth starting any migrations here, or we might find out the maintenance burden is significantly lower, justifying a migration.

[richardlau]

I'm interested in seeing a POC of a runner so we can compare maintenance burden with Jenkins. From a configuration perspective, setting up Actions jobs seems a lot more straightforward than Jenkins.

OTOH, we can't move things permanently from Jenkins to Actions right now as we first need to make sure we can cover the minimum compiler and operating system versions on Actions. Even if we can cover, there are several platforms not supported by Actions (FreeBSD, SmartOS, AIX, PPC, LinuxOne). With that being said, the runner is open sourced, so adding support for these platforms is possible (of course, I don't expect the WG to do it).

The runner may be open sourced but would require rewriting in something more portable than .NET core:

• Support for PowerPC and Z series CPU architectures actions/runner#296
• Expanded platform support: Consider porting to go or typescript actions/runner#243

[mmarchini]

On the security concerns: running on containers doesn't prevent issues on OS X since OS X doesn't really have containers (Docker runs on a Linux VM). We would also be vulnerable to zero-day container escape vulnerabilities as well as any speculative execution vulnerability, and so forth.

The security model of Actions is not so different from Jenkins in that regard, we just need to avoid running self-hosted runners on pull_request events. We still need collaborators to start a workflow manually in that case. There are blockers on how to do that though, since all events on a Pull Request from a fork are read-only, which means a collaborator can't comment on the PR and trigger an Action. We could use the github-bot as a bridge though (it parses the comment and trigger the workflow using a webhook).

[rvagg]

We're migrating to Orka for macOS, which is container-like, MacStadium have built a whole image + ephemeral VM infrastructure that acts like contains for practical purposes. That might help, except the orchestration isn't something you can just plug in to as if it were Docker or some other common container product. I don't know how the GA agent is doing it but I also wouldn't be surprised if MacStadium have something already in place to work with it via Orka.

[AshCripps]

ORKA does seem to have some intergration with GitHub Actions.

See: https://orkadocs.macstadium.com/docs/github-actions and https://github.com/macstadium/orka-integrations/blob/master/GitHub/single-self-hosted-runner.md.

Seems to be the runner is permament and not ephemeral like the jenkins one would be

[mmarchini]

I would like to give it a try with a simple Linux x64 Runner to learn what we can expect from it and any limitations we might face. How can I get a fresh Linux (Ubuntu server 18.04 or 20.04 would be fine) instance to try it on?

[mmarchini]

@nodejs/build-infra any objections? Might be a good task for me to learn how to spin up instances.

[mmarchini]

For context, I intend to use this runner either for the Commit Queue/Auto Start CI actions (as it might make schedule actions more consistent, although that's just theory), or for the Linter Actions.

[rvagg]

Fine by me, stay minimal, clean up after yourself. I think you should have a rough idea for where might be best to spin up new VMs now for this kind of experimentation.

[github-actions]

This issue is stale because it has been open many days with no activity. It will be closed soon unless the stale label is removed or a comment is made.

[nschonni]

Saw this was announced today https://github.blog/2022-11-10-introducing-github-actions-importer/ which might help figure out if at least parts could be migrated "easily"

[mbrevda]

Hi - I noticed a issue where releasing a fix for a regression was held up due to an infra issue and was wondering if Github Actions could lessen the burden on maintainers (which brought me here). With Actions being managed (and free to Open Source projects), this could help reduce the burden on the infra team. In this particular case, it would seem that Action's ephemeral builds would have prevented the above issue (running out of disk space).

While appreciating that there are unique requirements to a project like Node that may not be immediately apparent to outsiders, perhaps there are small steps the community can take that would be beneficial to the team. Is this a topic the relevant teams are interested in pursuing? Can community involvement accelerate the conversation (in the form of technical experience or a PoC, perhaps)?