Provide scaler for GitHub Actions Runner

[tomkerkhove]

Proposal

Provide scaler for GitHub Actions Runner
https://github.com/actions/runner

Scaler Source

GitHub Actions Runner queue

Scaling Mechanics

Automatically scale out based on the pending actions to run.

Authentication Source

TBD

[melisatanrverdi]

Hi, I’ve been learning how to write a Keda scaler for a while. I wrote a basic scaler just to understand the scalers logic and wrote a blog about it, you can read my article here: https://melisatanrverdi.medium.com/how-do-i-cool-pods-with-keda-how-to-write-a-new-scaler-26b4671830bc
I am interested this issue now, could you please give more details about it?

[zroubalik]

Great article! :)

@tomkerhove is the right person that could give you more info.

[tomkerkhove]

The goal here is to autoscale self-hosted GitHub Actions runners to meet the current demand for it.

This would please a lot of people as per actions/runner#395.

In order to achieve that, we need to be able to get the related information:

• How do we get the # of pending actions to run on the agent? I'm checking with GitHub because I don't see an API for that (unless I'm mistaken).
• What are people using as Docker image today, because there isn't an official image yet

[ghost]

I've done something similar here https://github.com/devjoes/github-runner-autoscaler It's not a custom scaler. It's just a custom metrics APIService that you can point the ExternalMetrics scaler at and autoscale a stateful set up and down.

[tomkerkhove]

That looks interesting, thanks for sharing! Feel free to send a PR to https://github.com/kedacore/external-scalers if you want.

However, this seems to rely on Prometheus which is a dependency we'd like to avoid. Can you share a bit on where you get the metrics from?

[ghost]

Thanks! PR is here.

It doesn't rely on Prometheus, it just exports its own metrics which can be consumed by Prometheus if the user wants. Essentially it just exposes a custom metric like this:

kubectl get --raw '/apis/custom.metrics.k8s.io/v1beta1/namespaces/example-repo/Scaledactionrunners/example-repo/*' | jq .
{
  "kind": "MetricValueList",
  "apiVersion": "custom.metrics.k8s.io/v1beta1",
  "metadata": {
    "selfLink": "/apis/custom.metrics.k8s.io/v1beta1/namespaces/example-repo/Scaledactionrunners/example-repo/%2A"
  },
  "items": [
    {
      "describedObject": {
        "kind": "ScaledActionRunner",
        "namespace": "example-repo",
        "name": "example-repo",
        "apiVersion": "v1alpha1"
      },
      "metricName": "example-repo",
      "timestamp": "2021-04-13T10:14:35Z",
      "value": "2",
      "selector": {
        "matchLabels": {
          "cr_name": "example-repo",
          "cr_namespace": "example-repo",
          "cr_owner": "devjoes",
          "cr_repo": "example-repo",
          "wf_id": "7939907",
          "wf_name": "main",
          "wf_runs_on": "self-hosted",
          "wf_runs_on_self-hosted": "self-hosted"
        }
      }
    }
  ]
}

It then uses KEDA's metrics-api trigger to scale on items.0.value.

The operator in that repo just wraps up the creation of StatefulSets and ScaledObjects. But there is no reason why you couldn't just deploy the API server and CRDs and point KEDA at it if you wanted to.

[tomkerkhove]

Thanks for opening the PR!

So the statefulset is then in charge of getting the metrics with GitHub to see how many pending actions there are, is that correct?

[ghost]

So the stateful set just runs https://github.com/myoung34/docker-github-actions-runner
The operator generates a ScaledObject with a trigger like this:

  triggers:
    - authenticationRef:
        kind: ClusterTriggerAuthentication
        name: metrics
      metadata:
        authMode: tls
        targetValue: '1'
        url: >-
          https://metrics.foo.svc/apis/custom.metrics.k8s.io/v1beta1/namespaces/example-repo/Scaledactionrunners/example-repo/*
        valueLocation: items.0.value
      type: metrics-api

This queries a deployment called metrics, it is that deployment that checks the queued jobs

[tomkerkhove]

A-ha, now I'm starting to get it - Thanks!

Basically the runner itself has the metrics on the queued jobs, thanks!

[tomkerkhove]

They have added webhook support but that's not really ideal: https://github.blog/changelog/2021-09-20-github-actions-ephemeral-self-hosted-runners-new-webhooks-for-auto-scaling/

[hieumoscow]

@tomkerkhove there is a queue depth API @nilofer from GitHub is focusing on, hopefully available next month after their internal hackfest. I could make a PR to provide native scaler for GitHub Action

[tomkerkhove]

That would be a major help - Thanks!

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[chrisreddington]

Hey @tomkerkhove - Is there any progress on this one? Also interested, and happy to contribute if extra help is needed! Sounds like there are some limitations on the GitHub side to get this going though from what I'm reading.

[tomkerkhove]

As far as I know, unless that changed, there is no simple way to get metrics today. GitHub advises to use actions-runner-controller/actions-runner-controller in their documentation.

However, @nilofer might have an update on an API to poll as I'd love to have one next to our Azure Pipelines scaler.

[Pwd9000-ML]

For anyone on this thread who is interested, I wrote a little post/tutorial on Autoscaling self hosted GitHub runner containers with Azure Container Apps (ACA)

[tomkerkhove]

Nice workaround

[tomkerkhove]

I've been in contact with GitHub related to github/roadmap#555

[Eldarrin]

Needs some work, but I need a git scaler to sit alongside my azure pipelines scaler so writing one. It currently has the issue of if you have a lot of repos it will hit rate limits, so putting some code together to refine that bit.

[Eldarrin]

@tomkerkhove this is working in my dev clusters now. I haven't done a webhook version, this is a scale to zero, pull awaiting jobs version. Would you like this submitting? If so, I'll document it.

[JorTurFer]

do you have it working without webhooks? I mean, are you recovering the info using the rest API?
If yes, we'd love it

[Eldarrin]

Yeah, all using the rest api. Will document it and submit a PR. Cheers, Andy

[tomkerkhove]

Wonderful, thank you!

[zroubalik]

Implemented

[avdhoot]

FYI The new implementation of action-runner controller from github is promising.