Skip to content

Releases: cloudbase/garm

garm v0.1.5

08 Aug 13:41
@gabriel-samfira gabriel-samfira
v0.1.5
237afdb
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
garm v0.1.5 Latest
Latest

Welcome to GARM v0.1.5!

Before we dive into the highlights, as always a few notes.

Database upgrade ⚠️

This version will execute a database migration. While we try not to break anything when making database migrations, it's always a good idea to create a backup of your DB as well as any configuration files. If anything goes wrong, you can revert the changes. Database downgrades are not supported. If you need to go back to a previous version, please also restore the DB from backup.

To safely create a backup of you DB, you can use the sqlite3 CLI:

# Change the path to your DB to the one configured in your config.toml
sqlite3 /etc/garm/garm.db
.backup /path/to/backup/location/garm-backup.db

Breaking changes ⚠️

This version has moved a number of options from the config file, to the database. During the upgrade process from v0.1.4 to v0.1.5 please do not make any changes to your current config. GARM will read the config and migrate any of the settings present there, to the DB, automatically. After you start GARM with the new version and the migration has finished (it should be just a matter of seconds), you can then edit your config and remove the relevant fields.

Things that have been moved from the config to the database:

  • The metadata_url, callback_url and webhook_url settings have now been moved to the database in the controller table. These settings can now be updated using the garm-cli command line tool.
  • Github credentials - credentials are now created via the garm-cli command line tool. Details bellow.

Highlights

This version is packed with features and changes. Although GARM can handle a large amount of runners with ease, we are working towards enabling it to eventually scale out. As a result, we've started moving some of the settings that were in the config file, to the database. The goal is to eventually have a single source of truth for those settings and enable GARM to react to any changes immediately without the need to reload the app. This is just one motivation behind the changes in this release, but as you'll see, there are many other changes that enable you to do more with GARM.

GitHub endpoints

With the migration of credentials to the database, GARM now has the notion of "github endpoints". A github endpoint represents a deployment of GHES or GitHub itself. It is a way to tell GARM to which API endpoints it needs to connect in order to use the supplied credentials or to execute operations against a particular entity (repo, org or enterprise).

All github credentials and all entities are now bound to a github endpoint. This ensures consistency and correctness when dealing with these types of resources.

See the github endpoint section of the documentation.

Github App Support

We now have the ability to configure GARM to use GitHub Apps to access the github API. This gives us a much higher rate limit for API calls (15.000/hour compared to 5000/hour as is the case of PATs).

Check out the github credentials documentation section for more info on how to configure app credentials, as well as the required credentials for GitHub Apps.

All sensitive info is encrypted at rest and is never returned by the API.

Runner default labels

Starting with version v2.305.0 of the GitHub actions runner, the default lables ($ARCH, $OS and self-hosted) are no longer added if the --no-default-labels flag is used. GARM now takes advantage of this feature to create runners without default labels. The default labels can still be added explicitly if you require them.

Pool balancing strategies

You can now specify two modes of balancing for jobs that get handled by a configured entity. The balancing mode can be set on the entity itself, and this will influence how the pool manager for that entity will behave. The two modes of operation are:

  • roundrobin - During the consolidation loop, if we have multiple queued jobs, the roundrobin strategy will balance each job to a different matching pool. This is useful if you want to spread the workload evenly across multiple clouds or regions. This is the default.
  • pack - The pack strategy will try to fill up the first pool before moving on to the next, This is useful if you have cheap runners you want to use first before moving on to pools of more expensive runners.

To set the balancing stratedy for a repo, org or enterprise you can use the following command:

garm-cli repo update --pool-balancer-type pack <repo_id>

Additionally, pools now have a priority field. This priority field is used when sorting pools that match jobs which need to be handled. You can set the priority when you create the pool or you can update it later. The higher the number, the more likely the pool is to be at the beginning of the list.

A new events websocket endpoint

We now have a new websocket endpoint that can be used by projects which integrate with GARM. This websocket endpoint streams database events as they happen in GARM itself, allowing applications to not need to poll for updates.

Here is a demo of an earlier implementation, in action:

asciicast

For details regarding the current implementation and how to use it, check out the documentation.

There are many other fixes and features added to this release, a lot of which are internal to GARM but they do have a positive impact on stability and reliability. I invite you to check out the full list of changes, bellow.

Thanks to everyone that has contributed to this release, be it via pull requests, raising issues or helping us debug weird behavior!

What's Changed

Read more

Contributors

gabriel-samfira, jepio, and 2 other contributors
Assets 14
0 Join discussion

garm v0.1.4

12 Feb 20:07
@gabriel-samfira gabriel-samfira
v0.1.4
747604e
Compare
Choose a tag to compare
garm v0.1.4

Welcome to GARM version v0.1.4!

Documentation relevant to this version is available at: https://github.com/cloudbase/garm/tree/v0.1.4. The main branch contains code that is still under development, and the documentation is relevant to that code. When deploying stable versions, please view the docs available under the relevant tag.

Before we dive into the highlights, there are a couple of things we need to mention.

Breaking change warning ⚠️

This version removes the LXD internal provider in favor of two new external providers:

This was done mainly due to the licensing change of LXD. If you're using LXD and are upgrading from previous versions, you must download the LXD external provider and configure it.

Another breaking (sort of) change is in the garm-cli. The --force flag in the runner remove command, is no longer mandatory when removing a runner. Moreover, the --force flag now ignores provider errors. In previous versions, if a provider erred when removing a runner, GARM would keep trying until it succeeded. The problem was that if a provider was misconfigured, we would be stuck with a runner in error state until re manually removed it from the database. The --force flag allows us to remove a runner from GARM even if the provider returns an error.

Database upgrade ⚠️

This version will execute a database migration. While we try not to break anything when making database migrations, it's always a good idea to create a backup of your DB as well as any configuration files. If anything goes wrong, you can revert the changes.

To safely create a backup of you DB, you can use the sqlite3 CLI:

# Change the path to your DB to the one configured in your config.toml
sqlite3 /etc/garm/garm.db
.backup /path/to/backup/location/garm-backup.db

Highlights

This release is packed with optimizations, changes and a couple of nice features.

JIT runners

The main highlight of this release is the ability to use just-in-time self-hosted runners. To accommodate this change, GARM now has more robust metadata endpoints that can serve the needed JIT files. This will most likely be expanded in the future to include setup scripts for clouds where we have userdata size constraints.

JIT runners allows us to avoid sending a runner registration token over the wire. Registration tokens can be used to register multiple runners and have a validity of one hour. A bad actor may intercept this token and use it to register their own runners. With JIT runners, only one runner may use the credentials at any given point in time. The credentials are fetched from GARM by the runner when it spins up.

This feature can be disabled in the provider config by specifying:

[[provider]]
disable_jit_config = true

easily access controller info

We now have the ability to gain more insight into how GARM is configured. We've added a new command which can show us some info about the controller:

ubuntu@garm:~$ garm-cli controller-info show
+------------------------+----------------------------------------------------------------------------+
| FIELD                  | VALUE                                                                      |
+------------------------+----------------------------------------------------------------------------+
| Controller ID          | a4dd5f41-8e1e-42a7-af53-c0ba5ff6b0b3                                       |
| Hostname               | garm                                                                       |
| Metadata URL           | https://garm.example.com/api/v1/metadata                                   |
| Callback URL           | https://garm.example.com/api/v1/callbacks                                  |
| Webhook Base URL       | https://garm.example.com/webhooks                                          |
| Controller Webhook URL | https://garm.example.com/webhooks/a4dd5f41-8e1e-42a7-af53-c0ba5ff6b0b3     |
+------------------------+----------------------------------------------------------------------------+

Easy webhook installation

Installing webhooks for entities (repos, orgs, enterprises) can be a chore. In this version, we've added the ability to let GARM install the needed webhook, with the correct settings by running a simple command. To make use of this feature, the PAT you're using must have access to admin:org_hook:

ubuntu@garm:~$ garm-cli org add \
    --credentials org_token \
    --name exampleOrg \
    --install-webhook  \
    --random-webhook-secret
+----------------------+--------------------------------------+
| FIELD                | VALUE                                |
+----------------------+--------------------------------------+
| ID                   | b90911e1-8727-4bb7-a1eb-96855d73a27b |
| Name                 | exampleOrg                           |
| Credentials          | org_token                            |
| Pool manager running | true                                 |
+----------------------+--------------------------------------+

This command will add the org exampleOrg to your GARM controller and install a webhook for it with a random webhook secret. You can also install a webhook for an existing organization or repository:

ubuntu@garm:~$ garm-cli repo webhook install b90911e1-8727-4bb7-a1eb-96855d73a27b
+--------------+----------------------------------------------------------------------------+
| FIELD        | VALUE                                                                      |
+--------------+----------------------------------------------------------------------------+
| ID           | 449734752                                                                  |
| URL          | https://garm.example.com/webhooks/a4dd5f41-8e1e-42a7-af53-c0ba5ff6b0b3     |
| Events       | [workflow_job]                                                             |
| Active       | true                                                                       |
| Insecure SSL | false                                                                      |
+--------------+----------------------------------------------------------------------------+

To view the status of a webhook:

ubuntu@garm:~$ garm-cli repo webhook show b90911e1-8727-4bb7-a1eb-96855d73a27b
+--------------+----------------------------------------------------------------------------+
| FIELD        | VALUE                                                                      |
+--------------+----------------------------------------------------------------------------+
| ID           | 449734752                                                                  |
| URL          | https://garm.example.com/webhooks/a4dd5f41-8e1e-42a7-af53-c0ba5ff6b0b3     |
| Events       | [workflow_job]                                                             |
| Active       | true                                                                       |
| Insecure SSL | false                                                                      |
+--------------+----------------------------------------------------------------------------+

You can also uninstall a webhook:

ubuntu@garm:~$ garm-cli repo webhook uninstall b90911e1-8727-4bb7-a1eb-96855d73a27b

Webhooks installed by the above commands are always namespaced to the controller ID you get when you run garm-cli controller-info show. This way, when we remove a webhook, we don't accidentally remove someone else's hook. You can also manually install we webhook just like before. You don't need to namespace it to the controller ID. GARM will see the webhook and let you know if the hook is already installed.

Removing a repo or org will also clean up the webhook if it was namespaced to our controller. You can opt to keep the hook by passing the --keep-webhook flag.

Webhook installation is not available for enterprises.

Structured logging

GARM has now switched to the slog standard package for structured logging. As part of this change, we now have a dedicated [logging] config section where you can set the log level, log format, enable the log streamer, etc. Check out the sample config for more info.

Other updates

There are a lot of bug fixes and stability updates that should make this version behave better in relation to the GitHub API.

What's Changed

Read more

Contributors

gabriel-samfira, ionutbalutoiu, and 2 other contributors
Assets 14
Loading
0 Join discussion

garm v0.1.4-rc1

18 Dec 19:25
@gabriel-samfira gabriel-samfira
v0.1.4-rc1
50f691e
Compare
Choose a tag to compare
garm v0.1.4-rc1 Pre-release
Pre-release

Welcome to GARM version v0.1.4-rc1!

This is a pre-release of GARM and it is packed with changes.

Before we dive into the highlights, there are a couple of things we need to mention.

Breaking change warning ⚠️

This version removes the LXD internal provider in favour of two new external providers:

This was done mainly due to the licensing change of LXD. If you're using LXD and are upgrading from previous versions, you must download the LXD external provider and configure it.

Database upgrade ⚠️

This version will execute a database migration. While we try not to break anything when making database migrations, it's always a good idea to create a backup of your DB as well as any configuration files. If anything goes wrong, you can revert the changes.

To safely create a backup of you DB, you can use the sqlite3 CLI:

# Change the path to your DB to the one configured in your config.toml
sqlite3 /etc/garm/garm.db
.backup /path/to/backup/location/garm-backup.db

Highlights

This release is packed with optimizations, changes and a couple of nice features.

JIT runners

The main highlight of this release is the ability to use just-in-time self-hosted runners. To accommodate this change, GARM now has more robust metadata endpoints that can serve the needed JIT files. This will most likely be expanded in the future to include setup scripts for clouds where we have userdata size constraints.

JIT runners allows us to avoid sending a runner registration token over the wire. Registration tokens can be used to register multiple runners and have a validity of one hour. A bad actor may intercept this token and use it to register their own runners. With JIT runners, only one runner may use the credentials at any given point in time. The credentials are fetched from GARM by the runner when it spins up.

This feature can be disabled in the provider config by specifying:

[[provider]]
disable_jit_config = true

easily access controller info

We now have the ability to gain more insight into how GARM is configured. We've added a new command which can show us some info about the controller:

ubuntu@garm:~$ garm-cli controller-info show
+------------------------+----------------------------------------------------------------------------+
| FIELD                  | VALUE                                                                      |
+------------------------+----------------------------------------------------------------------------+
| Controller ID          | a4dd5f41-8e1e-42a7-af53-c0ba5ff6b0b3                                       |
| Hostname               | garm                                                                       |
| Metadata URL           | https://garm.example.com/api/v1/metadata                                   |
| Callback URL           | https://garm.example.com/api/v1/callbacks                                  |
| Webhook Base URL       | https://garm.example.com/webhooks                                          |
| Controller Webhook URL | https://garm.example.com/webhooks/a4dd5f41-8e1e-42a7-af53-c0ba5ff6b0b3     |
+------------------------+----------------------------------------------------------------------------+

Easy webhook installation

Installing webhooks for entities (repos, orgs, enterprises) can be a chore. In this version, we've added the ability to let GARM install the needed webhook, with the correct settings by running a simple command. To make use of this feature, the PAT you're using must have access to admin:org_hook:

ubuntu@garm:~$ garm-cli org add \
    --credentials org_token \
    --name exampleOrg \
    --install-webhook  \
    --random-webhook-secret
+----------------------+--------------------------------------+
| FIELD                | VALUE                                |
+----------------------+--------------------------------------+
| ID                   | b90911e1-8727-4bb7-a1eb-96855d73a27b |
| Name                 | exampleOrg                           |
| Credentials          | org_token                            |
| Pool manager running | true                                 |
+----------------------+--------------------------------------+

This command will add the org exampleOrg to your GARM controller and install a webhook for it with a random webhook secret. You can also install a webhook for an existing organization or repository:

ubuntu@garm:~$ garm-cli repo webhook install b90911e1-8727-4bb7-a1eb-96855d73a27b
+--------------+----------------------------------------------------------------------------+
| FIELD        | VALUE                                                                      |
+--------------+----------------------------------------------------------------------------+
| ID           | 449734752                                                                  |
| URL          | https://garm.example.com/webhooks/a4dd5f41-8e1e-42a7-af53-c0ba5ff6b0b3     |
| Events       | [workflow_job]                                                             |
| Active       | true                                                                       |
| Insecure SSL | false                                                                      |
+--------------+----------------------------------------------------------------------------+

To view the status of a webhook:

ubuntu@garm:~$ garm-cli repo webhook show b90911e1-8727-4bb7-a1eb-96855d73a27b
+--------------+----------------------------------------------------------------------------+
| FIELD        | VALUE                                                                      |
+--------------+----------------------------------------------------------------------------+
| ID           | 449734752                                                                  |
| URL          | https://garm.example.com/webhooks/a4dd5f41-8e1e-42a7-af53-c0ba5ff6b0b3     |
| Events       | [workflow_job]                                                             |
| Active       | true                                                                       |
| Insecure SSL | false                                                                      |
+--------------+----------------------------------------------------------------------------+

You can also uninstall a webhook:

ubuntu@garm:~$ garm-cli repo webhook uninstall b90911e1-8727-4bb7-a1eb-96855d73a27b

Webhooks installed by the above commands are always namespaced to the controller ID you get when you run garm-cli controller-info show. This way, when we remove a webhook, we don't accidentally remove someone else's hook. You can also manually install we webhook just like before. You don't need to namespace it to the controller ID. GARM will see the webhook and let you know if the hook is already installed.

Removing a repo or org will also clean up the webhook if it was namespaced to our controller. You can opt to keep the hook by passing the --keep-webhook flag.

Webhook installation is not available for enterprises.

Other updates

There are a lot of bug fixes and stability updates that should make this version behave better in relation to the GitHub API. I'm pretty sure I'm forgetting something, but you can consult the list bellow for a full list of changes.

In any case, take it for a spin! If you run into any issue, feel free to open a new issue.

What's Changed

Read more

Contributors

gabriel-samfira, ionutbalutoiu, and 2 other contributors
Assets 14
Loading
0 Join discussion

garm v0.1.3

25 Jul 20:05
@gabriel-samfira gabriel-samfira
v0.1.3
d2fb5e3
This commit was signed with the committer’s verified signature.
gabriel-samfira Gabriel
GPG key ID: 7D073DCC2C074CB5
Learn about vigilant mode.
Compare
Choose a tag to compare
garm v0.1.3

Welcome to GARM version v0.1.3!

This release brings a number of really nice improvements.

Highlights

Thanks to @mihaelabalutoiu, we now have proper OpenAPI annotations, as well as a full client generated from them. The GARM CLI has switched to this new client, which is located in the root of the project. If you plan to integrate with GARM in any way, I encourage you to use this client instead of the old, hand-written one we previously had.

Another interesting change in this release is the fact that we split some functionality from GARM itself, into an external package. This new package will hold functionality that is common to GARM itself and external providers.

As part of this migration we also added the ability to:

  • Completely override the runner installation script template. This can be done with any provider that leverages the new external package and calls the helper functions that generate the install script. This includes the current LXD, OpenStack and Azure providers.
  • Send additional template context (if overriding the install template)
  • For Linux only (for now):
    • Add pre-install scripts. These scripts are executed in alphabetical order before the runner install script is run.

This can all be done via extra-specs.

Upgrade notes

This update does not come with database schema changes, however, we do recommend you update any external providers to their latest versions if you update GARM.

What's Changed

Full Changelog: v0.1.2...v0.1.3

Contributors

gabriel-samfira, ionutbalutoiu, and mihaelabalutoiu
Assets 14
Loading

garm v0.1.2

06 Jul 06:41
@gabriel-samfira gabriel-samfira
v0.1.2
a0a5611
This commit was signed with the committer’s verified signature.
gabriel-samfira Gabriel
GPG key ID: 7D073DCC2C074CB5
Learn about vigilant mode.
Compare
Choose a tag to compare
garm v0.1.2

Welcome to garm version v0.1.2!

This release brings a a lot of performance improvements and bug fixes.

Highlights

The most notable change in this release is in the way garm reacts to jobs sent by GitHub. Previously, garm reacted to new web hooks, but never recorded them in any way. When a queued hook came in, garm would attempt to create a new runner for it. The outcome of that action would depend on a few factors like if we had room for a new runner (max-runners), or if we had a pool that matched the requested tags.

One other shortcoming of the old approach was the fact that if we had multiple hierarchy levels configured (repo, org, enterprise), they would all receive the web hook with the queued job and would each create one runner for the same job. This had the potential to spin up resources for no reason.

This release introduces a new job tracking feature. From now on, garm will record all jobs that GitHub sends and the entity that received them (repo, org, enterprise). It will then attempt to periodically (roughly every 5 seconds) consume the jobs found in queued state and spin up runners for them. If we have multiple hierarchy levels configured, they will contend to spin up a runner for a job, but in the end only one of them will spin up a runner. This reduces the number of runners that get spun up for the same job. Job tracking also means that if a pool reaches max-runners we no longer lose jobs. We will eventually spin up a runner if the job is still in queued state. Which also means that it should be safe to use pools with min-idle-runners set to zero.

Other highlights

  • WAL is now enabled for the DB store. This was an oversight that caused load on large scale deployments.
  • Aditional indexes have been created for further speedups
  • Pool manager loops have been refactored. Each operation now runs in their own loop.
  • If multiple pools match a certain label, garm will now attempt to round-robin runners in matching pools

Upgrade notes

This release brings a number of changes to the database. Please back up your sqlite database before updating. The procedure is simple:

systemctl stop garm
cp /etc/garm/garm.db   /etc/garm/garm.db.backup
# update garm then start
systemctl start garm

What's Changed

New Contributors

Full Changelog: v0.1.1...v0.1.2

Contributors

gabriel-samfira, SystemKeeper, and 2 other contributors
Assets 10
Loading

garm v0.1.1

16 Jun 14:47
@gabriel-samfira gabriel-samfira
v0.1.1
481ce34
This commit was signed with the committer’s verified signature.
gabriel-samfira Gabriel
GPG key ID: 7D073DCC2C074CB5
Learn about vigilant mode.
Compare
Choose a tag to compare
garm v0.1.1

Welcome to garm version v0.1.1!

This release adds some resiliency fixes to garm.

Highlights

  • Instances that fail to set up their GitHub agent and transition to failed are now reaped.
  • The cloud-init userdata now retries setting up the github agent for up to 5 times before giving up
  • You can now specify extra_packages extra specs with the LXD provider, which allows you to install additional packages on runner nodes. (#103)

What's Changed

New Contributors

Full Changelog: v0.1.0...v0.1.1

Contributors

HippocampusGirl, gabriel-samfira, and mihaelabalutoiu
Assets 2
Loading

garm v0.1.0

08 Jun 13:56
@gabriel-samfira gabriel-samfira
v0.1.0
dd1569a
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
garm v0.1.0

This is the first versioned release of garm.

Garm is still under development and may change drastically before it reaches a stable version, but we will always try to keep it as functional and as backwards compatible as possible.

What's Changed

New Contributors

Full Changelog: https://github.com/cloudbase/garm/commits/v0.1.0

Contributors

icadariu, gabriel-samfira, and 5 other contributors
Assets 2
Loading