Add job tracking

[gabriel-samfira]

This branch changes the way garm decides when to spawn a new runner, by implementing some rudimentary job tracking. The bullet points of what is happening are:

• A new model that tracks jobs has been added
• When webhooks are sent by github as a result of a job event, we record that job and it's state in the database
• We no longer create a new runner when a "queued" job shows up, we just record the job
• When a job transitions to "in_progress", we call ensureIdleRunnersForOnePool() for the pool to which the runner that has picked up the job, belongs to. Doing so will replace the idle runner if a minimum idle runner value is set to something other than zero, while obeying max runners.
• A new function called consumeQueuedJobs() was added. This function will attempt to create a new runner in response to jobs that are in queued state. It will run before the consolidate() function. It's purpose is to attempt to consume all queued jobs by creating any needed runners until the queue is gone. There is a back-off period of 15 seconds before it will attempt to create a new runner. The consolidate loop runs every 5 seconds, so there will be at least ~10 seconds in which an idle runner has time to pick up the job. After that, it will add a new runner.

Some notes:

We cannot control which runners pick up which jobs. We may have pools of runners defined on multiple hierarchy levels that are suitable to sun a particular job. For example, if the job requests a runner with self-hosted, all pools on all levels (repo, org and enterprise) will be able to handle that job.

We may create a runner in pool A as a response to a queued job, but a runner from pool B may pick it up. The runner we spawned may pick up a completely different job than the one that prompted us to create the runner. This does not present an issue as long as all runners were spawned to handle jobs with the same label set. Even if we do end up with some dangling runners, the scale-down logic we have will clean those up after a short period of being idle. While there is no way to guarantee there will be no runner churn, it should be minimal.

For best results, make sure to use unique sets of labels to target runners in a particular pool. Something like this should work:

• self-hosted
• x64
• linux
• ubuntu-latest
• gpu

Or if you want to keep it breef, simply use:

• ubuntu-latest-gpu

If you want to target a particular pool, just add something to narrow down the possibilities just to that pool:

• ubuntu-latest-gpu
• ml-team

Fixes: #47
Related: #78 #73

[gabriel-samfira]

CC @maigl @MoritzKeppler

If you're feeling brave, feel free to test this out. This branch should, in theory, properly handle zero min idle runner pools, with a small max runner value, without loosing jobs. Once slots become available, new runners will be added to handle the remaining queued jobs.

[maigl]

sounds good .. we'll check it out, when we find some time .. (maybe not brave enough :))
But the new provider currently has higher priority.

[gabriel-samfira]

But the new provider currently has higher priority.

There is no rush at all. Whenever you get a chance. No issues popped up during testing for me, but your use case is more varied than mine. When you do get a chance to test this, ping back.

[MoritzKeppler]

first of all: thanks one more time for the PR!

some thoughts:

• wonder if we could run some component test that simulate various scenarios.
  Should be possible to mock GitHub and an external provider; little bit tricky with all the async activities...
• how will the algorithm work if garm is scaled horizontally?
• what will happen if we miss some of the events?

[gabriel-samfira]

first of all: thanks one more time for the PR!

some thoughts:

* wonder if we could run some component test that simulate various scenarios.
  Should be possible to mock GitHub and an external provider; little bit tricky with all the async activities...

Yup. Should be doable. I need to write proper tests for the whole pool manager.

* how will the algorithm work if garm is scaled horizontally?

Sadly, not well. There is currently no global lock. There is a plan to try out etcd as a store, which gives us some nice primitives to work with like watchers and global locks. But I have no ETA on that.

* what will happen if we miss some of the events?

They will linker on github until they expire. Sadly, that can't be helped. See this unnecessarily long comment for more details:

https://github.com/cloudbase/garm/pull/86/files#diff-0928304a4f9afb0bf84875a5a19d4d42692d7ed54843569166a233b61cbb7603R1190

Coincidentally, github itself does the same thing. When there is an issue with the gh runners, jobs started while the issue manifested will never be picked up. My guess is that internally they use the same hooks to queue the jobs for their own runners.

[maigl]

thx .. will try this out today :)

[gabriel-samfira]

thx .. will try this out today :)

Curious how it works out compared to the old workflow.