Testing infrastructure with Github Actions using custom-runners on nf-core infra

[abhi18av]

The goal of this task is to

• Use GitHub actions
• Setup a test nomad cluster
• Use CSI-hostpath volumes

[abhi18av]

Hi @tomiles and @matthdsm , please feel free to add details/subtasks to this task as you see fit. I've marked this as part of v0.1.0

@jagedn and I would focus on the local dev cluster and the mock testing setup for the initial milestones.

[abhi18av]

Adding notes after meeting with @jhaezebr and @matthdsm

Test-Cluster Phase-1: Nomad cluster on Azure

• Host Volume / CSI volume setup using Azure FileShare
• One Azure VM (Bastian host)
• Minimal Nomad ACL setup
• Share the tf codebase on Github.com
• The cluster has 6-node setup ( 3-servers and 3-clients )
• Tweaks for the TF setup regarding nodes
  - Bastion Host => Standard_B8ms
  - Server => Standard_B2ms
  - Clients => Standard_B8ms / B16ms
• Bastion host tweaks
  • Nextflow
  • Java-17 Amazon Correto-LTS (ideally via SDKman)
  • Docker
  • Conda
• [ ]

Test-Cluster Phase-2: Multiple nodes and Real World ACLs

• Github runner for the Azure nomad dev-cluster
• Iterate on ACL for real world setup https://developer.hashicorp.com/nomad/docs/other-specifications/acl-policy
• (LATER) A Nomad TLS certificate
• (LATER) Hybrid cluster (mixed client types GPU/CPU)

[abhi18av]

Sync meeting :
@jhaezebr , @tomiles , @jagedn , @abhi18av , @matthdsm

Milestone-2 (initiating now)

• Jules: Github runner for the Azure nomad dev-cluster
• Jules: Iterate on the minimal ACL policy starting from Error running nf-nomad with acl enabled #56 (comment) on az-nomadlab
• Jules: A Nomad TLS certificate
• Tweak the Terraform template for reducing the head nodes and compute nodes. Maintain the bastion host as a stable basis.

Milestone-3

• (LATER) Hybrid cluster (mixed client types GPU/CPU)

[jagedn]

for the record

from now we have a GH action able to run pipelines using the Azure cluster, see https://github.com/nextflow-io/nf-nomad/actions/runs/10614308658

[jhaezebr]

for the record

from now we have a GH action able to run pipelines using the Azure cluster, see https://github.com/nextflow-io/nf-nomad/actions/runs/10614308658

I'm looking into setting up scripts so that the runner creates and destroys the nomad cluster each time, so you can test against a nomad cluster