fix/pvc — Support for multi-node scheduling and S3-backed working directories (S3 Mountpoint CSI)

[lbeckman314]

Overview ⚙️

This PR includes support running the Server, Worker, and Executor across different nodes with an S3-backed working directory (via the S3 Mountpoint CSI Driver)— this driver allows the Worker and Executor to be scheduled on different nodes while sharing the same “working directory” stored in a single S3 bucket!

• The Funnel Server maintains a Persistent Volume for the Task Database (e.g. MongoDB)
• The Funnel Worker and Executors share the same ephemeral volume (e.g. S3-backed PVC)
• The Executor Outputs are uploaded to their appropriate endpoints (e.g. S3 buckets)
• The shared Funnel Worker/Executor volume is deleted after all the Executors are completed

Next Steps 🌀

• Run S3 Nextflow Workflow (reproducible_example.nf) and verify results

Testing Notes✍️

Steps for a basic S3 testing demonstrating:

• Scheduling over three different nodes between the Funnel Server, Worker, and Executor
• S3 Working Directory (via the S3 Mountpoint CSI Driver, recorded in the attached screenshots)
   

1. Update Helm (to latest version — v0.1.13):

➜ help repo update ohsu

➜ helm search repo ohsu

NAME            CHART VERSION   APP VERSION     DESCRIPTION
ohsu/funnel     0.1.13          1.11.0          A toolkit for distributed task execution ⚙️

 
2. Deploy Funnel:

➜ helm upgrade --install funnel ohsu/funnel --namespace funnel --values Secrets/aws.conf

 
3. Port Forward:

➜ kubectl port-forward svc/funnel 8000:8000 --namespace funnel

 
4. Submit Task

➜ funnel task create s3.json
ct0kaggb7bajfrh134e0

 
5. Get Pods (different nodes)

➜ kubectl get pods -o wide --namespace funnel
NAME                           READY   STATUS      NODE
ct0kaggb7bajfrh134e0-0-dmlm9   0/1     Completed   ip-172-31-174-49.us-west-2.compute.internal
ct0kaggb7bajfrh134e0-twt6m     0/1     Completed   ip-172-31-167-25.us-west-2.compute.internal
funnel-6686f75b5f-gdr2j        1/1     Running     ip-172-31-174-157.us-west-2.compute.internal

6. Verify Task

➜ funnel task get ct0kaggb7bajfrh134e0 --view MINIMAL
{
  "id": "ct0kaggb7bajfrh134e0",
  "state": "COMPLETE"
}

Development 🧑‍💻

Stay tuned for this feature's development! This feature is being tracked with the following:

• GitHub Branch: https://github.com/ohsu-comp-bio/funnel/tree/feature/pvc

• Pull Request: fix/pvc — Support for multi-node scheduling and S3-backed working directories (S3 Mountpoint CSI) #1107

[netlify]

✅ Deploy Preview for ohsu-comp-bio-funnel canceled.

Name	Link
🔨 Latest commit	910c395
🔍 Latest deploy log	https://app.netlify.com/sites/ohsu-comp-bio-funnel/deploys/6757aa5857fbe60008f00456