dask-gateway deployment

infrastructure/cluster/flux-v2/dask/README.md

@@ -0,0 +1,32 @@
+This deploymewnt allows multiple users to spawn multiple dask clusters by creating a scheduler each time a user asks for it, following the logic o fthe [Dask Gateway](https://gateway.dask.org/). 
+The deployment is done on top of the `jhub` namespace where the deployment of the Jupyterhub service is. The Dask packages are available inthe image called Dask at notebooks' start up. 
+If you decide to select this image, you can run a notebook and check that you can spawn a cluster by running:
+
+```
+import dask
+import dask.distributed
+from dask_gateway import Gateway
+from dask_gateway import GatewayCluster
+import dask.array as da
+import requests
+from dask_gateway import scheduler_preload
+import os
+```
+
+To check the service of the `dask-gateway` is reachable, run:
+
+```
+requests.get("http://traefik-dask-dask-gateway/api/health").content
+```
+
+And then connect to the Gateway:
+
+```
+gateway = Gateway(address="http://traefik-dask-dask-gateway/",
+    auth="jupyterhub")
+gateway.list_clusters()
+cluster = gateway.new_cluster()
+cluster.scale(4)  # to create 4 worker nodes
+```
+
+You will see the scheduler and the worker pods appearing in the k8s cluster under the `jhub` namespace. 

infrastructure/cluster/flux-v2/dask/daskhub-charts.yaml → infrastructure/cluster/flux-v2/dask/dask-gateway-chart.yaml

@@ -1,8 +1,9 @@
+# apiVersion: source.toolkit.fluxcd.io/v2beta1
 apiVersion: source.toolkit.fluxcd.io/v1beta1
 kind: HelmRepository
 metadata:
-  name: daskhub-charts
-  namespace: daskhub
+  name: dask-gateway-charts
+  namespace: jhub
 spec:
   interval: 10m
   url: https://helm.dask.org/

infrastructure/cluster/flux-v2/dask/dask-gateway-release.yaml

@@ -0,0 +1,59 @@
+apiVersion: helm.toolkit.fluxcd.io/v2beta1
+kind: HelmRelease
+metadata:
+  name: dask-gateway
+  namespace: jhub
+  annotations:
+    flux.weave.works/automated: "false" 
+
+spec:
+  releaseName: dask
+  interval: 5m
+  chart:
+    spec:
+      sourceRef:
+        kind: HelmRepository
+        name: dask-gateway-charts
+        namespace: jhub
+      interval: 1m
+      chart: dask-gateway
+      # version: 2023.1.0 # support for python 3.11
+      version: 2022.4.0 # support for python 3.9
+
+  valuesFrom:
+    - kind: Secret
+      name: api-dask-dask-gateway
+      valuesKey: jupyterhub-api-token
+      targetPath: gateway.auth.jupyterhub.apiToken
+
+
+  values:
+
+    gateway:
+      # extraConfig:
+      #   clusteroptions: |
+      #       from dask_gateway_server.options import Options, Integer, Float, String
+
+      #       def option_handler(options):
+      #           return {
+      #               "worker_cores": options.worker_cores,
+      #               "worker_memory": "%fG" % options.worker_memory,
+      #               "image": options.image,
+      #           }
+
+      #       c.Backend.cluster_options = Options(
+      #           Integer("worker_cores", 2, min=1, max=4, label="Worker Cores"),
+      #           Float("worker_memory", 4, min=1, max=8, label="Worker Memory (GiB)"),
+      #           String("image", default="ghcr.io/dask/dask-gateway:2022.4.0", label="Image"),
+      #           handler=option_handler,
+      #       )
+
+      # Number of instances of the gateway-server to run
+      replicas: 1
+
+      # prefix: "/services/dask-gateway"
+
+      auth:
+        type: jupyterhub
+
+