[Epic] Multi-tenant source control

[brendangadd]

As a user, I want to have easy access to Git source control management so that I can practice GitOps in my team even when working with sensitive data.

Broad strokes:

1. Multi-tenant Gitea (instance per namespace)
2. Look into PostgreSQL backend – probably a shared managed DB instance with a logical database per Gitea instance
3. Kubernetes controller (profile) for automated deployment

• Kubernetes resources
  • Network policy restricting to Gitea app?
  • Connectivity limited to ProB node pool
  • X509 cert auth with PostgreSQL via Spiffe? No, per @blairdrummond's comment below
• PostgreSQL database creation etc.

4. Gitea upgrades over time – via controller?
5. Copy/paste architecture to unclassified context

Roadmap to Production

Local Development (5 day)

• feat(gitea): setup local environment #886
• Modify the local gitea deployment to use a local postgres instance #897

Profile Controller SQlite Deployment (5 day)

• feat(profile-controller): Configure a kubeflow profile controller  #887
• Flush out profile controller to fully configure gitea #898

Create Managed DB (5 day)

• feat(postgres-terraform): Create Managed DB #888

Automate Interaction with Database (10 day)

• feat(profile-controller): Automate Interaction with Database #889

UI Enhancements

• Add Gitea UI to Kubeflow UI #1031
• Add virtual service generation to kubeflow profile controller #1038
• Configure kubeflow central dashboard configmap with Gitea link #1039

Finalize the Deployment (16 day)

Protected-b

• feat(gitea): protected-b setup part 1 #890
• Gitea: Deploy azure managed postgres in terraform repo #1105
• Gitea: Configure networking for protected-b #1190
• Gitea: Test and confirm that prot-b deployment works in dev #1191
• Gitea: Create a new secret for the prot-b azure managed db #1192

Production

• Gitea: Deploy the profiles-argocd-system to prod and fix the RBAC (ClusterRole+ClusterRoleBinding) #1081
• Gitea: Update argocd-manifests repo with latest controller deployment #1367
• Gitea: Gatekeeper + Netpol creation #1368
• Gitea: parameterize the gitea helm chart in statcan charts repo for dev or prod #1278
• Gitea: make the git branch for the argo apps in the aaw-kubeflow-profiles-controller a cli argument #1279
• Gitea: provision a db for prod in terraform (two databases, unclassified and protected-b) #1280
• Gitea: Add psql k8s secrets in terraform #1370
• Gitea: add to central dashboard #1372
• Gitea: Confirm gitea unclassified and prot-b works in prod #1373
• Gitea: Re-enabled auto sync for gitea apps once RMI is accepted #1377

Docs

• Revise Source Control recommendations aaw-security-proposal#6
• Profiles Controller: documentation for debugging the controller #1118

Advantages

• Increased security via namespace isolation
• Increased collaboration with external users
• Future compatibility with MLOps (seldon model deployments, convention based)
  • With a shared gitlab instance, you would need to add a layer that references specific locations in gitlab with access tokens or equivalent to try and set this up, but per-namespace gitea handles this more elegantly and simply.
• If your namespace has any external user, you are forbidden from using gitlab

[blairdrummond]

Plain postgres auth will probably be ok (instead of spiffe)! We might want the controller to create postgres secrets and possibly rotate those secrets on occasion (fixed interval).

[sylus]

Out of curiousity can I get some further information on this? @blairdrummond

a) This won't take away resources from KF 1.3 and notebook culling which really needs to happen to Prod soon?
b) What was the point of gitlab protected b and just using RBAC there for isolation is it not strict enough?
c) Will gitlab be removed in favor of this design?
d) Not every namespace will be given this correct as gitea requires 2 cores and 1.5GB of memory so the cluster cost would be quite high for every namespace

[blairdrummond]

a) KF 1.3 will not be affected, @cboin1996 is independent of that
b) Yeah this is in-part to do unclassified & protected-b
c) Yes, Gitlab will be removed
d) @sylus I think we'll dial the CPU way back (maybe half a cpu?). We might only apply Gitea to shared-namespaces, we have labels in-place that allow that.

[blairdrummond]

@brendangadd @chritter I am realizing, this shared postgres + namespaced deployment setup, would enable MLFlow very very quickly. You could basically copy-paste the deployment for MLFlow.

[chuckbelisle]

This Epic is being blocked by Cyber Sec priorities and new platform requirements being introduced. All is in place to have this deployed.