Deprecate MinIO in dev

[chuckbelisle]

Related to #1753
See relevant information by Collin Brown here #1001 (comment)

Component Cleanup

IMPORTANT

• Investigate methods for backing up or enabling 'soft deleting' prior to doing anything
• DO NOT DELETE THESE https://gitlab.k8s.cloud.statcan.ca/cloudnative/aaw/daaas-infrastructure/aaw-dev-cc-00/-/blob/main/minio_gateway.tf#L5-25
• DO NOT DELETE THIS https://gitlab.k8s.cloud.statcan.ca/cloudnative/aaw/daaas-infrastructure/aaw-dev-cc-00/-/blob/main/minio_gateway.tf#L56-72
• Make Azure object replication policy doing a back-up of the entire prod / dev storage accounts while we do this migration - then delete once migration is successfully complete. - > Need access to storage accounts

Deprecating old profiles controller

• when we're deprecating MinIO, we should also do the migration of the old profiles controller - there will only be a few components left, so we can use this opportunity to migrate the various functionality over to aaw-kubeflow-profiles-controller. I'm assuming this is aaw-kubeflow-controller which has already been scaled down for a while.

Roadmap to Deprecation

• Remove minio-credential-injector argo-cd-manifest

• Remove the boathouse MinIO ingresses

• Delete deployment for goofys injector in argocd manfiests

• Delete the entire terraform config file for boathouse (dev and prod)

• Delete goofys-injector terraform https://gitlab.k8s.cloud.statcan.ca/cloudnative/aaw/daaas-infrastructure/aaw-dev-cc-00/-/blob/main/goofys_injector.tf

• Delete everything in this range https://gitlab.k8s.cloud.statcan.ca/cloudnative/aaw/daaas-infrastructure/aaw-dev-cc-00/-/blob/main/istio_clusterrbacconfig.tf#L28-36 - do not delete fdi-gateway exclusion yet b/c OPA gateway is in this namespace and needs to communicate with other namespaces. -> This has been deleted in dev/prod by me a while ago. - souheil

• Delete the read_instances and oidc_instance (https://gitlab.k8s.cloud.statcan.ca/cloudnative/aaw/daaas-infrastructure/aaw-dev-cc-00/-/blob/main/minio_gateway.tf#L26-54)

• Delete these https://gitlab.k8s.cloud.statcan.ca/cloudnative/aaw/daaas-infrastructure/aaw-dev-cc-00/-/blob/main/minio_gateway.tf#L74-200

• Delete helm chart for boathouse https://gitlab.k8s.cloud.statcan.ca/cloudnative/aaw/daaas-infrastructure/aaw-dev-cc-00/-/tree/main/charts/boathouse

• Add rbac components in kubeflow-controller to the rbac.go file in the kubeflow-profiles-controller repo. -> this may not be needed, have to double check.

• Remove the kubeflow controller deployment from argocd manifests https://github.com/StatCan/aaw-argocd-manifests/tree/aaw-dev-cc-00/daaas-system/profile-controllers/kubeflow-controller

• Flatten directory structure of profiles controller here https://github.com/StatCan/aaw-argocd-manifests/tree/aaw-dev-cc-00/daaas-system/profile-controllers (no need for two deployments anymore)

• Delete minio.go from https://github.com/StatCan/aaw-kubeflow-profiles-controller/blob/main/cmd/minio.go

• Delete all minio-related network policies https://github.com/StatCan/aaw-kubeflow-profiles-controller/blob/main/cmd/network.go

• Delete deployment-buckets https://github.com/StatCan/charts/blob/master/stable/profiles-controller/templates/deployment-buckets.yaml from charts repo

• Remove any network.go references to vault in aaw-kubeflow-profiles-controller

• Upgrade chart https://github.com/StatCan/charts/blob/master/stable/profiles-controller/Chart.yaml (uptick chart version)

• Update values.yaml file (delete any references to buckets or vault agent)

• Delete vault minio config map https://github.com/StatCan/charts/blob/master/stable/profiles-controller/templates/configmap-vaultagent.yaml

• https://github.com/StatCan/aaw-kubeflow-profiles-controller/blob/main/cmd/notebook.go#L109-L131 delete mounting minio to fs

• https://github.com/StatCan/aaw-argocd-manifests/blob/aaw-dev-cc-00/daaas-system/profile-controllers/profiles-controller/application.jsonnet This no longer needs to be parameterized as jsonnet (no need for vault/minio) - can just make this a plain manifest.yaml file. Delete anything unused and update the helm chart

• Delete entire folder storage-system https://github.com/StatCan/aaw-argocd-manifests/tree/aaw-dev-cc-00/storage-system -

• Delete goofys injector network policies https://github.com/StatCan/aaw-network-policies/blob/aaw-dev-cc-00/goofys-injector-system.yaml (note: none of the network policies are deployed by themselves; need to edit the kustomize file in https://github.com/StatCan/aaw-network-policies/blob/aaw-dev-cc-00/environments/aaw-dev-cc-00/kustomization.yaml for dev and prod)

• Delete https://github.com/StatCan/aaw-network-policies/blob/aaw-dev-cc-00/boathouse-system.yaml

• Delete network policies for minio-credential-injector and goofys-injector https://github.com/StatCan/aaw-network-policies/blob/aaw-dev-cc-00/daaas-system.yaml#L139-L169

• Delete https://github.com/StatCan/aaw-network-policies/blob/aaw-dev-cc-00/minio-legacy-system.yaml

• Delete https://github.com/StatCan/aaw-network-policies/blob/aaw-dev-cc-00/minio-premium-legacy-system.yaml

• Delete https://github.com/StatCan/aaw-network-policies/blob/aaw-dev-cc-00/vault-system.yaml#L18-L54

• Delete https://github.com/StatCan/terraform-aaw-vault/blob/master/secret_minio_standard_tenant_1.tf

• Delete https://github.com/StatCan/terraform-aaw-vault/blob/master/secret_minio_premium_tenant_1.tf

• Delete https://github.com/StatCan/terraform-aaw-vault/blob/master/policy_profile_configurator.tf

• Delete https://github.com/StatCan/terraform-aaw-vault/blob/master/policy_minio.tf

• Delete https://github.com/StatCan/terraform-aaw-vault/blob/master/policy_goofys_injector.tf

• Delete https://github.com/StatCan/terraform-aaw-vault/blob/master/policy_boathouse.tf

• https://gitlab.k8s.cloud.statcan.ca/cloudnative/aaw/daaas-infrastructure/terraform-aaw-dev-cc-00-vault Delete policy_boathouse.tf and policy_profile_configurator.tf, do additional pass through and delete any old references

• Delete https://gitlab.k8s.cloud.statcan.ca/cloudnative/aaw/daaas-infrastructure/aaw-dev-cc-00/-/tree/main/modules/minio-gateway minio gateway terraform module

• Delete minio_gateway unclassified and protected-b in here https://gitlab.k8s.cloud.statcan.ca/cloudnative/aaw/daaas-infrastructure/aaw-dev-cc-00/-/blob/main/fdi_gateway.tf

• Delete https://gitlab.k8s.cloud.statcan.ca/cloudnative/aaw/daaas-infrastructure/aaw-dev-cc-00/-/blob/main/goofys_injector.tf

• Any variables https://gitlab.k8s.cloud.statcan.ca/cloudnative/aaw/daaas-infrastructure/aaw-dev-cc-00/-/blob/main/variables.tf that relate to minio

• Unregister MinIO from OIDC app regsitry

• Update DAaaS docs end user documentation for blob csi

  • Remove old references to MinIO that are no longer valid
  • Add user-facing explanation of new Blob-CSI driver + S3Proxy approach
  • Add Blob CSI documentation to the daaas/docs/dev folder

[vexingly]

FYI the fdi-gateway-unclassified-system/protected-b-systems have been failing for a while as the storage accounts were already deleted by the FDI team (See jira issue CODAS-2172). Maybe it is a good idea to decom these systems first as they seem to be causing lots of failure for our pods trying to configure them. :)

[Souheil-Yazji]

https://gitlab.k8s.cloud.statcan.ca/cloudnative/aaw/daaas-infrastructure/aaw-dev-cc-00/-/merge_requests/199

[Souheil-Yazji]

Changes required for dev-vault https://gitlab.k8s.cloud.statcan.ca/cloudnative/aaw/daaas-infrastructure/terraform-aaw-dev-cc-00-vault/-/commit/b459ad1cdb4ad817c58f934d18e8522a081ae614

[Souheil-Yazji]

After deleting most minio resources:

Notebooks spin up fine (even with the old minio mount label on them) and we can see that the minio mount endpoints are gone:

the mc command will have to be removed from the containers.

[Souheil-Yazji]

Closing Remarks

• Storage accounts to be deleted once Minio is fully deprecated from AAW.
• I don't have access to the oidc app reg so I'm assuming that would be done by CNS. Will create an issue to remove both prod and dev when deprecating prod.
• We can update kubeflow containers to remove MC, but this is low priority and trivial task to be scheduled later.