Blob-CSI: Investigate container configuration in profiles controller for AAW containers

[cboin1996]

Description

Weight the pros and cons of ADLS versus blob, and select which one to implement.

ADLS versus Blob

https://medium.com/awesome-azure/azure-difference-between-azure-blob-storage-and-azure-data-lake-storage-comparison-azure-blob-vs-adls-gen2-81af5ef2a6e1#:~:text=Key%20Differences&text=Blob%3A%20General%20purpose%20object%20store,for%20big%20data%20analytics%20workloads.&text=Blob%3A%20Good%20storage%20retrieval%20performance,ADLS%3A%20Better%20storage%20retrieval%20performance.

ADLS

• Hierarchical permissions
• Files actually exist in the container in the structure they represent
• More performant for big data tasks
• Matches the recommendations from FDI, and in general the cloud teams backing container storage solutions

Blob

• Still have hierarchal folder structure for the user via the blob-csi driver
• Less performant for big data tasks
• Older technology

Recommendation

Use ADLS, as a migration will likely need to happen in the future and this is the time to do it (no one is using the system yet, so its the easiest time to migrate)

Estimated Steps for Creating ADLS Containers

In https://github.com/statcan/aaw-kubeflow-profiles-controller, we want to update the controller to provision ADLS containers.

• Add hierarchical namespace option that defaults to false within:
  • Update https://gitlab.k8s.cloud.statcan.ca/cloudnative/terraform/modules/terraform-azurerm-storage-account/-/tree/main/ to include reference to https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/storage_account#is_hns_enabled as an output
  • Update reference here https://gitlab.k8s.cloud.statcan.ca/cloudnative/aaw/daaas-infrastructure/aaw-dev-cc-00/-/blob/main/minio_gateway.tf#L60 to new version once above is complete.
  • Add the new option for hierarchical name-spacing
  • *** DOUBLE CHECK WHAT HAPPENS WHEN THIS IS IMPLEMENTED TO ENSURE NO DATA LOSS ***
• Provisioning of new containers is done here: https://github.com/StatCan/aaw-kubeflow-profiles-controller/blob/ddcdb6708f31be927c83507023d22cf655a7c5af/cmd/blob-csi.go#L524
• PV mount options will need updated to use ADLS AAW containers:
  • Take this line here: https://github.com/StatCan/aaw-kubeflow-profiles-controller/blob/main/cmd/blob-csi.go#L329, and add here: https://github.com/StatCan/aaw-kubeflow-profiles-controller/blob/main/cmd/blob-csi.go#L309
• Replace endpoint https://github.com/StatCan/aaw-kubeflow-profiles-controller/blob/main/cmd/blob-csi.go#L309 to be dfs instead of blob (see: https://docs.microsoft.com/en-us/azure/storage/common/storage-account-overview#standard-endpoints)
• Delete one container, observe it get re-created and confirm that it is ADLS

[cboin1996]

Migration of the profiles-controller has been documented above to the best of my knowledge.
In the future, this will need to be confirmed with testing. In the meantime, ADLS storage accounts have been created with PR: https://gitlab.k8s.cloud.statcan.ca/cloudnative/aaw/daaas-infrastructure/aaw-dev-cc-00/-/merge_requests/134