[Epic Blob-CSI] Fix ~~Blob-CSI access via Azure RBAC~~ Azure Subnet Issue (Need to add AKS node pool subnet to allow list for storage accounts)

[blairdrummond]

Blair:
I'm looking at this

https://github.com/kubernetes-sigs/blob-csi-driver/blob/master/docs/install-driver-on-aks.md

But wondering who has access/where I change that

I had it working with secrets in kind, but on azure it's ignoring my secrets I think

Salwa:
There is the user managed identity of the cluster located in the cluster resource group. It just needs to be given the role Contributor in the cluster resource group.
I did it manually through azure when I was testing, but we can probably do it through terraform.

[blairdrummond]

This is a sample PV + PVC for testing. Once you bind a pod to this PVC (e.g. by creating a notebook that mounts it) the errors show up on the blob-csi pods.

apiVersion: v1
kind: PersistentVolume
metadata:
  labels:
    data.statcan.gc.ca/classification: unclassified
    profile: blair-drummond
  name: blair-drummond-standard
spec:
  accessModes:
  - ReadWriteMany
  capacity:
    storage: 100G
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: standard
    namespace: blair-drummond
  csi:
    driver: blob.csi.azure.com
    nodeStageSecretRef:
      name: aawdevcc00samgstandard
      namespace: azure-blob-csi-system
    volumeAttributes:
      containerName: blair-drummond
    volumeHandle: blair-drummond-standard
  mountOptions:
  - -o allow_other
  persistentVolumeReclaimPolicy: Retain
  volumeMode: Filesystem
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  labels:
    blob.aaw.statcan.gc.ca/automount: "true"
    data.statcan.gc.ca/classification: unclassified
  name: standard
  namespace: blair-drummond
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 100G
  storageClassName: ""
  volumeMode: Filesystem
  volumeName: blair-drummond-standard

[blairdrummond]

As per conversation with @zachomedia , ideally this will be accessible from all user workloads, both Unclassified and Protected-B.

We have both Protected-B (protected-b) and Unclassified (standard, premium) Storage accounts.

Unclassified Storage accounts will be accessed by Protected-B and Unclassified nodes. (Read-only mirror for protected-b)

Protected-B storage accounts will only be accessed by Protected-B nodes.

Let me know if I can provide anything else @zachomedia !

[zachomedia]

Per the conversation on Slack, I have temporarily granted Contributor to the RG. Longer term, we will need to find the reduced set of permissions that it needs since we shouldn't be granting Contributor.

[blairdrummond]

Resolved! The issue had nothing to do with RBAC, it was that permission had not been granted to the unclassified subnet in the storage accounts. Resolved by modifying the storage account terraform.

[Collinbrown95]

Context: this comment is for documentation purposes - it will be moved to a separate location in the future.

1. The addition of AKS node pool subnets happens here https://gitlab.k8s.cloud.statcan.ca/cloudnative/aaw/daaas-infrastructure/aaw-dev-cc-00/-/blob/main/minio_gateway.tf#L7-12 e.g.

2. https://gitlab.k8s.cloud.statcan.ca/cloudnative/aaw/daaas-infrastructure/aaw-dev-cc-00/-/blob/main/minio_gateway.tf#L68 This is where the AKS node pool subnet is declared in the set of subnets the storage account can talk to.

Important Concept: there is an asymmetry with subnet access for the storage accounts:

1. Pro-B node pool has read access to the unclassified storage account, but not write access (e.g. users can't exfiltrate information).
2. Unclassified node pool has no access to the protected-b storage account (b/c there should be neither read nor write access to the protected-b storage account).