Automate cluster setup for Azure

[yuvipanda]

Description

We should write terraform code to programmatically deploy hubs on Azure infrastructure, similar to our AWS and GCP terraform deployments. We already have two hubs running on Azure (#288 and #413) that we can use as reference.

Benefit

Azure probably the least-common cloud that our communities want to run infrastructure on. However, it's still one of "the big three" and will certainly be requested again in the future. Moreover, since we're already running two hubs on Azure, we can use a terraform deployment to make it easier to update and maintain those hubs in the future.

Implementation

Here is the terraform code for the UToronto deployment. We should clean this up, generalize it, and move it to this repository.

We should try to use AzureFile if possible so that we don't have to hassle with setting up our own filesystems on Azure.

Steps to complete this goal

• Add Terraform setup for Azure cluster #513
  • Move the terraform directory from our utoronto deployment repo to this repo, taking care to preserve history
  • Modify our current terraform code to work alongside the new Azure terraform stuff, making sure we have the right level of abstraction
  • Update Azure terraform code to setup home directory storage on Azure with AzureFile
• Document our Azure cluster deployment setup #718

[yuvipanda]

/cc @sgibson91 who has a lot of Azure experience!

[yuvipanda]

@sgibson91 do you have any experience with Azure Files? I am hoping we can use that the way we use EFS on AWS

[sgibson91]

A small amount, nothing that ever managed to get into production though.

• I followed this guide to create the Azure Files volume in AKS (There's also this guide for dynamic creation)
• Then, the z2jh guide for shared volumes

This issue kinda devolved into security of the data in the volume and authentication methods, i.e., using the Storage Access Key probably isn't the best move as it has full read/write access. But I also don't think any of the proposed solutions actually work for a Kubernetes deployment (they came from the context of a Data Safe Haven which is a bunch of VMs not controlled by k8s). So I'm just flagging that in case you want to factor it into your considerations.

[choldgraf]

Could we keep this one open until our azure cluster setup is documented? We have a placeholder page for this here: https://pilot-hubs.2i2c.org/en/latest/howto/operate/new-cluster/azure.html

Doesn't need to be complex since we're using terraform. I wonder if we could just reuse much of the instructions here: https://pilot-hubs.2i2c.org/en/latest/howto/operate/new-cluster/google-cloud.html ?

[yuvipanda]

Yep, that PR should've closed #373 not this one! Sorry for not verifying that.

[choldgraf]

No problem 🙂

[choldgraf]

Next steps on this one: @yuvipanda and @sgibson91 will collaborate together to write out some rough documentation for this, and we can refine this documentation in the future when we need to deploy a hub on Azure.

[choldgraf]

I'm removing this from our backlog and replacing it with something a bit more focused on the documentation, since that's the only remaining step: #718

We can close this one when that is done as well

[sgibson91]

#718 is merged so I believe this can be closed now