Azure SQL Workspace Service (#3970)

* Azure SQL Workspace Service

- New Azure SQL workspace service, based on existing MySQL workspace
- TRE Core changes:
  - Add Azure SQL privatelink DNS zone to core terraform
- Base workspace changes:
  - Add Azure SQL privatelink DNS zone virtual network link to base workspace
- Documentation

* Update CHANGELOG.md

* Fix lint issues

* Add #tflint-ignore directive until a new release is created on microsoft/terraform-azurerm-environment-configuration

* Pin 0.5.0 version to https://github.com/microsoft/terraform-azurerm-environment-configuration.git

* Update CHANGELOG.md

* Update terraform versions

.github/workflows/deploy_tre_reusable.yml

@@ -399,6 +399,8 @@ jobs:
              BUNDLE_DIR: "./templates/workspace_services/databricks"}
           - {BUNDLE_TYPE: "workspace_service",
              BUNDLE_DIR: "./templates/workspace_services/ohdsi"}
+          - {BUNDLE_TYPE: "workspace_service",
+             BUNDLE_DIR: "./templates/workspace_services/azuresql"}
           - {BUNDLE_TYPE: "user_resource",
              BUNDLE_DIR: "./templates/workspace_services/guacamole/user_resources/guacamole-azure-windowsvm"}
           - {BUNDLE_TYPE: "user_resource",
@@ -558,6 +560,8 @@ jobs:
              BUNDLE_DIR: "./templates/workspace_services/databricks"}
           - {BUNDLE_TYPE: "workspace_service",
              BUNDLE_DIR: "./templates/workspace_services/ohdsi"}
+          - {BUNDLE_TYPE: "workspace_service",
+             BUNDLE_DIR: "./templates/workspace_services/azuresql"}
 
     environment: ${{ inputs.environmentName }}
     steps:

CHANGELOG.md

@@ -4,6 +4,7 @@
 **BREAKING CHANGES & MIGRATIONS**:
 
 FEATURES:
+* Azure SQL Workspace Service ([#3969](https://github.com/microsoft/AzureTRE/issues/3969))
 
 ENHANCEMENTS:
 * Add Case Study Docs ([#1366](https://github.com/microsoft/AzureTRE/issues/1366))

core/terraform/dns_zones_non_core.tf

@@ -21,6 +21,17 @@ resource "azurerm_private_dns_zone_virtual_network_link" "mysql" {
   lifecycle { ignore_changes = [tags] }
 }
 
+# since shared services are in the core network, their dns link could exist once and must be defined here.
+resource "azurerm_private_dns_zone_virtual_network_link" "azuresql" {
+  resource_group_name   = azurerm_resource_group.core.name
+  virtual_network_id    = module.network.core_vnet_id
+  private_dns_zone_name = azurerm_private_dns_zone.non_core["privatelink.database.windows.net"].name
+  name                  = azurerm_private_dns_zone.non_core["privatelink.database.windows.net"].name
+  registration_enabled  = false
+  tags                  = local.tre_core_tags
+  lifecycle { ignore_changes = [tags] }
+}
+
 # Once the deployment of the app gateway is complete, we can proceed to include the required DNS zone for Nexus, which is dependent on the FQDN of the app gateway.
 resource "azurerm_private_dns_zone" "nexus" {
   name                = "nexus-${module.appgateway.app_gateway_fqdn}"

core/terraform/locals.tf

@@ -32,6 +32,7 @@ locals {
     "privatelink.notebooks.azure.net",
     "privatelink.postgres.database.azure.com",
     "privatelink.mysql.database.azure.com",
+    "privatelink.database.windows.net",
     "privatelink.azuredatabricks.net"
   ])
 

core/terraform/main.tf

@@ -179,6 +179,6 @@ module "resource_processor_vmss_porter" {
 }
 
 module "terraform_azurerm_environment_configuration" {
-  source          = "git::https://github.com/microsoft/terraform-azurerm-environment-configuration.git?ref=0.2.0"
+  source          = "git::https://github.com/microsoft/terraform-azurerm-environment-configuration.git?ref=0.5.0"
   arm_environment = var.arm_environment
 }

core/version.txt

@@ -1 +1 @@
-__version__ = "0.10.2"
+__version__ = "0.10.3"

docs/tre-templates/workspace-services/azuresql.md

@@ -0,0 +1,54 @@
+# Azure SQL Workspace Service
+
+See: [Azure SQL Database](https://learn.microsoft.com/en-us/azure/azure-sql/database)
+
+## Prerequisites
+
+- The base workspace deployed, or a workspace derived from the base workspace
+
+- The Azure SQL workspace service container image published to your TRE:
+
+  `make workspace_service_bundle BUNDLE=azuresql`
+
+- Guacamole, with a VM containing SQL Server Management Studio or Azure Data Studio in order to connect - the Azure Data Science VM template contains both of these
+
+
+## Authentication
+
+- Server name:  Shown on the details page of the service in the Azure TRE portal under **Azure SQL FQDN**
+- Authentication method: **SQL Server Authentication**
+- Administrator credentials:
+  - Username:  **azuresqladmin**
+  - Password:  *(available in the workspace keyvault)*
+
+## Supported SKUs
+
+The following Azure SQL SKUs have been added to the template:
+
+| Service Tier | Level | DTUs     |
+|--------------|-------|----------|
+| Standard     | S1    | 20 DTUs  |
+| Standard     | S2    | 50 DTUs  |
+| Standard     | S3    | 100 DTUs |
+| Standard     | S4    | 200 DTUs |
+| Standard     | S6    | 400 DTUs |
+
+For costs please [Azure SQL Database pricing](https://azure.microsoft.com/en-us/pricing/details/azure-sql-database/single/) and select **DTU** as the purchase model.
+
+### Adding new SKUs
+
+To add new SKU options within the template, please determine the SKU names using:
+
+```bash
+az sql db list-editions --location <AZURE_REGION> --output table
+```
+
+Then add the SKUs in the following places:
+
+1. In the `templates/workspace_services/azuresql/template_schema.yaml` file under `properties.sql_sku.enum`.
+2. In the `templates/workspace_services/azuresql/terraform/locals.tf` file under `azuresql_sku`.
+3. Above in this document.
+
+Once added, increment the version number in the `templates/workspace_services/azuresql/porter.yaml` file, and republish the template with the following command:
+
+  `make workspace_service_bundle BUNDLE=azuresql`

e2e_tests/resources/strings.py

@@ -19,6 +19,7 @@
 MLFLOW_SERVICE = "tre-service-mlflow"
 MYSQL_SERVICE = "tre-workspace-service-mysql"
 HEALTH_SERVICE = "tre-workspace-service-health"
+AZURESQL_SERVICE = "tre-workspace-service-azuresql"
 
 FIREWALL_SHARED_SERVICE = "tre-shared-service-firewall"
 GITEA_SHARED_SERVICE = "tre-shared-service-gitea"

e2e_tests/test_workspace_services.py

@@ -13,6 +13,7 @@
     strings.MLFLOW_SERVICE,
     strings.MYSQL_SERVICE,
     strings.HEALTH_SERVICE,
+    strings.AZURESQL_SERVICE
 ]
 
 

mkdocs.yml

@@ -99,6 +99,7 @@ nav:
           - Azure Databricks: tre-templates/workspace-services/databricks.md
           - OHDSI: tre-templates/workspace-services/ohdsi.md
           - MySQL: tre-templates/workspace-services/mysql.md
+          - Azure SQL: tre-templates/workspace-services/azuresql.md
       - Shared Services:
           - Gitea (Source Mirror): tre-templates/shared-services/gitea.md
           - Nexus (Package Mirror): tre-templates/shared-services/nexus.md

templates/workspace_services/azuresql/.dockerignore

@@ -0,0 +1,7 @@
+# See https://docs.docker.com/engine/reference/builder/#dockerignore-file
+# Put files here that you don't want copied into your bundle's invocation image
+.gitignore
+Dockerfile.tmpl
+
+# Local .terraform directories
+**/.terraform/*

templates/workspace_services/azuresql/.env.sample

@@ -0,0 +1,5 @@
+ID="__CHANGE_ME__"
+WORKSPACE_ID="__CHANGE_ME__"
+SQL_SKU="__CHANGE_ME__"
+STORAGE_GB="__CHANGE_ME__"
+DB_NAME="__CHANGE_ME__"

templates/workspace_services/azuresql/.gitignore

@@ -0,0 +1 @@
+.cnab/

templates/workspace_services/azuresql/Dockerfile.tmpl

@@ -0,0 +1,15 @@
+# syntax=docker/dockerfile-upstream:1.4.0
+FROM --platform=linux/amd64 debian:bullseye-slim
+
+# PORTER_INIT
+
+RUN rm -f /etc/apt/apt.conf.d/docker-clean; echo 'Binary::apt::APT::Keep-Downloaded-Packages "true";' > /etc/apt/apt.conf.d/keep-cache
+
+# Git is required for terraform_azurerm_environment_configuration
+RUN --mount=type=cache,target=/var/cache/apt --mount=type=cache,target=/var/lib/apt \
+    apt-get update && apt-get install -y git --no-install-recommends
+
+# PORTER_MIXINS
+
+# Use the BUNDLE_DIR build argument to copy files into the bundle
+COPY --link . ${BUNDLE_DIR}/

templates/workspace_services/azuresql/parameters.json

@@ -0,0 +1,68 @@
+{
+  "schemaType": "ParameterSet",
+  "schemaVersion": "1.0.1",
+  "namespace": "",
+  "name": "tre-workspace-service-azuresql",
+  "parameters": [
+    {
+      "name": "tre_id",
+      "source": {
+        "env": "TRE_ID"
+      }
+    },
+    {
+      "name": "id",
+      "source": {
+        "env": "ID"
+      }
+    },
+    {
+      "name": "tfstate_container_name",
+      "source": {
+        "env": "TERRAFORM_STATE_CONTAINER_NAME"
+      }
+    },
+    {
+      "name": "tfstate_resource_group_name",
+      "source": {
+        "env": "MGMT_RESOURCE_GROUP_NAME"
+      }
+    },
+    {
+      "name": "tfstate_storage_account_name",
+      "source": {
+        "env": "MGMT_STORAGE_ACCOUNT_NAME"
+      }
+    },
+    {
+      "name": "sql_sku",
+      "source": {
+        "env": "SQL_SKU"
+      }
+    },
+    {
+      "name": "storage_gb",
+      "source": {
+        "env": "STORAGE_GB"
+      }
+    },
+    {
+      "name": "db_name",
+      "source": {
+        "env": "DB_NAME"
+      }
+    },
+    {
+      "name": "workspace_id",
+      "source": {
+        "env": "WORKSPACE_ID"
+      }
+    },
+    {
+      "name": "arm_environment",
+      "source": {
+        "env": "ARM_ENVIRONMENT"
+      }
+    }
+  ]
+}

templates/workspace_services/azuresql/porter.yaml

@@ -0,0 +1,121 @@
+---
+schemaVersion: 1.0.0
+name: tre-workspace-service-azuresql
+version: 1.0.9
+description: "An Azure SQL workspace service"
+registry: azuretre
+dockerfile: Dockerfile.tmpl
+
+credentials:
+  - name: azure_tenant_id
+    env: ARM_TENANT_ID
+  - name: azure_subscription_id
+    env: ARM_SUBSCRIPTION_ID
+  - name: azure_client_id
+    env: ARM_CLIENT_ID
+  - name: azure_client_secret
+    env: ARM_CLIENT_SECRET
+parameters:
+  - name: workspace_id
+    type: string
+  - name: tre_id
+    type: string
+
+  # the following are added automatically by the resource processor
+  - name: id
+    type: string
+    description: "Resource ID"
+    env: id
+  - name: tfstate_resource_group_name
+    type: string
+    description: "Resource group containing the Terraform state storage account"
+  - name: tfstate_storage_account_name
+    type: string
+    description: "The name of the Terraform state storage account"
+  - name: tfstate_container_name
+    env: tfstate_container_name
+    type: string
+    default: "tfstate"
+    description: "The name of the Terraform state storage container"
+  - name: arm_use_msi
+    env: ARM_USE_MSI
+    type: boolean
+    default: false
+  - name: arm_environment
+    env: ARM_ENVIRONMENT
+    type: string
+    default: "public"
+  - name: sql_sku
+    type: string
+    default: "S2 | 50 DTUs"
+  - name: storage_gb
+    type: integer
+    default: 5
+  - name: db_name
+    type: string
+    default: tredb
+
+mixins:
+  - exec
+  - terraform:
+      clientVersion: 1.9.2
+
+outputs:
+  - name: azuresql_fqdn
+    type: string
+    applyTo:
+      - install
+      - upgrade
+
+install:
+  - terraform:
+      description: "Deploy Azure SQL workspace service"
+      vars:
+        workspace_id: ${ bundle.parameters.workspace_id }
+        tre_id: ${ bundle.parameters.tre_id }
+        tre_resource_id: ${ bundle.parameters.id }
+        sql_sku: ${ bundle.parameters.sql_sku }
+        storage_gb: ${ bundle.parameters.storage_gb }
+        db_name: ${ bundle.parameters.db_name }
+        arm_environment: ${ bundle.parameters.arm_environment }
+      backendConfig:
+        resource_group_name: ${ bundle.parameters.tfstate_resource_group_name }
+        storage_account_name: ${ bundle.parameters.tfstate_storage_account_name }
+        container_name: ${ bundle.parameters.tfstate_container_name }
+        key: tre-workspace-service-azuresql-${ bundle.parameters.id }
+      outputs:
+        - name: azuresql_fqdn
+upgrade:
+  - terraform:
+      description: "Upgrade Azure SQL workspace service"
+      vars:
+        workspace_id: ${ bundle.parameters.workspace_id }
+        tre_id: ${ bundle.parameters.tre_id }
+        tre_resource_id: ${ bundle.parameters.id }
+        sql_sku: ${ bundle.parameters.sql_sku }
+        storage_gb: ${ bundle.parameters.storage_gb }
+        db_name: ${ bundle.parameters.db_name }
+        arm_environment: ${ bundle.parameters.arm_environment }
+      backendConfig:
+        resource_group_name: ${ bundle.parameters.tfstate_resource_group_name }
+        storage_account_name: ${ bundle.parameters.tfstate_storage_account_name }
+        container_name: ${ bundle.parameters.tfstate_container_name }
+        key: tre-workspace-service-azuresql-${ bundle.parameters.id }
+      outputs:
+        - name: azuresql_fqdn
+uninstall:
+  - terraform:
+      description: "Tear down Azure SQL workspace service"
+      vars:
+        workspace_id: ${ bundle.parameters.workspace_id }
+        tre_id: ${ bundle.parameters.tre_id }
+        tre_resource_id: ${ bundle.parameters.id }
+        sql_sku: ${ bundle.parameters.sql_sku }
+        storage_gb: ${ bundle.parameters.storage_gb }
+        db_name: ${ bundle.parameters.db_name }
+        arm_environment: ${ bundle.parameters.arm_environment }
+      backendConfig:
+        resource_group_name: ${ bundle.parameters.tfstate_resource_group_name }
+        storage_account_name: ${ bundle.parameters.tfstate_storage_account_name }
+        container_name: ${ bundle.parameters.tfstate_container_name }
+        key: tre-workspace-service-azuresql-${ bundle.parameters.id }