Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat (deployment): [rgs] follow the region resource recommendations #99

Merged
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 3 additions & 22 deletions cluster-stamp.bicep
Original file line number Diff line number Diff line change
Expand Up @@ -16,27 +16,6 @@ param k8sControlPlaneAuthorizationTenantId string
@secure()
param appGatewayListenerCertificate string

@allowed([
'australiaeast'
'canadacentral'
'centralus'
'eastus'
'eastus2'
'westus2'
'francecentral'
'germanywestcentral'
'northeurope'
'southafricanorth'
'southcentralus'
'uksouth'
'westeurope'
'japaneast'
'southeastasia'
])
@description('AKS Service, Node Pools, and supporting services (KeyVault, App Gateway, etc) region. This needs to be the same region as the vnet provided in these parameters.')
@minLength(4)
param location string = 'eastus2'

@description('The Azure resource ID of a VM image that will be used for the jump box.')
@minLength(70)
param jumpBoxImageResourceId string
Expand All @@ -57,6 +36,9 @@ param gitOpsBootstrappingRepoBranch string = 'main'

var kubernetesVersion = '1.30'

@description('AKS Service, Node Pool, and supporting services (KeyVault, App Gateway, etc) region. This needs to be the same region as the vnet provided in these parameters. This defaults to the resource group\'s location for higher reliability.')
var location = resourceGroup().location

var subRgUniqueString = uniqueString('aks', subscription().subscriptionId, resourceGroup().id)
var clusterName = 'aks-${subRgUniqueString}'
var jumpBoxDefaultAdminUserName = uniqueString(clusterName, resourceGroup().id)
Expand Down Expand Up @@ -1008,7 +990,6 @@ module ensureClusterIdentityHasRbacToSelfManagedResources 'modules/ensureCluster
miClusterControlPlanePrincipalId: miClusterControlPlane.properties.principalId
clusterControlPlaneIdentityName: miClusterControlPlane.name
vnetSpokeName: vnetSpoke.name
location: location
}
}

Expand Down
12 changes: 6 additions & 6 deletions docs/deploy/04-subscription.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,12 +12,12 @@ This reference implementation is split across several resource groups in a singl

The following four resource groups will be created in these steps.

| Name | Purpose |
|---------------------------------|-------------------------------------------|
| rg-enterprise-networking-hubs | Contains all of your organization's regional hubs. A regional hub resources in this implementation include the hub Virtual Network, egress firewall, Azure Bastion, and Log Analytics for network logging. They may also contain your VPN Gateways, which are not addressed in this implementation. |
| rg-enterprise-networking-spokes | Contains all of your organization's regional spokes and related networking resources. All spokes will peer with their regional hub and subnets will egress through the regional firewall in the hub. |
| rg-bu0001a0005 | Contains the regulated cluster resources. |
| networkWatcherRG | Contains regional Network Watchers. *(Most subscriptions already have this.)* |
| Name | Purpose |
|-------------------------------------------|-------------------------------------------|
| rg-enterprise-networking-hubs-centralus | Contains all of your organization's regional hubs. A regional hub resources in this implementation include the hub Virtual Network, egress firewall, Azure Bastion, and Log Analytics for network logging. They may also contain your VPN Gateways, which are not addressed in this implementation. |
| rg-enterprise-networking-spokes-centralus | Contains all of your organization's regional spokes and related networking resources. All spokes will peer with their regional hub and subnets will egress through the regional firewall in the hub. |
| rg-bu0001a0005-centralus | Contains the regulated cluster resources. |
| networkWatcherRG | Contains regional Network Watchers. *(Most subscriptions already have this.)* |

Both Azure Kubernetes Service and Azure Image Builder Service use a concept of a dynamically-created *infrastructure* resource group. So in addition to the four resource groups mentioned above, as you follow these instructions, you'll end up with six resource groups; two of which are automatically created and their lifecycle tied to their owning service. You will not see these two infrastructure resource groups get created until later in the walkthrough when their owning service is created.

Expand Down
2 changes: 1 addition & 1 deletion docs/deploy/05-networking-hub.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ Since this walkthrough is expected to be deployed isolated from existing infrast

```bash
# [This takes about eight minutes to run.]
az deployment group create -g rg-enterprise-networking-hubs -f networking/hub-region.v0.bicep -p location=eastus2
az deployment group create -g rg-enterprise-networking-hubs-centralus -f networking/hub-region.v0.bicep
```

The hub deployment will output the following:
Expand Down
16 changes: 8 additions & 8 deletions docs/deploy/06-aks-jumpboximage.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,10 +37,10 @@ You are going to be using Azure Image Builder to generate a Kubernetes-specific
1. Create the AKS jump box image builder network spoke.

```bash
RESOURCEID_VNET_HUB=$(az deployment group show -g rg-enterprise-networking-hubs -n hub-region.v0 --query properties.outputs.hubVnetId.value -o tsv)
RESOURCEID_VNET_HUB=$(az deployment group show -g rg-enterprise-networking-hubs-centralus -n hub-region.v0 --query properties.outputs.hubVnetId.value -o tsv)

# [This takes about one minute to run.]
az deployment group create -g rg-enterprise-networking-spokes -f networking/spoke-BU0001A0005-00.bicep -p location=eastus2 hubVnetResourceId="${RESOURCEID_VNET_HUB}"
az deployment group create -g rg-enterprise-networking-spokes-centralus -f networking/spoke-BU0001A0005-00.bicep -p hubVnetResourceId="${RESOURCEID_VNET_HUB}"
```

1. Update the regional hub deployment to account for the requirements of the spoke.
Expand All @@ -50,10 +50,10 @@ You are going to be using Azure Image Builder to generate a Kubernetes-specific
> :eyes: If you're curious to see what changed in the regional hub, [view the diff](https://diffviewer.azureedge.net/?l=https://raw.githubusercontent.com/mspnp/aks-baseline-regulated/main/networking/hub-region.v0.bicep&r=https://raw.githubusercontent.com/mspnp/aks-baseline-regulated/main/networking/hub-region.v1.bicep).

```bash
RESOURCEID_SUBNET_AIB=$(az deployment group show -g rg-enterprise-networking-spokes -n spoke-BU0001A0005-00 --query properties.outputs.imageBuilderSubnetResourceId.value -o tsv)
RESOURCEID_SUBNET_AIB=$(az deployment group show -g rg-enterprise-networking-spokes-centralus -n spoke-BU0001A0005-00 --query properties.outputs.imageBuilderSubnetResourceId.value -o tsv)

# [This takes about five minutes to run.]
az deployment group create -g rg-enterprise-networking-hubs -f networking/hub-region.v1.bicep -p location=eastus2 aksImageBuilderSubnetResourceId="${RESOURCEID_SUBNET_AIB}"
az deployment group create -g rg-enterprise-networking-hubs-centralus -f networking/hub-region.v1.bicep -p aksImageBuilderSubnetResourceId="${RESOURCEID_SUBNET_AIB}"
```

### Build and deploy the jump box image
Expand Down Expand Up @@ -88,21 +88,21 @@ Now that we have our image building network created, egressing through our hub,
ROLEID_IMGDEPLOY=$(az deployment sub show -n DeployAibRbacRoles --query 'properties.outputs.roleResourceIds.value.customImageBuilderImageCreationRole.guid' -o tsv)

# [This takes about one minute to run.]
az deployment group create -g rg-bu0001a0005 -f jumpbox/azuredeploy.bicep -p buildInSubnetResourceId=${RESOURCEID_SUBNET_AIB} location=eastus2 imageBuilderNetworkingRoleGuid="${ROLEID_NETWORKING}" imageBuilderImageCreationRoleGuid="${ROLEID_IMGDEPLOY}" -n CreateJumpBoxImageTemplate
az deployment group create -g rg-bu0001a0005-centralus -f jumpbox/azuredeploy.bicep -p buildInSubnetResourceId=${RESOURCEID_SUBNET_AIB} imageBuilderNetworkingRoleGuid="${ROLEID_NETWORKING}" imageBuilderImageCreationRoleGuid="${ROLEID_IMGDEPLOY}" -n CreateJumpBoxImageTemplate
```

1. Build the general-purpose AKS jump box image.

Now you'll build the actual VM golden image you will use for your jump box. This uses the image template created in the prior step and is executed by Azure Image Builder under the authority of the managed identity (and its role assignments) also created in the prior step.

```bash
IMAGE_TEMPLATE_NAME=$(az deployment group show -g rg-bu0001a0005 -n CreateJumpBoxImageTemplate --query 'properties.outputs.imageTemplateName.value' -o tsv)
IMAGE_TEMPLATE_NAME=$(az deployment group show -g rg-bu0001a0005-centralus -n CreateJumpBoxImageTemplate --query 'properties.outputs.imageTemplateName.value' -o tsv)

# [This takes about >> 30 minutes << to run.]
az image builder run -n $IMAGE_TEMPLATE_NAME -g rg-bu0001a0005
az image builder run -n $IMAGE_TEMPLATE_NAME -g rg-bu0001a0005-centralus
```

> A successful run of the command above is typically shown with no output or a success message. An error state will be typically be presented if there was an error. To see whether your image was built successfully, you can go to the **rg-bu0001a0005** resource group in the portal and look for a created VM Image resource. It will have the same name as the Image Template resource created in Step 2.
> A successful run of the command above is typically shown with no output or a success message. An error state will be typically be presented if there was an error. To see whether your image was built successfully, you can go to the **rg-bu0001a0005-centralus** resource group in the portal and look for a created VM Image resource. It will have the same name as the Image Template resource created in Step 2.

:coffee: This does take a significant amount of time to run. While the image building is happening, feel free to read ahead, but you should not proceed until this is complete. If you need to perform this reference implementation walk through multiple times, we suggest you create this image in a place that can survive the deleting and re-creating of this reference implementation to save yourself this time in a future execution of this guide.

Expand Down
14 changes: 7 additions & 7 deletions docs/deploy/08-cluster-networking.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ The regional spoke network in which your cluster is laid into acts as the first

## Expected results

Your `rg-enterprise-networking-spokes` will be populated with the dedicated regional spoke network in which your cluster (and its direct adjacent resources will be connected to). This spoke will have limited Internet exposure and will support Network Security Groups (NSGs) at various levels to further limit network traffic as necessary.
Your `rg-enterprise-networking-spokes-centralus` will be populated with the dedicated regional spoke network in which your cluster (and its direct adjacent resources will be connected to). This spoke will have limited Internet exposure and will support Network Security Groups (NSGs) at various levels to further limit network traffic as necessary.

- The network spoke will be called `vnet-spoke-bu0001a0005-01` and have a range of `10.240.0.0/16`.
- The spoke is broken into multiple subnets, each with a clearly defined purpose, appropriate IP range, and maximally restrictive NSG.
Expand All @@ -20,10 +20,10 @@ Your `rg-enterprise-networking-spokes` will be populated with the dedicated regi
1. Deploy the cluster spoke.

```bash
RESOURCEID_VNET_HUB=$(az deployment group show -g rg-enterprise-networking-hubs -n hub-region.v0 --query properties.outputs.hubVnetId.value -o tsv)
RESOURCEID_VNET_HUB=$(az deployment group show -g rg-enterprise-networking-hubs-centralus -n hub-region.v0 --query properties.outputs.hubVnetId.value -o tsv)

# [This takes about five minutes to run.]
az deployment group create -g rg-enterprise-networking-spokes -f networking/spoke-BU0001A0005-01.bicep -p location=eastus2 hubVnetResourceId="${RESOURCEID_VNET_HUB}"
az deployment group create -g rg-enterprise-networking-spokes-centralus -f networking/spoke-BU0001A0005-01.bicep -p hubVnetResourceId="${RESOURCEID_VNET_HUB}"
```

1. Update the regional hub deployment to account for the runtime requirements of the virtual network.
Expand All @@ -33,12 +33,12 @@ Your `rg-enterprise-networking-spokes` will be populated with the dedicated regi
> :eyes: If you're curious to see what changed in the regional hub, [view the diff](https://diffviewer.azureedge.net/?l=https://raw.githubusercontent.com/mspnp/aks-baseline-regulated/main/networking/hub-region.v1.bicep&r=https://raw.githubusercontent.com/mspnp/aks-baseline-regulated/main/networking/hub-region.v2.bicep).

```bash
RESOURCEID_SUBNET_AIB=$(az deployment group show -g rg-enterprise-networking-spokes -n spoke-BU0001A0005-00 --query properties.outputs.imageBuilderSubnetResourceId.value -o tsv)
RESOURCEID_SUBNET_NODEPOOLS="['$(az deployment group show -g rg-enterprise-networking-spokes -n spoke-BU0001A0005-01 --query "properties.outputs.nodepoolSubnetResourceIds.value | join ('\',\'',@)" -o tsv)']"
RESOURCEID_SUBNET_JUMPBOX=$(az deployment group show -g rg-enterprise-networking-spokes -n spoke-BU0001A0005-01 --query properties.outputs.jumpboxSubnetResourceId.value -o tsv)
RESOURCEID_SUBNET_AIB=$(az deployment group show -g rg-enterprise-networking-spokes-centralus -n spoke-BU0001A0005-00 --query properties.outputs.imageBuilderSubnetResourceId.value -o tsv)
RESOURCEID_SUBNET_NODEPOOLS="['$(az deployment group show -g rg-enterprise-networking-spokes-centralus -n spoke-BU0001A0005-01 --query "properties.outputs.nodepoolSubnetResourceIds.value | join ('\',\'',@)" -o tsv)']"
RESOURCEID_SUBNET_JUMPBOX=$(az deployment group show -g rg-enterprise-networking-spokes-centralus -n spoke-BU0001A0005-01 --query properties.outputs.jumpboxSubnetResourceId.value -o tsv)

# [This takes about seven minutes to run.]
az deployment group create -g rg-enterprise-networking-hubs -f networking/hub-region.v2.bicep -p location=eastus2 aksImageBuilderSubnetResourceId="${RESOURCEID_SUBNET_AIB}" nodepoolSubnetResourceIds="${RESOURCEID_SUBNET_NODEPOOLS}" aksJumpboxSubnetResourceId="${RESOURCEID_SUBNET_JUMPBOX}"
az deployment group create -g rg-enterprise-networking-hubs-centralus -f networking/hub-region.v2.bicep -p aksImageBuilderSubnetResourceId="${RESOURCEID_SUBNET_AIB}" nodepoolSubnetResourceIds="${RESOURCEID_SUBNET_NODEPOOLS}" aksJumpboxSubnetResourceId="${RESOURCEID_SUBNET_JUMPBOX}"
```

### Next step
Expand Down
12 changes: 6 additions & 6 deletions docs/deploy/09-pre-cluster-stamp.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,15 +31,15 @@ An Azure user managed identity is going to be deployed. This identity is the ing
> :book: The app team will be deploying to a spoke Virtual Network, that was already provisioned by the network team.

```bash
export RESOURCEID_VNET_CLUSTERSPOKE=$(az deployment group show -g rg-enterprise-networking-spokes -n spoke-BU0001A0005-01 --query properties.outputs.clusterVnetResourceId.value -o tsv)
export RESOURCEID_VNET_CLUSTERSPOKE=$(az deployment group show -g rg-enterprise-networking-spokes-centralus -n spoke-BU0001A0005-01 --query properties.outputs.clusterVnetResourceId.value -o tsv)
echo RESOURCEID_VNET_CLUSTERSPOKE: $RESOURCEID_VNET_CLUSTERSPOKE
```

1. Deploy the bootstrapping resources template.

```bash
# [This takes about eight minutes.]
az deployment group create -g rg-bu0001a0005 -f pre-cluster-stamp.bicep -p targetVnetResourceId=${RESOURCEID_VNET_CLUSTERSPOKE} aksIngressControllerCertificate=${INGRESS_CONTROLLER_CERTIFICATE_BASE64} location=eastus2
az deployment group create -g rg-bu0001a0005-centralus -f pre-cluster-stamp.bicep -p targetVnetResourceId=${RESOURCEID_VNET_CLUSTERSPOKE} aksIngressControllerCertificate=${INGRESS_CONTROLLER_CERTIFICATE_BASE64}
```

## Quarantine pattern
Expand Down Expand Up @@ -74,7 +74,7 @@ Using a security agent that is container-aware and can operate from within the c
# Get your quarantine Azure Container Registry service name
# You only deployed one ACR instance in this walkthrough, but this could be
# a separate, dedicated quarantine instance managed by your IT team.
ACR_NAME_QUARANTINE=$(az deployment group show -g rg-bu0001a0005 -n pre-cluster-stamp --query properties.outputs.quarantineContainerRegistryName.value -o tsv)
ACR_NAME_QUARANTINE=$(az deployment group show -g rg-bu0001a0005-centralus -n pre-cluster-stamp --query properties.outputs.quarantineContainerRegistryName.value -o tsv)

# [Combined this takes about eight minutes.]
az acr import --source docker.io/falcosecurity/falco-no-driver:0.36.0 -t quarantine/falcosecurity/falco-no-driver:0.36.0 -n $ACR_NAME_QUARANTINE && \
Expand Down Expand Up @@ -114,7 +114,7 @@ Using a security agent that is container-aware and can operate from within the c

```bash
# Get your live Azure Container Registry service name
ACR_NAME=$(az deployment group show -g rg-bu0001a0005 -n pre-cluster-stamp --query properties.outputs.containerRegistryName.value -o tsv)
ACR_NAME=$(az deployment group show -g rg-bu0001a0005-centralus -n pre-cluster-stamp --query properties.outputs.containerRegistryName.value -o tsv)

# [Combined this takes about eight minutes.]
az acr import --source quarantine/falcosecurity/falco-no-driver:0.36.0 -r $ACR_NAME_QUARANTINE -t live/falcosecurity/falco-no-driver:0.36.0 -n $ACR_NAME && \
Expand Down Expand Up @@ -149,7 +149,7 @@ Once web traffic hits Azure Application Gateway (deployed in a future step), pub
1. Obtain the Azure Key Vault details and give the current user permissions and network access to import certificates.

```bash
KEYVAULT_NAME=$(az deployment group show --resource-group rg-bu0001a0005 -n pre-cluster-stamp --query properties.outputs.keyVaultName.value -o tsv)
KEYVAULT_NAME=$(az deployment group show --resource-group rg-bu0001a0005-centralus -n pre-cluster-stamp --query properties.outputs.keyVaultName.value -o tsv)
TEMP_ROLEASSIGNMENT_TO_UPLOAD_CERT=$(az role assignment create --role a4417e6f-fecd-4de8-b567-7b0420556985 --assignee-principal-type user --assignee-object-id $(az ad signed-in-user show --query 'id' -o tsv) --scope $(az keyvault show --name $KEYVAULT_NAME --query 'id' -o tsv) --query 'id' -o tsv)
echo TEMP_ROLEASSIGNMENT_TO_UPLOAD_CERT: $TEMP_ROLEASSIGNMENT_TO_UPLOAD_CERT

Expand Down Expand Up @@ -204,7 +204,7 @@ Your cluster will be bootstrapped using the Microsoft-provided GitOps extension,
You'll be using the [Secrets Store CSI Driver for Kubernetes](https://learn.microsoft.com/azure/aks/csi-secrets-store-driver) to mount the ingress controller's certificate which you stored in Azure Key Vault. Once mounted, your ingress controller will be able to use it. To make the CSI Provider aware of this certificate, it must be described in a `SecretProviderClass` resource. You'll update the supplied manifest file with this information now.

```bash
INGRESS_CONTROLLER_WORKLOAD_IDENTITY_CLIENT_ID_BU0001A0005_01=$(az deployment group show --resource-group rg-bu0001a0005 -n pre-cluster-stamp --query properties.outputs.ingressClientid.value -o tsv)
INGRESS_CONTROLLER_WORKLOAD_IDENTITY_CLIENT_ID_BU0001A0005_01=$(az deployment group show --resource-group rg-bu0001a0005-centralus -n pre-cluster-stamp --query properties.outputs.ingressClientid.value -o tsv)
echo INGRESS_CONTROLLER_WORKLOAD_IDENTITY_CLIENT_ID_BU0001A0005_01: $INGRESS_CONTROLLER_WORKLOAD_IDENTITY_CLIENT_ID_BU0001A0005_01

sed -i -e "s/KEYVAULT_NAME/${KEYVAULT_NAME}/" -e "s/KEYVAULT_TENANT/${TENANTID_AZURERBAC}/" -e "s/INGRESS_CONTROLLER_WORKLOAD_IDENTITY_CLIENT_ID_BU0001A0005_01/${INGRESS_CONTROLLER_WORKLOAD_IDENTITY_CLIENT_ID_BU0001A0005_01}/" ingress-nginx/akv-tls-provider.yaml
Expand Down
Loading