Skip to content

Commit

Permalink
Migrate to Flux v2 AKS Extension (#256)
Browse files Browse the repository at this point in the history
* Add required preview registrations

* Add required firewall changes, remove flux container fetching allowance

* Missed firewall entries

* Install flux into cluster

* No longer install flux (yaml only, docs changes later)

* One more FW change

* default, bootstrapping configuration

* add repo config override option

* Update the flow to support insta-bootstrapping

* Update image source to use my ACR instance instead of a public container registry. IGNORE TESTING ONLY

* fixup (unrelated to change)

* test adding mic exception

* minor updates

* poo -> pod

* Enable just the flux controllers we need

* file naming and readme cleanups. Also aks 1.22.4 bump

* Update Kured

* wording updates

* More minor wording tweaks

* remove testing values

* proper docs links

* missed an env variable rename

* manifest file whitespace

* revert wording change

* simplify the repo structure comment

* remove duplicate FW rule

* PR feedback

* Apply consistent casing

* Remove ".  " spacing :)
ckittel authored Dec 27, 2021
1 parent ba8a5d6 commit dd73e12
Showing 31 changed files with 671 additions and 681 deletions.
11 changes: 9 additions & 2 deletions 01-prerequisites.md
Original file line number Diff line number Diff line change
@@ -28,14 +28,21 @@ This is the starting point for the instructions on deploying the [AKS Baseline r

1. [Register the Azure Event Grid preview feature - `EventgridPreview`](https://docs.microsoft.com/azure/aks/quickstart-event-grid#register-the-eventgridpreview-preview-feature)

1. [Register the AKS Extensions preview feature - `AKS-ExtensionManager`](https://docs.microsoft.com/azure/aks/cluster-extensions?tabs=azure-cli#register-the-aks-extensionmanager-preview-features)

1. [Register the Kubernetes Configuration preview feature = `fluxConfigurations`](https://docs.microsoft.com/azure/azure-arc/kubernetes/tutorial-use-gitops-flux2#for-azure-kubernetes-service-clusters)

```bash
az feature register --namespace "Microsoft.ContainerService" -n "EventgridPreview"
az feature register --namespace "Microsoft.ContainerService" -n "AKS-ExtensionManager"
az feature register --namespace "Microsoft.KubernetesConfiguration" -n "fluxConfigurations"

# Keep running until all say "Registered." (This may take up to 20 minutes.)
az feature list -o table --query "[?name=='Microsoft.ContainerService/EventgridPreview'].{Name:name,State:properties.state}"
az feature list -o table --query "[?name=='Microsoft.ContainerService/EventgridPreview' || name=='Microsoft.ContainerService/AKS-ExtensionManager' || name=='Microsoft.KubernetesConfiguration/fluxConfigurations'].{Name:name,State:properties.state}"

# When all say "Registered" then re-register the AKS resource provider
# When all say "Registered" then re-register the AKS and related resource providers
az provider register --namespace Microsoft.ContainerService
az provider register --namespace Microsoft.KubernetesConfiguration
```

1. Clone/download this repo locally, or even better fork this repository.
2 changes: 1 addition & 1 deletion 02-ca-certificates.md
Original file line number Diff line number Diff line change
@@ -12,7 +12,7 @@ Now that you have the [prerequisites](./01-prerequisites.md) met, follow the ste

1. Generate a client-facing self-signed TLS certificate

> :book: Contoso Bicycle needs to procure a CA certificate for the web site. As this is going to be a user-facing site, they purchase an EV cert from their CA. This will serve in front of the Azure Application Gateway. They will also procure another one, a standard cert, to be used with the AKS Ingress Controller. This one is not EV, as it will not be user facing.
> :book: Contoso Bicycle needs to procure a CA certificate for the web site. As this is going to be a user-facing site, they purchase an EV cert from their CA. This will serve in front of the Azure Application Gateway. They will also procure another one, a standard cert, to be used with the AKS Ingress Controller. This one is not EV, as it will not be user facing.
:warning: Do not use the certificate created by this script for actual deployments. The use of self-signed certificates are provided for ease of illustration purposes only. For your cluster, use your organization's requirements for procurement and lifetime management of TLS certificates, _even for development purposes_.

6 changes: 3 additions & 3 deletions 03-aad.md
Original file line number Diff line number Diff line change
@@ -67,13 +67,13 @@ Following the steps below you will result in an Azure AD configuration that will

## Kubernetes RBAC backing store

AKS supports backing Kubernetes with Azure AD in two different modalities. One is direct association between Azure AD and Kubernetes `ClusterRoleBindings`/`RoleBindings` in the cluster. This is possible no matter if the Azure AD tenant you wish to use to back your Kubernetes RBAC is the same or different than the Tenant backing your Azure resources. If however the tenant that is backing your Azure resources (Azure RBAC source) is the same tenant you plan on using to back your Kubernetes RBAC, then instead you can add a layer of indirection between Azure AD and your cluster by using Azure RBAC instead of direct cluster `RoleBinding` manipulation. When performing this walk-through, you may have had no choice but to associate the cluster with another tenant (due to the elevated permissions necessary in Azure AD to manage groups and users); but when you take this to production be sure you're using Azure RBAC as your Kubernetes RBAC backing store if the tenants are the same. Both cases still leverage integrated authentication between Azure AD and AKS, Azure RBAC simply elevates this control to Azure RBAC instead of direct yaml-based management within the cluster which usually will align better with your organization's governance strategy.
AKS supports backing Kubernetes with Azure AD in two different modalities. One is direct association between Azure AD and Kubernetes `ClusterRoleBindings`/`RoleBindings` in the cluster. This is possible no matter if the Azure AD tenant you wish to use to back your Kubernetes RBAC is the same or different than the Tenant backing your Azure resources. If however the tenant that is backing your Azure resources (Azure RBAC source) is the same tenant you plan on using to back your Kubernetes RBAC, then instead you can add a layer of indirection between Azure AD and your cluster by using Azure RBAC instead of direct cluster `RoleBinding` manipulation. When performing this walk-through, you may have had no choice but to associate the cluster with another tenant (due to the elevated permissions necessary in Azure AD to manage groups and users); but when you take this to production be sure you're using Azure RBAC as your Kubernetes RBAC backing store if the tenants are the same. Both cases still leverage integrated authentication between Azure AD and AKS, Azure RBAC simply elevates this control to Azure RBAC instead of direct yaml-based management within the cluster which usually will align better with your organization's governance strategy.

### Azure RBAC _[Preferred]_

If you are using a single tenant for this walk-through, the cluster deployment step later will take care of the necessary role assignments for the groups created above. Specifically, in the above steps, you created the Azure AD security group `cluster-ns-a0008-readers-bu0001a000800` that is going to be a namespace reader in namespace `a0008` and the Azure AD security group `cluster-admins-bu0001a000800` is is going to contain cluster admins. Those group Object IDs will be associated to the 'Azure Kubernetes Service RBAC Reader' and 'Azure Kubernetes Service RBAC Cluster Admin' RBAC role respectively, scoped to their proper level within the cluster.
If you are using a single tenant for this walk-through, the cluster deployment step later will take care of the necessary role assignments for the groups created above. Specifically, in the above steps, you created the Azure AD security group `cluster-ns-a0008-readers-bu0001a000800` that is going to be a namespace reader in namespace `a0008` and the Azure AD security group `cluster-admins-bu0001a000800` is is going to contain cluster admins. Those group Object IDs will be associated to the 'Azure Kubernetes Service RBAC Reader' and 'Azure Kubernetes Service RBAC Cluster Admin' RBAC role respectively, scoped to their proper level within the cluster.

Using Azure RBAC as your authorization approach is ultimately preferred as it allows for the unified management and access control across Azure Resources, AKS, and Kubernetes resources. At the time of this writing there are four [Azure RBAC roles](https://docs.microsoft.com/azure/aks/manage-azure-rbac#create-role-assignments-for-users-to-access-cluster) that represent typical cluster access patterns.
Using Azure RBAC as your authorization approach is ultimately preferred as it allows for the unified management and access control across Azure Resources, AKS, and Kubernetes resources. At the time of this writing there are four [Azure RBAC roles](https://docs.microsoft.com/azure/aks/manage-azure-rbac#create-role-assignments-for-users-to-access-cluster) that represent typical cluster access patterns.

### Direct Kubernetes RBAC management _[Alternative]_

2 changes: 1 addition & 1 deletion 04-networking.md
Original file line number Diff line number Diff line change
@@ -104,4 +104,4 @@ The following two resource groups will be created and populated with networking
### Next step

:arrow_forward: [Deploy the AKS cluster](./05-aks-cluster.md)
:arrow_forward: [Prep for cluster bootstrapping](./05-bootstrap-prep.md)
88 changes: 88 additions & 0 deletions 05-bootstrap-prep.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,88 @@
# Prep for cluster bootstrapping

Now that the [hub-spoke network is provisioned](./04-networking.md), the next step in the [AKS secure Baseline reference implementation](./) is preparing what your AKS cluster should be bootstrapped with.

## Expected results

Container registries often have a lifecycle that extends beyond the scope of a single cluster. They can be scoped broadly at organizational or business unit levels, or can be scoped at workload levels, but usually are not directly tied to the lifecycle of any specific cluster instance. For example, you may do blue/green _cluster instance_ deployments, both using the same container registry. Even though clusters came and went, the registry stays intact.

* Azure Container Registry (ACR) is deployed, and exposed as a private endpoint.
* ACR is populated with images your cluster will need as part of its bootstrapping process.
* Log Analytics is deployed and ACR platform logging is configured. This workspace will be used by your cluster as well.

The role of this pre-existing ACR instance is made more prominant when we think about cluster bootstrapping. That is the process that happens after Azure Resource deployment of the cluster, but before your first workload lands in the cluster. The cluster will be bootstrapped immedately and automatically after resource deployment, which means you'll need ACR in place to act as your official OCI artifact repository for required images and Helm charts used in that bootstrapping process.

### Method

We'll be bootstrapping this cluster with the Flux GitOps agent as installed by the AKS extension. This specific choice does not imply that Flux or GitOps in general is the only approach to bootstrapping. Consider your organizational familiarity and acceptance of tooling like this and decide if cluster baseline management should be performed with GitOps or via your deployment pipelines. If you are running a fleet of clusters, a GitOps approach is highly recommended for uniformity and easier governance. When running only a few clusters, GitOps might be seen as "too much" and you might instead opt for integrating that process into one or more deployment pipelines to ensure bootstrapping takes place. No matter which way you go, you'll need your bootstrapping artifacts ready to go before you start your cluster deployment so that you can minimize the time between cluster deployment and bootstrapping. Using the Flux AKS Extension allows your cluster to start already bootstrapped and sets you up with a solid management foundation going forward.

## Steps

1. Create the AKS cluster resource group.

> :book: The app team working on behalf of business unit 0001 (BU001) is looking to create an AKS cluster of the app they are creating (Application ID: 0008). They have worked with the organization's networking team and have been provisioned a spoke network in which to lay their cluster and network-aware external resources into (such as Application Gateway). They took that information and added it to their [`acr-stamp.json`](./acr-stamp.json), [`cluster-stamp.json`](./cluster-stamp.json), and [`azuredeploy.parameters.prod.json`](./azuredeploy.parameters.prod.json) files.
>
> They create this resource group to be the parent group for the application.
```bash
# [This takes less than one minute.]
az group create --name rg-bu0001a0008 --location eastus2
```

1. Get the AKS cluster spoke VNet resource ID.

> :book: The app team will be deploying to a spoke VNet, that was already provisioned by the network team.
```bash
export RESOURCEID_VNET_CLUSTERSPOKE_AKS_BASELINE=$(az deployment group show -g rg-enterprise-networking-spokes -n spoke-BU0001A0008 --query properties.outputs.clusterVnetResourceId.value -o tsv)
```

1. Deploy the container registry template.

```bash
# [This takes about three minutes.]
az deployment group create -g rg-bu0001a0008 -f acr-stamp.json -p targetVnetResourceId=${RESOURCEID_VNET_CLUSTERSPOKE_AKS_BASELINE}
```

1. Import cluster management images to your container registry.

> Public container registries are subject to faults such as outages or request throttling. Interruptions like these can be crippling for a system that needs to pull an image _right now_. To minimize the risks of using public registries, store all applicable container images in a registry that you control, such as the SLA-backed Azure Container Registry.
```bash
# Get your ACR instance name
export ACR_NAME_AKS_BASELINE=$(az deployment group show -g rg-bu0001a0008 -n acr-stamp --query properties.outputs.containerRegistryName.value -o tsv)

# Import core image(s) hosted in public container registries to be used during bootstrapping
az acr import --source docker.io/weaveworks/kured:1.9.0 -n $ACR_NAME_AKS_BASELINE
```

> In this walkthrough, there is only one image that is included in the bootstrapping process. It's included as an example/reference for this process. Your choice to use kured or any other images, including helm charts, as part of your bootstrapping is yours to make.
1. Update bootstrapping manifests to pull from your ACR instance. _Optional. Fork required._

> Your cluster will immedately begin processing the manifests in [`cluster-manifests/`](./cluster-manifests/) due to the bootstrapping configuration that will be applied to it. So, before you deploy the cluster now would be the right time push the following changes to your fork so that it will use your files instead of the files found in the original mspnp repo which point to public container registries:
>
> * update the one `image:` value in [`kured.yaml`](./cluster-manifests/cluster-baseline-settings/kured.yaml) to use your container registry instead of a public container registry. See the comment in the file for instructions (or you can simply run the command below.)
:warning: Without updating these files and using your own fork, you will be deploying your cluster such that it takes dependencies on public container registries. This is generally okay for exploratory/testing, but not suitable for production. Before going to production, ensure _all_ image references you bring to your cluster are from _your_ container registry (link imported in the prior step) or another that you feel confident relying on.

```bash
sed -i "s:docker.io:${ACR_NAME_AKS_BASELINE}.azurecr.io:" ./cluster-manifests/cluster-baseline-settings/kured.yaml

git commit -a -m "Update image source to use my ACR instance instead of a public container registry."
git push
```

### Save your work in-progress

```bash
# run the saveenv.sh script at any time to save environment variables created above to aks_baseline.env
./saveenv.sh

# if your terminal session gets reset, you can source the file to reload the environment variables
# source aks_baseline.env
```

### Next step

:arrow_forward: [Deploy the AKS cluster](./06-aks-cluster.md)
Loading

0 comments on commit dd73e12

Please sign in to comment.