Skip to content

Commit

Permalink
merge alessandro dev branch (#4)
Browse files Browse the repository at this point in the history 
* argocd self-manage

* infra name

* policy

* root-on root

* name

* argocd app and values

* exclude values

* exclude

* multi sources

* root path for values

* ref

* syncOptions

* root-app-infra

* root-app-infra no

* move values

* allow resource in proj

* better excl

* new path

* rm values

* * in exclude

* argocd name

* application.namespaces: "*"

* ingress-nginx-values.yaml

* ext-dns different ns

* - CreateNamespace=true

* ns

* ns

* ext-dns chart

* conf extdns

* parms for ext-dns

* annotation nginx

* no id

* v1.11.1cert-manage

* cluster-issuer

* cert-manager-issuers

* thanos

* monitoring

* objstoreConfig

* healthz

* grafana

* auth

* prom

* - ServerSideApply=true

* cert-manager/cert-manager#4114

* port

* 19291

* multicluster

* Main Org.

* dash1

* no need for app for dash

* exclude dashes from root

* dash name

* dash app

* ns

* remove app

* dash2

* all mixin dashes

* rename

* newline

* 2

* more dah

* fix some dash

* more dasj

* more dash

* pvv dash

* pv

* last dash

* fixes

* kyverno

* https://kyverno.github.io/kyverno/

* kyverno2

* replace true

* policy

* exclude

* appset

* workload

* tabs

* description

* infra proj

* typo

* proj

* (

* argocd ingress

* typo

* ingressClassName

* vcluster helm chart

* path

* sources

* url

* https://charts.loft.sh

* policy vc*

* cluster name

* test

* fixed dahs

* single source for vcluster

* prom on workload

* move to appset

* proper prom

* proj

* ns

* path

* no node-expo

* - ServerSideApply=true

* remove clusters

* vc34

* better prome

* nodeExporter:
  enabled: false

* 5-6

* 123

* use argocd-app

* disable argo self-manage

* re-enable

* replace=true

* - ServerSideApply=true

* annotations:
    argocd.argoproj.io/sync-wave: "5"

* disabled

* move

* mapServices

* prometheus-sample-app

* nane

* pullpolicy

* docker pull ams0/prometheus-sample-app

* capi-operator

* only capi-op

* -

* root-app-infra exclude

* ns

* ns2

* repo

* cluster-api-operator

* operato

* ignore diff

* ingorediff

* InfrastructureProvider

* test

* ''

* 1

* ""

* no infra

* azure

* X

* comment

* no core

* capi machinepool

* wave

* different strategy

* spaces

* disabled capi-operator

* gitops/management/argocd/argocd-values.yaml

* gitignore

* readme

* readme

* k8sis.fun

* k8sis.fun

* Added imdb app

* capz policy

* readme

* policy for capz cluster

* no precondition

* replace branch

* helm req

* capz appset

---------

Co-authored-by: Alessandro Vozza <alessandro.vozza@microsoft.com>
Co-authored-by: Joaquin Rodriguez <rjoaquin@microsoft.com>
  • Loading branch information
@joaquinrz
3 people committed Apr 20, 2023
1 parent 00f7a5f commit d05647f
Show file tree
Hide file tree
Showing 60 changed files with 26,447 additions and 102 deletions.
1 change: 1 addition & 0 deletions .gitignore
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
argocd-values-local.yaml
186 changes: 84 additions & 102 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,157 +1,139 @@
# Ephemeral Clusters as a Service with ClusterAPI and GitOps
# Ephemeral Clusters as a Service with vcluster, ClusterAPI and ArgoCD

![License](https://img.shields.io/badge/license-MIT-green.svg)

Session @KubeconEU 2023 in Amsterdam: [<https://sched.co/1HyXe>](https://sched.co/1HyXe)

## Overview

Welcome to Ephemeral Clusters as a Service with ClusterAPI and GitOps. GitOps has rapidly gained popularity in recent years, with its many benefits over traditional CI/CD tools. However, with increased adoption comes the challenge of managing multiple Kubernetes clusters across different cloud providers. At scale, ensuring observability and security across all clusters can be particularly difficult.
Welcome to Ephemeral Clusters as a Service with [vClusters](https://www.vcluster.com), [ClusterAPI](https://cluster-api.sigs.k8s.io) and [ArgoCD](https://argo-cd.readthedocs.io/en/stable/). GitOps has rapidly gained popularity in recent years, with its many benefits over traditional CI/CD tools. However, with increased adoption comes the challenge of managing multiple Kubernetes clusters across different cloud providers. At scale, ensuring observability and security across all clusters can be particularly difficult.

This repository demonstrates how open-source tools, such as ClusterAPI, ArgoCD, and Prometheus+Thanos, can be used to effectively manage and monitor large-scale Kubernetes deployments. We will walk you through a sample that automates the deployment of several clusters and applications securely and with observability in mind.

## Prerequisites

For our sample will be using Azure Kubernetes Service (AKS). Before starting, you will need the following:

- An Azure account
- An Azure account (already logged in with the Azure CLI)
- Azure CLI [download](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli?view=azure-cli-latest)
- ArgoCD CLI [download](https://argo-cd.readthedocs.io/en/stable/cli_installation/)
- kubectl [download](https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/)
- helm [download](https://helm.sh/docs/intro/install/)
- clusterctl [download](https://cluster-api.sigs.k8s.io/user/quick-start.html)
- [Helm CLI](https://helm.sh) and `envsubst`
- Optional but recommented (and the instructions below assume you have one) A working DNS zone in Azure, to use proper DNS names and automatic certifcates provisioning with Let'sEncrypt

Everything else is installed via ArgoCD, so no need for any extra CLI!

## Step 1: Create an ArgoCD management cluster with AKS

To create a new management cluster in AKS, run the following commands. Otherwise, if you already have an existing AKS cluster, you can skip this step and proceed to connecting to the existing AKS cluster.
To create a new management cluster in AKS, run the following commands. Otherwise, if you already have an existing AKS cluster, you can skip this step and proceed to connecting to the existing AKS cluster. Change accoring to your liking, specially the `AZURE_DNS_ZONE`:

```bash
export AZURE_SUBSCRIPTION_ID=$(az account show --query id -o tsv)
export CLUSTER_RG=clusters
export CLUSTER_NAME=management
export LOCATION=westeurope
export IDENTITY_NAME=gitops$RANDOM
export NODE_COUNT=2
export AZ_AKS_VERSION=1.25.6
export AZURE_DNS_ZONE=k8sis.fun
export AZURE_DNS_ZONE_RESOURCE_GROUP=dns
```

export AZURE_SUBSCRIPTION_ID=<yourSubscriptionId>

# Connecting to Azure
az login --use-device-code

# Change the active subscription using the subscription name
az account set --subscription $AZURE_SUBSCRIPTION_ID
Create a resource group for your AKS cluster with the following command, replacing <resource-group> with a name for your resource group and <location> with the Azure region where you want your resources to be located:

# Create a resource group for your AKS cluster with the following command, replacing <resource-group> with a name for your resource group and <location> with the Azure region where you want your resources to be located:
az group create --name <resource-group> --location <location>
```bash
az group create --name $CLUSTER_RG --location $LOCATION
```

# Create an AKS cluster with the following command, replacing <cluster-name> with a name for your cluster, and <node-count> with the number of nodes you want in your cluster:
az aks create --resource-group <resource-group> --name <cluster-name> --location <location> --generate-ssh-keys
To use automatic DNS name updates via external-dns, we need to create a new managed identity and assign the role of DNS Contributor to the resource group containg the zone resource

# Connect to the AKS cluster:
az aks get-credentials --resource-group <resource-group> --name <cluster-name>
```bash
IDENTITY=$(az identity create -k $AZ_AKS_VERSION -n $IDENTITY_NAME -g $CLUSTER_RG --query id -o tsv)
IDENTITY_CLIENTID=$(az identity show -g $CLUSTER_RG -n $IDENTITY_NAME -o tsv --query clientId)

#Verify that you can connect to the AKS cluster:
kubectl get nodes
DNS_ID=$(az network dns zone show --name $AZURE_DNS_ZONE \
--resource-group $AZURE_DNS_ZONE_RESOURCE_GROUP --query "id" --output tsv)

az role assignment create --role "DNS Zone Contributor" --assignee $IDENTITY_CLIENTID --scope $DNS_ID
```

## Step 2: Install ArgoCD
Create an AKS cluster with the following command:

ArgoCD is an open-source continuous delivery tool designed to simplify the deployment of applications to Kubernetes clusters. It allows developers to manage and deploy applications declaratively, reducing the amount of manual work involved in the process. With ArgoCD, developers can automate application deployment, configuration management, and application rollouts, making it easier to deliver applications reliably to production environments.
```bash
az aks create -k $AZ_AKS_VERSION -y -g $CLUSTER_RG -s Standard_B4ms -c $NODE_COUNT \
--assign-identity $IDENTITY --assign-kubelet-identity $IDENTITY --network-plugin kubenet -n $CLUSTER_NAME
```

You can install ArgoCD on your Kubernetes cluster by running the following commands in your terminal or command prompt. These commands will download and install ArgoCD on your cluster, allowing you to use it for GitOps-based continuous delivery of your applications
Connect to the AKS cluster:

```bash
kubectl create namespace argocd

# Add ArgoCD Helm Repo
helm repo add argo https://argoproj.github.io/argo-helm
az aks get-credentials --resource-group $CLUSTER_RG --name $CLUSTER_NAME
```

# Update Helm Repo
helm repo update
Verify that you can connect to the AKS cluster:
```bash
kubectl get nodes
```

# Install ArgoCD
helm install -n argocd argocd argo/argo-cd
## Step 2: Install ArgoCD

# Verify that ArgoCD is running:
kubectl get pods -n argocd
ArgoCD is an open-source continuous delivery tool designed to simplify the deployment of applications to Kubernetes clusters. It allows developers to manage and deploy applications declaratively, reducing the amount of manual work involved in the process. With ArgoCD, developers can automate application deployment, configuration management, and application rollouts, making it easier to deliver applications reliably to production environments.

# Access the ArgoCD web UI by running the following command, and then open the URL in a web browser:
kubectl port-forward svc/argocd-server -n argocd 8080:443
You can install ArgoCD on your Kubernetes cluster by running the following commands in your terminal or command prompt. These commands will download and install ArgoCD on your cluster, allowing you to use it for GitOps-based continuous delivery of your applications

# Log in to the ArgoCD http://localhost:8080 with the following credentials:
# - Username: admin
# - Password: Retrieve the ArgoCD password by running one of the following command:
> **_NOTE:_** Make sure to update the values for ingress hostname in the various helm charts under `gitos/management` folder; we will update the readme when we find a better way to dynamically inject these values into the helm charts deployed by ArgoCD
argocd admin initial-password -n argocd
Add ArgoCD Helm Repo:

# Alternatively, you can also retrieve the credentials using kubectl.
kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d
```bash
helm repo add argo https://argoproj.github.io/argo-helm
helm repo update
```

## Step 2: Install Prometheus and Grafana

Prometheus and Grafana are two popular open-source tools used for monitoring and visualizing data. Prometheus collects metrics from various sources, while Grafana provides customizable dashboards for displaying this data. Together, they offer a powerful and flexible monitoring solution that can help developers and system administrators gain insights into the performance of their applications and systems.

The following commands will help you install Prometheus and Grafana in your cluster
Edit the `gitops/management/argocd/argocd-values.yaml` with your hostname and domain name for ArgoCD ingress, then install ArgoCD:

```bash
# Add prometheus-community helm chart. This Helm chart by default also includes Grafana
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

# Create a monitoring namespace for these applications
kubectl create ns monitoring

# Install the Helm Chart
helm install -n monitoring kube-stack-prometheus prometheus-community/kube-prometheus-stack

# Access Grafana UI http://localhost:8081/
# Credentials: admin:prom-operator
kubectl port-forward service/kube-stack-prometheus-grafana -n monitoring 8081:80

# Access Prometheus UI http://localhost:9090/
kubectl port-forward service/kube-stack-prometheus-kube-prometheus -n monitoring 9090:9090
envsubst < gitops/management/argocd/argocd-values.yaml > gitops/management/argocd/argocd-values-local.yaml

helm upgrade -i -n argocd \
--version 5.29.1 \
--create-namespace \
--values gitops/management/argocd/argocd-values-local.yaml \
argocd argo/argo-cd

helm upgrade -i -n argocd \
--version 0.0.9\
--create-namespace \
--values argocd-initial-objects.yaml \
argocd-apps argo/argocd-apps
```

## Step 3: Installing Thanos

## Step 4: Bootstrap Management Cluster with ClusterAPI

To initialize the AKS cluster with Cluster API and turn it into the management cluster, follow these instructions. Once initialized, the management cluster will allow you to control and maintain a fleet of ephemeral clusters.
Verify that ArgoCD is running:

```bash
kubectl get pods -n argocd
```

# Enable support for managed topologies and experimental features
export CLUSTER_TOPOLOGY=true
export EXP_AKS=true
export EXP_MACHINE_POOL=true
Access the ArgoCD web UI by running the following command, and then open the URL in a web browser (ingress, external-dns and cert-manager take care of certificates and DNS hostname resolution):

# Create an Azure Service Principal in the Azure portal. (Note: Make sure this Service Principal has access to the resource group)
# Create an Azure Service Principal
export AZURE_SP_NAME="kubecon23capi"
```bash
open https://argocd.$AZURE_DNS_ZONE
```

az ad sp create-for-rbac \
--name $AZURE_SP_NAME \
--role contributor \
--scopes="/subscriptions/${AZURE_SUBSCRIPTION_ID}"
> **_NOTE:_** ArgoCD is in read-only mode for anonymous users, that should be enough to monitor the installaation progress, but if you want to change things, retrieve the secret with:
> `kubectl get secret -n argocd argocd-initial-admin-secret -o=jsonpath='{.data.password}'| base64 -D`
# TODO - Add command so that SP has role assigment to the resource group of the cluster
## Step 3: Bootstrap Management Cluster with ClusterAPI

export AZURE_TENANT_ID="<Tenant>"
export AZURE_CLIENT_ID="<AppId>"
export AZURE_CLIENT_SECRET="<Password>"
To initialize the AKS cluster with Cluster API and turn it into the management cluster, follow these instructions. Once initialized, the management cluster will allow you to control and maintain a fleet of ephemeral clusters. Unfortunately this part cannot be automated via ArgoCD just yet, although a promising effort is made in the `capi-operator` [repository](https://github.com/kubernetes-sigs/cluster-api-operator/tree/main):

# Base64 encode the variables
export AZURE_SUBSCRIPTION_ID_B64="$(echo -n "$AZURE_SUBSCRIPTION_ID" | base64 | tr -d '\n')"
export AZURE_TENANT_ID_B64="$(echo -n "$AZURE_TENANT_ID" | base64 | tr -d '\n')"
export AZURE_CLIENT_ID_B64="$(echo -n "$AZURE_CLIENT_ID" | base64 | tr -d '\n')"
export AZURE_CLIENT_SECRET_B64="$(echo -n "$AZURE_CLIENT_SECRET" | base64 | tr -d '\n')"
```bash
# Run the script, passing the namespace as a parameter (the Azure Managed Identity for the workload clusters)

# Settings needed for AzureClusterIdentity used by the AzureCluster
export AZURE_CLUSTER_IDENTITY_SECRET_NAME="cluster-identity-secret"
export CLUSTER_IDENTITY_NAME="cluster-identity"
export AZURE_CLUSTER_IDENTITY_SECRET_NAMESPACE="default"
./capz-init.sh default

# Create a secret to include the password of the Service Principal identity created in Azure
# This secret will be referenced by the AzureClusterIdentity used by the AzureCluster
kubectl create secret generic "${AZURE_CLUSTER_IDENTITY_SECRET_NAME}" --from-literal=clientSecret="${AZURE_CLIENT_SECRET}"
# Check the providers
kubectl get providers.clusterctl.cluster.x-k8s.io -A
```

# Initialize the management cluster for azure
clusterctl init --infrastructure azure
## Step 4: Deploy clusters via Pull Requests

# Create and apply an AzureClusterIdentity
envsubst < manifests/templates/aks-cluster-identity.yaml | kubectl apply -f -
``
Open a PR against your main branch, modifying
27 changes: 27 additions & 0 deletions apps/simple/deployment.yaml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# deployment
apiVersion: apps/v1
kind: Deployment
metadata:
name: prometheus-sample-app-deployment
spec:
selector:
matchLabels:
app: prometheus-sample-app
replicas: 5 # tells deployment to run 5 pods matching the template
template:
metadata:
labels:
app: prometheus-sample-app
annotations:
prometheus.io/scrape: 'true'
spec:
containers:
- name: prometheus-sample-app
image: ams0/prometheus-sample-app
imagePullPolicy: Always
command:
- "/bin/main"
args:
- "--metric_count=1"
ports:
- containerPort: 8080
13 changes: 13 additions & 0 deletions apps/simple/service-monitor.yaml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: prometheus-sample-app
labels:
app: prometheus-sample-app
release: prometheus
spec:
selector:
matchLabels:
app: prometheus-sample-app
endpoints:
- port: http
13 changes: 13 additions & 0 deletions apps/simple/service.yaml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
apiVersion: v1
kind: Service
metadata:
name: prometheus-sample-app
labels:
app: prometheus-sample-app
spec:
type: ClusterIP
selector:
app: prometheus-sample-app
ports:
- port: 8080
name: http
40 changes: 40 additions & 0 deletions argocd-initial-objects.yaml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
applications:
- name: root-app-infra
namespace: argocd
finalizers:
- resources-finalizer.argocd.argoproj.io
project: infra
source:
repoURL: https://github.com/joaquinrz/capi-gitops
targetRevision: main
path: gitops/management
directory:
recurse: true
exclude: '{*values.yaml,*dashboard.yaml,*policy.yaml,*resource.yaml}'
destination:
server: https://kubernetes.default.svc
namespace: argocd
syncPolicy:
automated:
selfHeal: true
prune: false
syncOptions:
- CreateNamespace=true
- ServerSideApply=true

projects:
- name: infra
namespace: argocd
finalizers:
- resources-finalizer.argocd.argoproj.io
description: Infrastructure components on management cluster
sourceRepos:
- '*'
destinations:
- namespace: "*"
server: https://kubernetes.default.svc
clusterResourceWhitelist:
- group: '*'
kind: '*'
sourceNamespaces:
- '*'
35 changes: 35 additions & 0 deletions capz-init.sh
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
#!/bin/bash
# Enable support for managed topologies and experimental features
export CLUSTER_TOPOLOGY=true
export EXP_MACHINE_POOL=true

# Create an Azure Service Principal in the Azure portal. (Note: Make sure this Service Principal has access to the resource group)
# Create an Azure Service Principal
export AZURE_SP_NAME="kubecon23capi$RANDOM"

export AZURE_SUBSCRIPTION_ID=$(az account show --query id -o tsv)
export AZURE_TENANT_ID=$(az account show --query tenantId -o tsv)
export AZURE_CLIENT_SECRET=$(az ad sp create-for-rbac --name $AZURE_SP_NAME --role contributor --scopes="/subscriptions/${AZURE_SUBSCRIPTION_ID}" --query password -o tsv)
export AZURE_CLIENT_ID=$(az ad sp list --display-name $AZURE_SP_NAME --query "[0].appId" -o tsv)

# Base64 encode the variables
export AZURE_SUBSCRIPTION_ID_B64="$(echo -n "$AZURE_SUBSCRIPTION_ID" | base64 | tr -d '\n')"
export AZURE_TENANT_ID_B64="$(echo -n "$AZURE_TENANT_ID" | base64 | tr -d '\n')"
export AZURE_CLIENT_ID_B64="$(echo -n "$AZURE_CLIENT_ID" | base64 | tr -d '\n')"
export AZURE_CLIENT_SECRET_B64="$(echo -n "$AZURE_CLIENT_SECRET" | base64 | tr -d '\n')"

# Settings needed for AzureClusterIdentity used by the AzureCluster
export AZURE_CLUSTER_IDENTITY_SECRET_NAME="cluster-identity-secret"
export CLUSTER_IDENTITY_NAME="cluster-identity"
export AZURE_CLUSTER_IDENTITY_SECRET_NAMESPACE=$1

# Create a secret to include the password of the Service Principal identity created in Azure
# This secret will be referenced by the AzureClusterIdentity used by the AzureCluster
kubectl create secret generic "${AZURE_CLUSTER_IDENTITY_SECRET_NAME}" --from-literal=clientSecret="${AZURE_CLIENT_SECRET}" -n ${AZURE_CLUSTER_IDENTITY_SECRET_NAMESPACE}

# Initialize the management cluster for azure
clusterctl init --infrastructure azure

sleep 10
# Create and apply an AzureClusterIdentity
envsubst < manifests/templates/aks-cluster-identity.yaml | kubectl apply -f -
Loading

0 comments on commit d05647f

Please sign in to comment.