airflow-materials-aws

Materials for the next course

Links

• 4.16 Configuring the workstation
• 4.17 Configuring Cloud9 with the Admin account
• 4.20 Creating and configuring the Git repository for GitOps
• 4.22 Creating the Cluster with eksctl
• 4.23 Installing and Configuring Flux
• 5.30 Installing the EBS Driver

Configuring the workstation

• Update packages of the instance

sudo yum -y update

• Create a python virtual environment

python3 -m venv .sandbox

• Active the python virtual environment

source .sandbox/bin/activate

• Upgrade pip

pip install --upgrade pip

• Download and extract the latest release of eksctl with the following command.

curl --silent --location "https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_amd64.tar.gz" | tar xz -C /tmp
sudo mv /tmp/eksctl /usr/local/bin

• Test that your installation was successful with the following command.

eksctl version

• Download the latest release of Kubectl with the command

curl -LO https://storage.googleapis.com/kubernetes-release/release/v1.24.8/bin/linux/amd64/kubectl

• Make the kubectl binary executable.

chmod +x ./kubectl

• Move the binary in to your PATH.

sudo mv ./kubectl /usr/local/bin/kubectl

• Test to ensure the version you installed is up-to-date:

kubectl version --client

• Install Helm3

curl -sSL https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bash

• Check the version

helm version --short

• Download the stable repo

helm repo add stable https://charts.helm.sh/stable

• Config git (change the name by airflow-workstation and keep the email. Save and exit the file.)

git config --global user.name "airflow-workstation"

Configuring Cloud9 with the Admin account

• upgrade aws cli

pip install --upgrade awscli && hash -r

• install some utilities

sudo yum -y install jq gettext bash-completion moreutils

• go the settings, AWS settings and turn off temporary credentials

• remove temporary credentials

rm -vf ${HOME}/.aws/credentials

• configure aws env variables
• The following get-caller-identity example displays information about the IAM identity used to authenticate the request

aws configure

aws sts get-caller-identity

export ACCOUNT_ID=

export AWS_REGION=

• update the file bash_profile and configure aws

echo "export ACCOUNT_ID=${ACCOUNT_ID}" | tee -a ~/.bash_profile
echo "export AWS_REGION=${AWS_REGION}" | tee -a ~/.bash_profile

aws configure set default.region ${AWS_REGION}
aws configure get default.region

Creating and configuring the Git repository for GitOps

• Press return for all questions by keeping the defaults and empty passphrase.

ssh-keygen -t rsa

• copy public key to github ssh keys

cat /home/ec2-user/.ssh/id_rsa.pub

Creating the Cluster with eksctl

cd airflow-materials-aws

• Install the key

aws ec2 import-key-pair --key-name "airflow-workstation" --public-key-material fileb:///home/ec2-user/.ssh/id_rsa.pub

• Install the key (pennvet)

aws ec2 import-key-pair --key-name "airflow-workstation-pennvet" --public-key-material fileb:///home/ec2-user/.ssh/id_rsa.pub

• Install the key (johnvirtuon)

aws ec2 import-key-pair --key-name "airflow-workstation-john" --public-key-material fileb:///home/ec2-user/.ssh/id_rsa.pub

• Install aws-iam-authenticator
• Othwerise, doesn't look for the token and kubectl can't connect

curl -Lo aws-iam-authenticator https://github.com/kubernetes-sigs/aws-iam-authenticator/releases/download/v0.5.9/aws-iam-authenticator_0.5.9_linux_amd64
chmod +x ./aws-iam-authenticator

mkdir -p $HOME/bin && cp ./aws-iam-authenticator $HOME/bin/aws-iam-authenticator && export PATH=$PATH:$HOME/bin
echo 'export PATH=$PATH:$HOME/bin' >> ~/.bashrc
aws-iam-authenticator help

• Install AWS V2

curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"

unzip awscliv2.zip
sudo ./aws/install --update

aws --version

cd airflow-materials-aws

• Create the cluster

eksctl create cluster -f cluster.yml

• Check if the cluster is healthy

kubectl get nodes

kubectl get pods --all-namespaces

Installing and Configuring Flux

curl -s https://fluxcd.io/install.sh | sudo bash

flux --version

export GITHUB_TOKEN=<your_token_from_github>

echo $GITHUB_TOKEN

flux bootstrap github \
  --owner=paulkur \
  --repository=airflow-eks-config \
  --branch=main \
  --interval=15s \
  --personal

mkdir airflow-eks-config/{workloads,releases,namespaces}
find airflow-eks-config/ -type d -exec touch {}/.keep \;

cd airflow-eks-config
git add .
git commit -am "directory structure"
git push

• after adding namespaces

cd airflow-eks-config
git add .
git commit -am "add a new namespaces"
git push

################## Other account

• for personal aws johnvirtuon account

flux bootstrap github \
  --owner=paulkur \
  --repository=airflow-eks-config-john \
  --branch=main \
  --interval=15s \
  --personal

mkdir airflow-eks-config-john/{workloads,releases,namespaces}
find airflow-eks-config-john/ -type d -exec touch {}/.keep \;

cd airflow-eks-config-john
git add .
git commit -am "directory structure"
git push

• after adding namespaces

cd airflow-eks-config-john
git add .
git commit -am "add a new namespaces"
git push

• check namespaces

kubectl get namespaces

Installing the EBS Driver

• oficial git repo here, installation here
• commands to run

cd airflow-eks-config
helm repo add aws-ebs-csi-driver https://kubernetes-sigs.github.io/aws-ebs-csi-driver
helm repo update

helm upgrade --install aws-ebs-csi-driver \
    --namespace kube-system \
    aws-ebs-csi-driver/aws-ebs-csi-driver

kubectl commands

• check pods

watch kubectl get pods -n dev

• enter pod

kubectl exec -it airflow-dev-scheduler-6bf9b46568-xrhgq -n dev -c scheduler -- /bin/bash

• list dags

airflow dags list

• delete all pods

kubectl delete pods --all --force --grace-period=0 -n dev

Flux commands

flux get all

• filter error messages

flux logs --follow --level=error --all-namespaces