Continuous Delivery to Kubernetes with Kotlin, Ktor, Gradle, Jib, Skaffold and Kubernetes (EKS & Minikube)
- Minikube (v1.29.0)
minikube start
- Kubernetes (v1.24.3 on Docker 20.10.17 on minikube v1.26.1, 1.23.7 on EKS)
- Java 17
- Kotlin (1.6.21)
- Gradle (v7.3)
- Skaffold (v2.3.0)
- Ktor (2.1.0)
- Jib (3.3.0)
skaffold dev
other terminal:
./web-eks.sh # EKS on AWS
# or
./web.sh # local minikube
The bash script checks the service endpoint for code changes becoming effective.
The web-eks.sh
script assumes kubernetes runs on EKS and the web kubernetes service is already deployed.
The web.sh
script assumes kubernetes runs on minikube and the web kubernetes service is already deployed.
The steps below assume an EKS Kubernetes cluster is in place.
The canonical way to create an EKS cluster would be eksctl: eksctl create cluster ...
.
That is a similar setup as in ALB INGRESS Controller CrashLoopBackOffs in AWS EKS on FARGATE.
The steps below assume an ENV variable $CLUSTER_NAME
that holds the Kubernetes cluster name in the terminal session.
- Define additional ENV variables for later use.
ENDPOINT_URL=$(aws eks describe-cluster --name $CLUSTER_NAME --query cluster.endpoint --output text)
echo $ENDPOINT_URL
CA_CERT=$(aws eks describe-cluster --name $CLUSTER_NAME --query cluster.certificateAuthority.data --output text)
echo $CA_CERT
AWS_PROFILE=test
echo $AWS_PROFILE
AWS_REGION=$(aws configure get region)
echo $AWS_REGION
- Configure
kubectl
with EKS cluster viaKUBECONFIG
ENV variable
ll ~/.kube
# in case the folder does not exists
mkdir -p ~/.kube
cat << KBCFG > ~/.kube/config-${CLUSTER_NAME}
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: $CA_CERT
server: $ENDPOINT_URL
name: $CLUSTER_NAME
contexts:
- context:
cluster: $CLUSTER_NAME
user: $CLUSTER_NAME
name: $CLUSTER_NAME
current-context: $CLUSTER_NAME
kind: Config
preferences: {}
users:
- name: $CLUSTER_NAME
user:
exec:
apiVersion: client.authentication.k8s.io/v1alpha1
args:
- --region
- $AWS_REGION
- eks
- get-token
- --cluster-name
- $CLUSTER_NAME
command: aws
env:
- name: AWS_PROFILE
value: $AWS_PROFILE
KBCFG
export KUBECONFIG=$KUBECONFIG:~/.kube/config-${CLUSTER_NAME}
echo $KUBECONFIG
- ECR login
# assuming aws cli v2 - https://aws.amazon.com/blogs/developer/aws-cli-v2-is-now-generally-available/
aws --version
# ecr login (https://github.com/aws/aws-cli/issues/4962)
echo $(aws ecr get-login-password)|docker login --password-stdin --username AWS https://$(aws sts get-caller-identity --query 'Account' --output text).dkr.ecr.${AWS_REGION}.amazonaws.com
- ECR repository
# use an existing ECR repository or create one: `aws ecr create-repository --repository-name ... `
export ECRREPO_NAME=[ktorjib]
echo $ECRREPO_NAME
export ECRREPO_URI=$(aws ecr describe-repositories --repository-names $ECRREPO_NAME --query 'repositories[0].repositoryUri' --output text)
echo $ECRREPO_URI
- Kube secret to configure docker registry
export KUBE_SECRET_LABEL=$(aws sts get-caller-identity --query 'Account' --output text)--${AWS_REGION}--${AWS_PROFILE}--ecr--registry--secret
echo $KUBE_SECRET_LABEL
export KUBE_SECRET_PASSWORD=$(aws ecr get-authorization-token --output text --query 'authorizationData[0].authorizationToken' | base64 -d | cut -d: -f2)
echo $KUBE_SECRET_PASSWORD
KUBE_SECRET_EMAIL=[me@you.com]
echo $KUBE_SECRET_EMAIL
# configure kubectl to log into ECR
kubectl delete secret --ignore-not-found $KUBE_SECRET_LABEL
kubectl create secret docker-registry $KUBE_SECRET_LABEL \
--docker-server=https://${ECRREPO_URI} \
--docker-username=AWS \
--docker-password="${KUBE_SECRET_PASSWORD}" \
--docker-email="${KUBE_SECRET_EMAIL}"
- Set up kubernetes namespace
export KTORJIB_K8S_NAMESPACE=ktorjib
echo ${KTORJIB_K8S_NAMESPACE}
kubectl create namespace ${KTORJIB_K8S_NAMESPACE}
kubectl get namespaces
- Skaffold config
# set up skaffold config
cat << SKFLDCFG > skaffold.yaml
# inspired by https://github.com/GoogleContainerTools/skaffold/blob/master/examples/jib-gradle/skaffold.yaml @ 2020 04 19
apiVersion: skaffold/v2beta2
kind: Config
build:
artifacts:
#- image: gcr.io/ktor-jib/kjib-image
- image: ${ECRREPO_URI}
jib: {}
SKFLDCFG
# optional: copy the skaffold config file for backup
cp skaffold.yaml skaffold-ecr.yaml_
- Start skaffold flow
skaffold dev --namespace KTORJIB_K8S_NAMESPACE
note: kubernetes namespace can be specified with ENV var SKAFFOLD_NAMESPACE
or cli parameter --namespace
- https://github.com/stelligent/skaffold_on_aws
- https://github.com/aws-samples/aws-microservices-deploy-options/blob/master/skaffold.md
#mac
minikube start --driver=hyperkit --kubernetes-version=1.21.2
#linux
minikube start --driver=docker --kubernetes-version=1.21.2
# start skaffold flow
skaffold dev
./gradlew jibDockerBuild --image=myimagename && docker run --rm -p 8080:8080 myimagename
access the endpoint:
curl http://0.0.0.0:8080
./gradlew run
access the endpoint:
curl http://0.0.0.0:8080
./gradlew test
./gradlew clean &&
docker stop $(docker ps -aq) &&
docker rm $(docker ps -aq) &&
docker rmi $(docker images -q)
FAILURE: Build failed with an exception
....
* What went wrong:
Script compilation error:
Line 29: java.sourceCompatibility = JavaVersion.VERSION_13
^ Unresolved reference: VERSION_13
1 error
FAILURE: Build failed with an exception.
...
* What went wrong:
Script compilation error:
Line 29: java.sourceCompatibility = JavaVersion.VERSION_14
^ Unresolved reference: VERSION_14
1 error
// works:
...
JavaVersion.VERSION_12
...
image = "openjdk:14"
// but for consistency
...
JavaVersion.VERSION_12
...
image = "openjdk:12"
- https://github.com/GoogleContainerTools/jib/search?q=java+14&type=Code
- https://github.com/GoogleContainerTools/jib/blob/master/jib-core/CHANGELOG.md
- GoogleContainerTools/jib#2015
- https://github.com/GoogleContainerTools/jib/pull/2017/files
- GoogleContainerTools/jib#2015 (comment)
- https://asm.ow2.io/versions.html
- java 8 or 11 -> https://github.com/GoogleContainerTools/distroless/tree/master/java
- https://github.com/GoogleContainerTools/distroless/tree/master/java#image-contents
- https://github.com/GoogleContainerTools/distroless/blob/master/examples/java/Dockerfile
- https://console.cloud.google.com/gcr/images/distroless/GLOBAL/java?gcrImageListsize=30&gcrImageListsort=-uploaded (version 11)