Skip to content
This repository has been archived by the owner on Jan 22, 2024. It is now read-only.

Commit

Permalink
Merge pull request #11 from clastix/issues/10
Browse files Browse the repository at this point in the history
refactoring of helm chart
  • Loading branch information
bsctl authored Apr 27, 2021
2 parents 849e820 + e6a8ba5 commit d721f89
Show file tree
Hide file tree
Showing 16 changed files with 232 additions and 163 deletions.
File renamed without changes.
9 changes: 8 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -242,7 +242,7 @@ Install through Helm:

$ helm install kubelived --namespace kube-system

Actually, the Helm chart does not install `keepalived` on the Kubernetes Control Plane. Instead, it deployes installer pods as as daemonset on the master nodes. The installer pods are responsible to deploy the `kube-keepalived.yaml` pod manifest in the `/etc/kubernetes/manifests` folder and the proper `keepalived.conf` configuration file in `/etc/keepalived` location of each master node.
Actually, the Helm chart does not install `keepalived` on the Kubernetes Control Plane. Instead, it deployes installer pods as daemonset on the master nodes. The installer pods are responsible to deploy the `kube-keepalived.yaml` pod manifest in the `/etc/kubernetes/manifests` folder and the proper `keepalived.conf` configuration file in `/etc/keepalived` location of each master node.

Check the installer pods:

Expand Down Expand Up @@ -378,6 +378,13 @@ Once we have the VIP and the VRRP working correctly, we need to assign the Kuber

Depending on the way we deploy the cluster, this endpoint can be required to be already available during the installation. To avoid this "*chicken and egg dilemma*" we can first assign the endpoint DNS name to one of the master nodes, deploy the cluster, and then deploy keepalived as static pods on all the masters. Once we have the floating VIP up and running we can assign the DNS name of the endpoint to the floating VIP.

## Usage on different namespaces
Kubelived can be used not only for assigning a VIP to the Control Plane. You can use kubelived also to assign a VIP to worker nodes, for example to expose an Ingress Controller or a service `NodePort` type. In suce case, you should not deploy on master nodes. Also, make sure you're checking the proper endpoint, eg. the Ingress Controller health page:

`health_service_check: '/usr/bin/curl -s http://localhost:1042/healthz -o /dev/null'`
instead of the Kubernetes APIs server.
## Monitoring keepalived
Sending USR1 signal to keepalive process will dump configuration data to `/tmp/keepalived.data`, and sending USR2 will write statistics to `/tmp/keepalived.stats` inside the container.

Expand Down
File renamed without changes.
6 changes: 3 additions & 3 deletions helm/Chart.yaml → charts/kubelived/Chart.yaml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
apiVersion: v2
name: kubelived
description: A Helm chart for installing Keepalived for Kubernetes Control Plane
description: A Helm chart for Kubernetes

# A chart can be either an 'application' or a 'library' chart.
#
Expand All @@ -15,9 +15,9 @@ type: application
# This is the chart version. This version number should be incremented each time you make changes
# to the chart and its templates, including the app version.
# Versions are expected to follow Semantic Versioning (https://semver.org/)
version: 0.1.1
version: 0.1.2

# This is the version number of the application being deployed. This version number should be
# incremented each time you make changes to the application. Versions are not expected to
# follow Semantic Versioning. They should reflect the version the application is using.
appVersion: 0.2.0
appVersion: 0.2.1
File renamed without changes.
Original file line number Diff line number Diff line change
Expand Up @@ -51,12 +51,12 @@ app.kubernetes.io/instance: {{ .Release.Name }}
{{- end }}

{{/*
Create the name of the service account to use
Create the name of the service account to use for installer
*/}}
{{- define "kubelived.serviceAccountName" -}}
{{- if .Values.serviceAccount.create }}
{{- default (include "kubelived.fullname" .) .Values.serviceAccount.name }}
{{- if .Values.installer.serviceAccount.create }}
{{- default (include "kubelived.fullname" .) .Values.installer.serviceAccount.name }}
{{- else }}
{{- default "default" .Values.serviceAccount.name }}
{{- default "default" .Values.installer.serviceAccount.name }}
{{- end }}
{{- end }}
47 changes: 47 additions & 0 deletions charts/kubelived/templates/keepalived-config.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: keepalived-config
namespace: {{ .Release.Namespace }}
labels:
{{- include "kubelived.labels" . | nindent 4 }}
data:
{{ .Values.keepalived.config_file }}: |
# keepalived global configuration
global_defs {
default_interface KEEPALIVED_INTERFACE
enable_script_security
}
vrrp_script {{ .Values.keepalived.health_service_name }} {
script {{ quote .Values.keepalived.health_service_check }}
interval {{ .Values.keepalived.health_service_interval }}
timeout {{ .Values.keepalived.health_service_timeout }}
rise {{ .Values.keepalived.health_service_rise }}
fall {{ .Values.keepalived.health_service_fall }}
user {{ .Values.keepalived.health_service_user }}
}
vrrp_instance VI_1 {
state {{ upper .Values.keepalived.initial_state }}
interface KEEPALIVED_INTERFACE
virtual_router_id {{ .Values.keepalived.router_id }}
priority KEEPALIVED_PRIORITY
advert_int {{ .Values.keepalived.health_service_interval }}
authentication {
auth_type PASS
auth_pass {{ .Values.keepalived.password | b64enc }}
}
track_script {
{{ .Values.keepalived.health_service_name }}
}
virtual_ipaddress {
{{ .Values.keepalived.virtual_address }} label KEEPALIVED_INTERFACE:VIP
}
{{- if .Values.keepalived.unicast_peers }}
unicast_peer {
{{- range .Values.keepalived.unicast_peers }}
{{ . }}
{{- end }}
}
{{- end }}
}
Original file line number Diff line number Diff line change
Expand Up @@ -4,67 +4,74 @@ metadata:
name: keepalived-installer
labels:
{{- include "kubelived.labels" . | nindent 4 }}
{{- with .Values.installer.annotations }}
annotations:
{{- toYaml . | nindent 4 }}
{{- end }}
namespace: {{ .Release.Namespace }}
spec:
selector:
matchLabels:
{{- include "kubelived.selectorLabels" . | nindent 6 }}
template:
metadata:
{{- with .Values.installer.podAnnotations }}
annotations:
scheduler.alpha.kubernetes.io/critical-pod: ""
{{- toYaml . | nindent 8 }}
{{- end }}
labels:
{{- include "kubelived.selectorLabels" . | nindent 8 }}
spec:
serviceAccountName: {{ include "kubelived.serviceAccountName" . }}
priorityClassName: system-node-critical
hostNetwork: true
hostNetwork: {{ .Values.installer.hostNetwork }}
nodeSelector:
node-role.kubernetes.io/master: ""
{{- toYaml .Values.installer.nodeSelector | nindent 8 }}
affinity:
{{- toYaml .Values.installer.affinity | nindent 8 }}
tolerations:
- key: CriticalAddonsOnly
operator: Exists
- effect: NoSchedule
key: node-role.kubernetes.io/master
{{- toYaml .Values.installer.tolerations | nindent 8 }}
containers:
- name: installer
securityContext:
privileged: true
image: {{ .Values.installer.repository }}:{{ .Values.installer.tag | default "latest" }}
imagePullPolicy: {{ .Values.installer.pullPolicy }}
command: ["./scripts/install.sh"]
image: {{ .Values.installer.image.repository }}:{{ .Values.installer.image.tag | default "latest" }}
imagePullPolicy: {{ .Values.installer.image.pullPolicy }}
args: ["sleep", "24h"]
lifecycle:
postStart:
exec:
command: ["/bin/sh", "-c", "./scripts/install.sh"]
preStop:
exec:
command: ["/bin/sh", "-c", "./scripts/uninstall.sh"]
volumeMounts:
- name: manifest
mountPath: /etc/kubernetes/manifests
- name: config
mountPath: {{ .Values.keepalived_config_path }}
mountPath: {{ .Values.keepalived.config_path }}
- name: keepalived-manifest
mountPath: /opt/kube-keepalived.yaml
subPath: kube-keepalived.yaml
- name: keepalived-config
mountPath: /opt/{{ .Values.keepalived_config_file }}
subPath: {{ .Values.keepalived_config_file }}
- name: keepalived-install-scripts
mountPath: /opt/{{ .Values.keepalived.config_file }}
subPath: {{ .Values.keepalived.config_file }}
- name: keepalived-scripts
mountPath: /scripts
volumes:
- name: manifest
hostPath:
path: /etc/kubernetes/manifests
- name: config
hostPath:
path: {{ .Values.keepalived_config_path }}
path: {{ .Values.keepalived.config_path }}
type: DirectoryOrCreate
- name: keepalived-manifest
configMap:
name: keepalived-manifest
- name: keepalived-config
configMap:
name: keepalived-config
- name: keepalived-install-scripts
- name: keepalived-scripts
configMap:
name: keepalived-install-scripts
name: keepalived-scripts
defaultMode: 0700
Original file line number Diff line number Diff line change
Expand Up @@ -16,17 +16,17 @@ data:
labels:
{{- include "kubelived.selectorLabels" . | nindent 8 }}
name: kube-keepalived
namespace: kube-system
namespace: {{ .Release.Namespace }}
spec:
containers:
- name: keepalived
image: {{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}
imagePullPolicy: {{ .Values.image.pullPolicy }}
image: {{ .Values.keepalived.image.repository }}:{{ .Values.keepalived.image.tag | default .Chart.AppVersion }}
imagePullPolicy: {{ .Values.keepalived.image.pullPolicy }}
args: # override options in the Dockerfile
- --vrrp
- --log-detail
- --dump-conf
- --use-file={{ .Values.keepalived_config_path }}/{{ .Values.keepalived_config_file }}
- --use-file={{ .Values.keepalived.config_path }}/{{ .Values.keepalived.config_file }}
livenessProbe:
exec:
command: ["pidof", "keepalived"]
Expand All @@ -40,7 +40,7 @@ data:
volumeMounts:
- mountPath: /etc/localtime
name: host-localtime
- mountPath: {{ .Values.keepalived_config_path }}/{{ .Values.keepalived_config_file }}
- mountPath: {{ .Values.keepalived.config_path }}/{{ .Values.keepalived.config_file }}
name: config
hostNetwork: true
priorityClassName: system-node-critical
Expand All @@ -50,5 +50,5 @@ data:
path: /etc/localtime
name: host-localtime
- hostPath:
path: {{ .Values.keepalived_config_path }}/{{ .Values.keepalived_config_file }}
path: {{ .Values.keepalived.config_path }}/{{ .Values.keepalived.config_file }}
name: config
33 changes: 33 additions & 0 deletions charts/kubelived/templates/keepalived-scripts.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: keepalived-scripts
namespace: {{ .Release.Namespace }}
labels:
{{- include "kubelived.labels" . | nindent 4 }}
data:
install.sh: |
#!/bin/sh
if [ ! -e "/etc/kubernetes/manifests/kube-keepalived.yaml" ]; then
echo $(date): "Copying keepalived manifest file to /etc/kubernetes/manifests/kube-keepalived.yaml"
cp /opt/kube-keepalived.yaml /etc/kubernetes/manifests/kube-keepalived.yaml
chmod 600 /etc/kubernetes/manifests/kube-keepalived.yaml
echo $(date): "Copying keepalived config file to {{ .Values.keepalived.config_path }}/{{ .Values.keepalived.config_file }}"
cp /opt/{{ .Values.keepalived.config_file }} {{ .Values.keepalived.config_path }}/{{ .Values.keepalived.config_file }}
echo $(date): "Finding the keepalived interface:" $KEEPALIVED_INTERFACE
KEEPALIVED_INTERFACE={{ .Values.keepalived.interface }}
echo $(date): "Calculating the keepalived priority:" $KEEPALIVED_PRIORITY
KEEPALIVED_PRIORITY=$(ifconfig $KEEPALIVED_INTERFACE | grep 'inet '| awk '{print $2}'| cut -d. -f4)
sed -i "s|KEEPALIVED_INTERFACE|$KEEPALIVED_INTERFACE|g" {{ .Values.keepalived.config_path }}/{{ .Values.keepalived.config_file }}
sed -i "s|KEEPALIVED_PRIORITY|$KEEPALIVED_PRIORITY|g" {{ .Values.keepalived.config_path }}/{{ .Values.keepalived.config_file }}
echo $(date): "Updated the keepalived config file"
else echo $(date): "Already installed keepalived, nothing to do!"
fi
uninstall.sh: |
#!/bin/sh
#
echo $(date): "Removing keepalived manifest file from /etc/kubernetes/manifests/kube-keepalived.yaml"
rm -rf /etc/kubernetes/manifests/kube-keepalived.yaml
echo $(date): "Removing keepalived config file from {{ .Values.keepalived.config_path }}/{{ .Values.keepalived.config_file }}"
rm -rf {{ .Values.keepalived.config_path }}/{{ .Values.keepalived.config_file }}
59 changes: 59 additions & 0 deletions charts/kubelived/templates/podsecuritypolicy.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
kind: PodSecurityPolicy
apiVersion: policy/v1beta1
metadata:
name: {{ include "kubelived.fullname" . }}
labels:
{{- include "kubelived.labels" . | nindent 4 }}
spec:
privileged: true
allowPrivilegeEscalation: true
allowedCapabilities:
- 'NET_ADMIN'
volumes:
- 'configMap'
- 'secret'
- 'hostPath'
hostNetwork: true
runAsUser:
rule: 'RunAsAny'
seLinux:
rule: 'RunAsAny'
supplementalGroups:
rule: 'RunAsAny'
fsGroup:
rule: 'RunAsAny'
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: {{ include "kubelived.fullname" . }}-use-psp
labels:
{{- include "kubelived.labels" . | nindent 4 }}
rules:
- apiGroups:
- extensions
resources:
- podsecuritypolicies
resourceNames:
- {{ include "kubelived.fullname" . }}
verbs:
- use
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: {{ include "kubelived.fullname" . }}-use-psp
labels:
{{- include "kubelived.labels" . | nindent 4 }}
namespace: {{ .Release.Namespace }}
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: {{ include "kubelived.fullname" . }}-use-psp
subjects:
- apiGroup: ""
kind: ServiceAccount
name: {{ include "kubelived.serviceAccountName" . }}
- apiGroup: rbac.authorization.k8s.io
kind: Group
name: system:nodes # required to start mirror pods
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
{{- if .Values.serviceAccount.create -}}
{{- if .Values.installer.serviceAccount.create -}}
apiVersion: v1
kind: ServiceAccount
metadata:
name: {{ include "kubelived.serviceAccountName" . }}
labels:
{{- include "kubelived.labels" . | nindent 4 }}
{{- with .Values.serviceAccount.annotations }}
{{- with .Values.installer.serviceAccount.serviceAccountAnnotations }}
annotations:
{{- toYaml . | nindent 4 }}
{{- end }}
Expand Down
45 changes: 45 additions & 0 deletions charts/kubelived/values.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
# Default values for kubelived.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.

installer:
image:
repository: busybox
pullPolicy: IfNotPresent
tag: latest
serviceAccount:
create: true
serviceAccountAnnotations: {}
name: ''
annotations: {}
podAnnotations: {}
hostNetwork: true
nodeSelector: {}
# node-role.kubernetes.io/master: ""
tolerations: []
# - key: CriticalAddonsOnly
# operator: Exists
# - effect: NoSchedule
# key: node-role.kubernetes.io/master

keepalived:
image:
repository: bsctl/keepalived
pullPolicy: IfNotPresent
tag: '0.2.0'
config_path: '/etc/keepalived'
config_file: 'keepalived.conf'
initial_state: 'backup'
password: 'password'
health_service_name: 'apiserver'
health_service_check: '/usr/bin/curl -s -k https://localhost:6443/healthz -o /dev/null'
health_service_user: 'root'
health_service_interval: '20'
health_service_timeout: '5'
health_service_rise: '1'
health_service_fall: '1'
router_id: '100'
advertisement_interval: '3'
virtual_address: '10.10.10.250'
interface: 'eth0'
unicast_peers: []
Loading

0 comments on commit d721f89

Please sign in to comment.