Skip to content

Commit

Permalink
Adjust health check to return 503 if all backends are down
Browse files Browse the repository at this point in the history
If all backends are down, the haproxy instance should not be served
anymore by kubernetes and the readiness probe should fail.
Furthermore, we also fail the liveness probe to enforce a restart of
haproxy, as this can resolve the issue in some cases (eg. network or dns
issues).

Signed-off-by: Nicolas Bigler <nicolas.bigler@vshn.ch>
  • Loading branch information
TheBigLee committed Oct 26, 2023
1 parent b7e0eae commit 05120bc
Show file tree
Hide file tree
Showing 7 changed files with 229 additions and 4 deletions.
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ helm repo add appuio https://charts.appuio.ch
| [![chart downloads](https://img.shields.io/github/downloads/appuio/charts/data-cube-curation-0.3.1/total)](https://github.com/appuio/charts/releases/tag/data-cube-curation-0.3.1) | [data-cube-curation](appuio/data-cube-curation/README.md) |
| [![chart downloads](https://img.shields.io/github/downloads/appuio/charts/exoip-1.0.4/total)](https://github.com/appuio/charts/releases/tag/exoip-1.0.4) | [exoip](appuio/exoip/README.md) |
| [![chart downloads](https://img.shields.io/github/downloads/appuio/charts/generic-0.1.2/total)](https://github.com/appuio/charts/releases/tag/generic-0.1.2) | [generic](appuio/generic/README.md) |
| [![chart downloads](https://img.shields.io/github/downloads/appuio/charts/haproxy-2.4.0/total)](https://github.com/appuio/charts/releases/tag/haproxy-2.4.0) | [haproxy](appuio/haproxy/README.md) |
| [![chart downloads](https://img.shields.io/github/downloads/appuio/charts/haproxy-2.5.0/total)](https://github.com/appuio/charts/releases/tag/haproxy-2.5.0) | [haproxy](appuio/haproxy/README.md) |
| [![chart downloads](https://img.shields.io/github/downloads/appuio/charts/mariadb-galera-1.2.4/total)](https://github.com/appuio/charts/releases/tag/mariadb-galera-1.2.4) | [mariadb-galera](appuio/mariadb-galera/README.md) |
| [![chart downloads](https://img.shields.io/github/downloads/appuio/charts/maxscale-2.0.1/total)](https://github.com/appuio/charts/releases/tag/maxscale-2.0.1) | [maxscale](appuio/maxscale/README.md) |
| [![chart downloads](https://img.shields.io/github/downloads/appuio/charts/metrics-server-2.12.1/total)](https://github.com/appuio/charts/releases/tag/metrics-server-2.12.1) | [metrics-server](appuio/metrics-server/README.md) |
Expand Down
2 changes: 1 addition & 1 deletion appuio/haproxy/Chart.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ apiVersion: v1
appVersion: 2.7.3
description: A Helm chart for HAProxy which can be customized by a config map.
name: haproxy
version: 2.4.0
version: 2.5.0
maintainers:
- name: APPUiO Team
email: info@appuio.ch
2 changes: 1 addition & 1 deletion appuio/haproxy/README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# haproxy

![Version: 2.4.0](https://img.shields.io/badge/Version-2.4.0-informational?style=flat-square) ![AppVersion: 2.7.3](https://img.shields.io/badge/AppVersion-2.7.3-informational?style=flat-square)
![Version: 2.5.0](https://img.shields.io/badge/Version-2.5.0-informational?style=flat-square) ![AppVersion: 2.7.3](https://img.shields.io/badge/AppVersion-2.7.3-informational?style=flat-square)

A Helm chart for HAProxy which can be customized by a config map.

Expand Down
Binary file added appuio/haproxy/haproxy-2.4.0.tgz
Binary file not shown.
70 changes: 70 additions & 0 deletions appuio/haproxy/spks.values
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels:
app.kubernetes.io/instance: '{{ .Release.Name }}'
app.kubernetes.io/name: '{{ .Chart.Name }}'
topologyKey: failure-domain.beta.kubernetes.io/zone
fullnameOverride: haproxy
haproxy:
config: ldap-tls
filterproxy:
enabled: true
url: exporter-filterproxy.syn-exporter-filterproxy.svc.cluster.local:8080
frontendPort: 3306
galera:
balance: first
check:
mysql:
enabled: false
metrics:
enabled: true
exposeLoadbalancer: false
nodes:
- address: mariadb-0.mariadb.test-mariadb-crossplane-update.svc.cluster.local
backup: false
port: 3306
- address: mariadb-1.mariadb.test-mariadb-crossplane-update.svc.cluster.local
backup: true
port: 3306
- address: mariadb-2.mariadb.test-mariadb-crossplane-update.svc.cluster.local
backup: true
port: 3306
galerak8s:
metrics:
exposeLoadbalancer: false
redisk8s:
metrics:
exposeLoadbalancer: false
securityContext:
runAsUser: 1000
image:
registry: ghcr-docker-remote.artifactory.swisscom.com
repository: vshn/haproxy-with-mysql
tag: 1.0.0
metrics:
serviceMonitor:
enabled: true
interval: 10s
metricRelabelings:
- action: drop
regex: haproxy_server_check_status
sourceLabels:
- __name__
relabelings:
- replacement: standard
sourceLabels:
- __address__
targetLabel: service_level
replicaCount: 2
resources:
limits:
cpu: 1000m
memory: 1024Mi
requests:
cpu: 10m
memory: 300Mi
service:
port: 3306
type: LoadBalancer
17 changes: 16 additions & 1 deletion appuio/haproxy/templates/_helpers.tpl
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,20 @@ Create chart name and version as used by the chart label.
{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" -}}
{{- end -}}

{{/*
Create backendName depending on the selected configuration
*/}}
{{- define "haproxy.backendName" -}}
{{- if eq .Values.haproxy.config "redisk8s" -}}
redis-nodes
{{- else if or (eq .Values.haproxy.config "galerak8s") (eq .Values.haproxy.config "galera") -}}
galera-nodes
{{- else -}}
backend
{{- end -}}
{{- end -}}

{{/*
{{/*
HAProxy config for stats and metrics
*/}}
Expand All @@ -43,8 +57,9 @@ frontend stats
stats uri /stats
stats refresh 3s
acl server_stopping stopping
acl service_down nbsrv({{ include "haproxy.backendName" . }}) eq 0
monitor-uri /healthz
monitor fail if server_stopping
monitor fail if server_stopping || service_down
option dontlog-normal
option httplog
http-request use-service prometheus-exporter if { path /metrics }
Expand Down
140 changes: 140 additions & 0 deletions appuio/mariadb-galera/params.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,140 @@
USER-SUPPLIED VALUES:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels:
app.kubernetes.io/instance: '{{ .Release.Name }}'
app.kubernetes.io/name: '{{ .Chart.Name }}'
topologyKey: failure-domain.beta.kubernetes.io/zone
extraEnvVars:
- name: MARIADB_GALERA_FORCE_SAFETOBOOTSTRAP
value: "no"
extraFlags: --innodb-buffer-pool-size=300M --key-buffer-size=64K --max-allowed-packet=128M
--max-heap-table-size=1M
forceUpdate: true
forceUpdateKubectlImage: quay-remote.artifactory.swisscom.com/bitnami/kubectl:1.21.9
fullnameOverride: mariadb
galera:
mariabackup:
forcePassword: true
password: 0yRcFQPTLLLU5JZilcNMkXJsFH048ZCL
global:
imageRegistry: remote-docker.artifactory.swisscom.com
image:
debug: true
pullPolicy: Always
tag: 10.5.13-debian-10-r82
livenessProbe:
initialDelaySeconds: 10
mariadbConfiguration: |
[client]
socket=/opt/bitnami/mariadb/tmp/mysql.sock
plugin_dir=/opt/bitnami/mariadb/plugin
[mysqld]
basedir=/opt/bitnami/mariadb
datadir=/bitnami/mariadb/data
plugin_dir=/opt/bitnami/mariadb/plugin
tmpdir=/opt/bitnami/mariadb/tmp
socket=/opt/bitnami/mariadb/tmp/mysql.sock
pid_file=/opt/bitnami/mariadb/tmp/mysqld.pid
ignore_db_dirs=lost+found
## Character set
collation_server=utf8_unicode_ci
init_connect='SET NAMES utf8'
character_set_server=utf8
## Safety
skip_host_cache
skip_name_resolve=ON
max_allowed_packet=512M
max_connect_errors=1000000
sysdate_is_now=ON
binlog_format=row
log_bin_trust_function_creators=ON
## Caches and Limits
query_cache_type=OFF
query_cache_size=0
query_cache_limit=0
max_connections=500
open_files_limit=65535
tmp_table_size=512M
table_definition_cache=1000
extra_port=33306
extra_max_connections=10
thread_handling=pool-of-threads
thread_pool_size=4
## InnoDB
innodb_autoinc_lock_mode=2
## Logging
log_error=/opt/bitnami/mariadb/logs/mysqld.log
slow_query_log=ON
slow_query_log_file=/opt/bitnami/mariadb/logs/mysqld.log
log_queries_not_using_indexes=OFF
log_slow_admin_statements=ON
min_examined_row_limit=2
plugin_load_add=query_response_time
query_response_time_stats=ON
[galera]
wsrep_on=ON
wsrep_provider=/opt/bitnami/mariadb/lib/libgalera_smm.so
wsrep_sst_method=mariabackup
wsrep_cluster_address=gcomm://
wsrep_cluster_name=galera
wsrep_sst_auth="root:"
wsrep_log_conflicts=ON
wsrep_strict_ddl=ON
wsrep_provider_options=cert.log_conflicts=yes;gcs.fc_master_slave=yes;repl.max_ws_size=134217728;gcache.size=512M;gcs.fc_limit=8
metrics:
enabled: true
extraFlags:
- --exporter.log_slow_filter
- --collect.engine_innodb_status
image:
tag: 0.13.0-debian-10-r239
serviceMonitor:
enabled: true
interval: 10s
metricRelabelings:
- action: drop
regex: mysql_global_status_commands_total|mysql_global_status_performance_schema_lost_total|mysql_global_status_handlers_total|mysql_info_schema_query_response_time_seconds_bucket|mysql_exporter_collector_duration_seconds|mysql_global_status_connection_errors_total
sourceLabels:
- __name__
relabelings:
- replacement: premium
sourceLabels:
- __address__
targetLabel: service_level
scrapeTimeout: 3s
persistence:
enabled: true
size: 10Gi
podAnnotations:
mariadb-config-revision: "2"
podDisruptionBudget:
create: true
maxUnavailable: 1
minAvailable: ""
resources:
limits:
cpu: 1000m
memory: 2Gi
requests:
cpu: 100m
memory: 100Mi
rootUser:
forcePassword: true
password: gZ0Piyj0YK/nYgM1c5q3i2LYJrJ4Rzj9
startupProbe:
enabled: true
failureThreshold: 108
initialDelaySeconds: 10

0 comments on commit 05120bc

Please sign in to comment.