Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Increase HAProxy check interval to prevent intermittent failures #3356

Merged
merged 2 commits into from
Apr 4, 2020
Merged

fix: Increase HAProxy check interval to prevent intermittent failures #3356

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
5c79f45
Increase HAProxy check interval to prevent intermittent failures/stat…
jannfis Apr 3, 2020
3009531
Restore original namespace
jannfis Apr 3, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 12 additions & 13 deletions manifests/ha/base/redis-ha/chart/upstream.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
# This is an auto-generated file. DO NOT EDIT
---
# Source: redis-ha/charts/redis-ha/templates/redis-ha-configmap.yaml
apiVersion: v1
Expand Down Expand Up @@ -155,9 +154,9 @@ data:
tcp-check expect string REPLACE_ANNOUNCE0
tcp-check send QUIT\r\n
tcp-check expect string +OK
server R0 argocd-redis-ha-announce-0:26379 check inter 1s
server R1 argocd-redis-ha-announce-1:26379 check inter 1s
server R2 argocd-redis-ha-announce-2:26379 check inter 1s
server R0 argocd-redis-ha-announce-0:26379 check inter 3s
server R1 argocd-redis-ha-announce-1:26379 check inter 3s
server R2 argocd-redis-ha-announce-2:26379 check inter 3s
# Check Sentinel and whether they are nominated master
backend check_if_redis_is_master_1
mode tcp
Expand All @@ -169,9 +168,9 @@ data:
tcp-check expect string REPLACE_ANNOUNCE1
tcp-check send QUIT\r\n
tcp-check expect string +OK
server R0 argocd-redis-ha-announce-0:26379 check inter 1s
server R1 argocd-redis-ha-announce-1:26379 check inter 1s
server R2 argocd-redis-ha-announce-2:26379 check inter 1s
server R0 argocd-redis-ha-announce-0:26379 check inter 3s
server R1 argocd-redis-ha-announce-1:26379 check inter 3s
server R2 argocd-redis-ha-announce-2:26379 check inter 3s
# Check Sentinel and whether they are nominated master
backend check_if_redis_is_master_2
mode tcp
Expand All @@ -183,9 +182,9 @@ data:
tcp-check expect string REPLACE_ANNOUNCE2
tcp-check send QUIT\r\n
tcp-check expect string +OK
server R0 argocd-redis-ha-announce-0:26379 check inter 1s
server R1 argocd-redis-ha-announce-1:26379 check inter 1s
server R2 argocd-redis-ha-announce-2:26379 check inter 1s
server R0 argocd-redis-ha-announce-0:26379 check inter 3s
server R1 argocd-redis-ha-announce-1:26379 check inter 3s
server R2 argocd-redis-ha-announce-2:26379 check inter 3s
# decide redis backend to use
#master
Expand All @@ -204,11 +203,11 @@ data:
tcp-check send QUIT\r\n
tcp-check expect string +OK
use-server R0 if { srv_is_up(R0) } { nbsrv(check_if_redis_is_master_0) ge 2 }
server R0 argocd-redis-ha-announce-0:6379 check inter 1s fall 1 rise 1
server R0 argocd-redis-ha-announce-0:6379 check inter 3s fall 1 rise 1
use-server R1 if { srv_is_up(R1) } { nbsrv(check_if_redis_is_master_1) ge 2 }
server R1 argocd-redis-ha-announce-1:6379 check inter 1s fall 1 rise 1
server R1 argocd-redis-ha-announce-1:6379 check inter 3s fall 1 rise 1
use-server R2 if { srv_is_up(R2) } { nbsrv(check_if_redis_is_master_2) ge 2 }
server R2 argocd-redis-ha-announce-2:6379 check inter 1s fall 1 rise 1
server R2 argocd-redis-ha-announce-2:6379 check inter 3s fall 1 rise 1
haproxy_init.sh: |
HAPROXY_CONF=/data/haproxy.cfg
cp /readonly/haproxy.cfg "$HAPROXY_CONF"
Expand Down
4 changes: 3 additions & 1 deletion manifests/ha/base/redis-ha/generate.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,4 +16,6 @@ helm2 template ./chart \
--name argocd \
--values ./chart/values.yaml \
${helm_execute} \
>> ./chart/upstream.yaml
>> ./chart/upstream_orig.yaml

sed -e 's/check inter 1s/check inter 3s/' ./chart/upstream_orig.yaml > ./chart/upstream.yaml && rm ./chart/upstream_orig.yaml
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 that looks good enough to me

24 changes: 12 additions & 12 deletions manifests/ha/install.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2230,9 +2230,9 @@ data:
tcp-check expect string REPLACE_ANNOUNCE0
tcp-check send QUIT\r\n
tcp-check expect string +OK
server R0 argocd-redis-ha-announce-0:26379 check inter 1s
server R1 argocd-redis-ha-announce-1:26379 check inter 1s
server R2 argocd-redis-ha-announce-2:26379 check inter 1s
server R0 argocd-redis-ha-announce-0:26379 check inter 3s
server R1 argocd-redis-ha-announce-1:26379 check inter 3s
server R2 argocd-redis-ha-announce-2:26379 check inter 3s
# Check Sentinel and whether they are nominated master
backend check_if_redis_is_master_1
mode tcp
Expand All @@ -2244,9 +2244,9 @@ data:
tcp-check expect string REPLACE_ANNOUNCE1
tcp-check send QUIT\r\n
tcp-check expect string +OK
server R0 argocd-redis-ha-announce-0:26379 check inter 1s
server R1 argocd-redis-ha-announce-1:26379 check inter 1s
server R2 argocd-redis-ha-announce-2:26379 check inter 1s
server R0 argocd-redis-ha-announce-0:26379 check inter 3s
server R1 argocd-redis-ha-announce-1:26379 check inter 3s
server R2 argocd-redis-ha-announce-2:26379 check inter 3s
# Check Sentinel and whether they are nominated master
backend check_if_redis_is_master_2
mode tcp
Expand All @@ -2258,9 +2258,9 @@ data:
tcp-check expect string REPLACE_ANNOUNCE2
tcp-check send QUIT\r\n
tcp-check expect string +OK
server R0 argocd-redis-ha-announce-0:26379 check inter 1s
server R1 argocd-redis-ha-announce-1:26379 check inter 1s
server R2 argocd-redis-ha-announce-2:26379 check inter 1s
server R0 argocd-redis-ha-announce-0:26379 check inter 3s
server R1 argocd-redis-ha-announce-1:26379 check inter 3s
server R2 argocd-redis-ha-announce-2:26379 check inter 3s

# decide redis backend to use
#master
Expand All @@ -2279,11 +2279,11 @@ data:
tcp-check send QUIT\r\n
tcp-check expect string +OK
use-server R0 if { srv_is_up(R0) } { nbsrv(check_if_redis_is_master_0) ge 2 }
server R0 argocd-redis-ha-announce-0:6379 check inter 1s fall 1 rise 1
server R0 argocd-redis-ha-announce-0:6379 check inter 3s fall 1 rise 1
use-server R1 if { srv_is_up(R1) } { nbsrv(check_if_redis_is_master_1) ge 2 }
server R1 argocd-redis-ha-announce-1:6379 check inter 1s fall 1 rise 1
server R1 argocd-redis-ha-announce-1:6379 check inter 3s fall 1 rise 1
use-server R2 if { srv_is_up(R2) } { nbsrv(check_if_redis_is_master_2) ge 2 }
server R2 argocd-redis-ha-announce-2:6379 check inter 1s fall 1 rise 1
server R2 argocd-redis-ha-announce-2:6379 check inter 3s fall 1 rise 1
haproxy_init.sh: |
HAPROXY_CONF=/data/haproxy.cfg
cp /readonly/haproxy.cfg "$HAPROXY_CONF"
Expand Down
24 changes: 12 additions & 12 deletions manifests/ha/namespace-install.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2145,9 +2145,9 @@ data:
tcp-check expect string REPLACE_ANNOUNCE0
tcp-check send QUIT\r\n
tcp-check expect string +OK
server R0 argocd-redis-ha-announce-0:26379 check inter 1s
server R1 argocd-redis-ha-announce-1:26379 check inter 1s
server R2 argocd-redis-ha-announce-2:26379 check inter 1s
server R0 argocd-redis-ha-announce-0:26379 check inter 3s
server R1 argocd-redis-ha-announce-1:26379 check inter 3s
server R2 argocd-redis-ha-announce-2:26379 check inter 3s
# Check Sentinel and whether they are nominated master
backend check_if_redis_is_master_1
mode tcp
Expand All @@ -2159,9 +2159,9 @@ data:
tcp-check expect string REPLACE_ANNOUNCE1
tcp-check send QUIT\r\n
tcp-check expect string +OK
server R0 argocd-redis-ha-announce-0:26379 check inter 1s
server R1 argocd-redis-ha-announce-1:26379 check inter 1s
server R2 argocd-redis-ha-announce-2:26379 check inter 1s
server R0 argocd-redis-ha-announce-0:26379 check inter 3s
server R1 argocd-redis-ha-announce-1:26379 check inter 3s
server R2 argocd-redis-ha-announce-2:26379 check inter 3s
# Check Sentinel and whether they are nominated master
backend check_if_redis_is_master_2
mode tcp
Expand All @@ -2173,9 +2173,9 @@ data:
tcp-check expect string REPLACE_ANNOUNCE2
tcp-check send QUIT\r\n
tcp-check expect string +OK
server R0 argocd-redis-ha-announce-0:26379 check inter 1s
server R1 argocd-redis-ha-announce-1:26379 check inter 1s
server R2 argocd-redis-ha-announce-2:26379 check inter 1s
server R0 argocd-redis-ha-announce-0:26379 check inter 3s
server R1 argocd-redis-ha-announce-1:26379 check inter 3s
server R2 argocd-redis-ha-announce-2:26379 check inter 3s

# decide redis backend to use
#master
Expand All @@ -2194,11 +2194,11 @@ data:
tcp-check send QUIT\r\n
tcp-check expect string +OK
use-server R0 if { srv_is_up(R0) } { nbsrv(check_if_redis_is_master_0) ge 2 }
server R0 argocd-redis-ha-announce-0:6379 check inter 1s fall 1 rise 1
server R0 argocd-redis-ha-announce-0:6379 check inter 3s fall 1 rise 1
use-server R1 if { srv_is_up(R1) } { nbsrv(check_if_redis_is_master_1) ge 2 }
server R1 argocd-redis-ha-announce-1:6379 check inter 1s fall 1 rise 1
server R1 argocd-redis-ha-announce-1:6379 check inter 3s fall 1 rise 1
use-server R2 if { srv_is_up(R2) } { nbsrv(check_if_redis_is_master_2) ge 2 }
server R2 argocd-redis-ha-announce-2:6379 check inter 1s fall 1 rise 1
server R2 argocd-redis-ha-announce-2:6379 check inter 3s fall 1 rise 1
haproxy_init.sh: |
HAPROXY_CONF=/data/haproxy.cfg
cp /readonly/haproxy.cfg "$HAPROXY_CONF"
Expand Down