Haproxy-ingress with AWS NLB

[sharma-raj]

How we can configure haproxy-ingress with AWS NLB. I tried many times but unable to get lucky. Getting below error:

cp: can't preserve ownership of '/etc/haproxy/lua/auth-request.lua': Operation not permitted
cp: can't preserve ownership of '/etc/haproxy/lua/send-response.lua': Operation not permitted
cp: can't preserve ownership of '/etc/haproxy/lua/services.lua': Operation not permitted
cp: can't preserve ownership of '/etc/haproxy/lua': Operation not permitted
I1229 13:50:57.608421 6 launch.go:203]
Name: HAProxy
Release: v0.11.1

[jcmoraisjr]

Hi, it seems the container file system has some restriction. Please edit the issue and add a few more details on how you've provisioned the controller.

[sharma-raj]

Hi, Below are the details:

kubectl get po -A
NAMESPACE              NAME                              READY   STATUS    RESTARTS   AGE
ingress-architecture   haproxy-ingress-f77bf744c-r4qqp   1/1     Running   0          20h
ingress-architecture   nginx-app-f9fbd4748-6sbk4         1/1     Running   0          20h

kubectl logs haproxy-ingress-f77bf744c-r4qqp -n ingress-architecture
cp: can't preserve ownership of '/etc/haproxy/lua/auth-request.lua': Operation not permitted
cp: can't preserve ownership of '/etc/haproxy/lua/send-response.lua': Operation not permitted
cp: can't preserve ownership of '/etc/haproxy/lua/services.lua': Operation not permitted
cp: can't preserve ownership of '/etc/haproxy/lua': Operation not permitted
I0104 09:05:48.174443       7 launch.go:203]
Name:       HAProxy
Release:    v0.11.1
Build:      git-07ffc15
Repository: https://github.com/jcmoraisjr/haproxy-ingress
I0104 09:05:48.174586       7 launch.go:206] Watching for ingress class: haproxy
I0104 09:05:48.174816       7 launch.go:462] Creating API client for https://172.20.0.1:443
I0104 09:05:48.182646       7 launch.go:474] Running in Kubernetes Cluster version v1.18+ (v1.18.9-eks-d1db3c) - git (clean) commit d1db3c46e55f95d6a7d3e5578689371318f95ff9 - platform linux/amd64
I0104 09:05:48.186235       7 launch.go:229] validated ingress-architecture/ingress-default-backend as the default backend
I0104 09:05:48.589195       7 listers.go:134] loading object cache...
I0104 09:05:48.689360       7 listers.go:145] cache successfully synced
I0104 09:05:48.689416       7 controller.go:87] HAProxy Ingress successfully initialized
I0104 09:05:48.689507       7 leaderelection.go:242] attempting to acquire leader lease  ingress-architecture/ingress-controller-leader-haproxy...
I0104 09:05:48.694167       7 status.go:177] new leader elected: haproxy-ingress-f77bf744c-zxkgn
I0104 09:05:48.797783       7 controller.go:316] starting HAProxy update id=1
W0104 09:05:48.797909       7 ingress.go:132] using auto generated fake certificate due to an error reading default TLS certificate: secret "tls-secret" not found
I0104 09:05:48.797924       7 ingress.go:145] using auto generated fake certificate
W0104 09:05:48.798017       7 ingress.go:390] skipping backend config of ingress 'ingress-architecture/ingress-architecture': port not found: '8080'
W0104 09:05:48.812704       7 instance.go:439] output from haproxy:
[ALERT] 003/090548 (24) : Starting frontend _front_http: cannot bind socket [0.0.0.0:80]
[ALERT] 003/090548 (24) : Starting frontend _front_https: cannot bind socket [0.0.0.0:443]
E0104 09:05:48.812729       7 instance.go:301] error reloading server:
exit status 1
I0104 09:05:48.812816       7 controller.go:348] finish HAProxy update id=1: parse_ingress=0.306554ms write_maps=0.114789ms write_config=0.953661ms total=1.375004ms
I0104 09:05:57.519064       7 controller.go:316] starting HAProxy update id=2
W0104 09:05:57.519136       7 ingress.go:132] using auto generated fake certificate due to an error reading default TLS certificate: secret "tls-secret" not found
I0104 09:05:57.519213       7 instance.go:293] old and new configurations match
I0104 09:05:57.519259       7 controller.go:348] finish HAProxy update id=2: parse_ingress=0.070924ms write_maps=0.008880ms total=0.079804ms
I0104 09:06:28.345120       7 leaderelection.go:252] successfully acquired lease ingress-architecture/ingress-controller-leader-haproxy
I0104 09:06:28.345203       7 status.go:177] new leader elected: haproxy-ingress-f77bf744c-r4qqp
I0104 09:13:42.400861       7 controller.go:316] starting HAProxy update id=3
W0104 09:13:42.401053       7 ingress.go:132] using auto generated fake certificate due to an error reading default TLS certificate: secret "tls-secret" not found
W0104 09:13:42.401194       7 ingress.go:390] skipping backend config of ingress 'ingress-architecture/ingress-architecture': service "ingress-architecture" not found
I0104 09:13:42.401455       7 instance.go:293] old and new configurations match
I0104 09:13:42.401498       7 controller.go:348] finish HAProxy update id=3: parse_ingress=0.417930ms write_maps=0.025425ms total=0.443355ms
I0104 09:13:48.858711       7 controller.go:316] starting HAProxy update id=4
W0104 09:13:48.858880       7 ingress.go:132] using auto generated fake certificate due to an error reading default TLS certificate: secret "tls-secret" not found
W0104 09:13:48.859010       7 ingress.go:390] skipping backend config of ingress 'ingress-architecture/ingress-architecture': port not found: '8080'
I0104 09:13:48.859171       7 instance.go:293] old and new configurations match
I0104 09:13:48.859216       7 controller.go:348] finish HAProxy update id=4: parse_ingress=0.343149ms write_maps=0.027757ms total=0.370906ms
I0104 09:13:51.019454       7 controller.go:316] starting HAProxy update id=5
W0104 09:13:51.019524       7 ingress.go:132] using auto generated fake certificate due to an error reading default TLS certificate: secret "tls-secret" not found
W0104 09:13:51.019620       7 ingress.go:390] skipping backend config of ingress 'ingress-architecture/ingress-architecture': port not found: '8080'
I0104 09:13:51.019794       7 instance.go:293] old and new configurations match
I0104 09:13:51.019841       7 controller.go:348] finish HAProxy update id=5: parse_ingress=0.201487ms write_maps=0.029669ms total=0.231156ms
I0104 09:19:07.410580       7 event.go:278] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"ingress-architecture", Name:"ingress-architecture", UID:"2ebf893f-d180-4b26-8db8-b418a1e6e048", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"5747083", FieldPath:""}): type: 'Normal' reason: 'DELETE' Ingress ingress-architecture/ingress-architecture
I0104 09:19:07.610426       7 controller.go:316] starting HAProxy update id=6

Please let me know you need more details

[jcmoraisjr]

Please let me know you need more details

Yes, how (steps) and where (kind, k3d, gke, eks, ?) you provisioned HAProxy Ingress?

[jcmoraisjr]

I can reproduce the following error running HAProxy Ingress as an uid != 0 and also != 1001

cp: can't create directory '/etc/haproxy/lua': Permission denied

However I couldn't reproduce a can't preserve ownership, but I'm running an on premise deployment instead of aws.

Please share a bit more information about the restrictions of the environment, eg uid running the controller or any other restriction to the file system. In the mean time I'll remove -p from the copy, this will not preserve timestamp but it's not important after all. Btw it'd be very helpful if you can generate a new image that changes the start.sh script as below and confirm that this fixed the problem. This can be made in a new image, no need to recompile the controller.

diff --git a/rootfs/start.sh b/rootfs/start.sh
index 8112ad0c..ff33bee7 100755
--- a/rootfs/start.sh
+++ b/rootfs/start.sh
@@ -21,6 +21,6 @@ if [ $# -gt 0 ] && [ "$(echo $1 | cut -b1-2)" != "--" ]; then
     exec "$@"
 else
     # Copy static files to /etc/haproxy, which cannot have static content
-    cp -R -p /etc/lua /etc/haproxy/
+    cp -R /etc/lua /etc/haproxy/
     exec /haproxy-ingress-controller "$@"
 fi

[sharma-raj]

I have changed some configMap after following
https://github.com/sharma-rajendra/aws-eks

Thanks for your help :)