Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Issues when ingress-nginx first loads on cluster with a lot (279) of ingresses #3473

Closed
Stono opened this issue Nov 27, 2018 · 1 comment · Fixed by #3474
Closed

Issues when ingress-nginx first loads on cluster with a lot (279) of ingresses #3473

Stono opened this issue Nov 27, 2018 · 1 comment · Fixed by #3474

Comments

@Stono
Copy link
Contributor

Stono commented Nov 27, 2018

8935", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress vds-admin/vds-admin-observability
W1127 22:23:05.144542       1 main.go:157] auth-snippet annotation is undefined and will not be set
I1127 22:23:05.144665       1 event.go:221] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"manufacturer-network-report", Name:"manufacturer-network-report-internal", UID:"1222d3db-c7bc-11e8-a49f-42010aa400f5", APIVersion:"extensions/v1beta1", ResourceVersion:"61813637", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress manufacturer-network-report/manufacturer-network-report-internal
I1127 22:23:05.145550       1 event.go:221] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"part-exchange-api", Name:"part-exchange-api-internal", UID:"5e31f073-e10d-11e8-a96c-42010aa40060", APIVersion:"extensions/v1beta1", ResourceVersion:"61797349", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress part-exchange-api/part-exchange-api-internal
I1127 22:23:05.225746       1 event.go:221] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"recruitment-website", Name:"recruitment-website-internal", UID:"856e2f8c-9c87-11e8-82d6-42010aa401dc", APIVersion:"extensions/v1beta1", ResourceVersion:"62201265", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress recruitment-website/recruitment-website-internal
I1127 22:23:05.227493       1 event.go:221] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"lead-oriented-logging-service", Name:"lead-oriented-logging-service-observability", UID:"9f7bc072-d79c-11e8-a20c-42010aa401a8", APIVersion:"extensions/v1beta1", ResourceVersion:"57037614", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress lead-oriented-logging-service/lead-oriented-logging-service-observability
E1127 22:23:06.840926       1 controller.go:184] Unexpected failure reloading the backend:

-------------------------------------------------------------------------------
Error: exit status 1
2018/11/27 22:23:06 [notice] 57#57: ModSecurity-nginx v1.0.0
2018/11/27 22:23:06 [emerg] 57#57: invalid number of arguments in "proxy_buffering" directive in /tmp/nginx-cfg931362031:489
nginx: [emerg] invalid number of arguments in "proxy_buffering" directive in /tmp/nginx-cfg931362031:489
nginx: configuration file /tmp/nginx-cfg931362031 test failed

-------------------------------------------------------------------------------
W1127 22:23:06.840967       1 queue.go:130] requeuing initial-sync, err
-------------------------------------------------------------------------------
Error: exit status 1
2018/11/27 22:23:06 [notice] 57#57: ModSecurity-nginx v1.0.0
2018/11/27 22:23:06 [emerg] 57#57: invalid number of arguments in "proxy_buffering" directive in /tmp/nginx-cfg931362031:489
nginx: [emerg] invalid number of arguments in "proxy_buffering" directive in /tmp/nginx-cfg931362031:489
nginx: configuration file /tmp/nginx-cfg931362031 test failed

-------------------------------------------------------------------------------

I1127 22:23:12.627043       1 controller.go:190] Backend successfully reloaded.
I1127 22:23:12.628744       1 controller.go:202] Initial sync, sleeping for 1 second.

If I look in the temp file, all the ingresses look like this:


                        proxy_connect_timeout                   0s;
                        proxy_send_timeout                      0s;
                        proxy_read_timeout                      0s;

                        proxy_buffering                         ;
                        proxy_buffer_size                       ;
                        proxy_buffers                           4 ;
                        proxy_request_buffering                 ;

                        proxy_http_version                      1.1;

                        proxy_cookie_domain                     ;
                        proxy_cookie_path                       ;

                        # In case of errors try the next upstream server before returning an error
                        proxy_next_upstream                     ;
                        proxy_next_upstream_tries               0;

                        proxy_pass http://upstream_balancer;

                        proxy_redirect                           ;

This sometimes causes the livenessCheck to fail, and restarts the pod.

However after the last I1127 22:23:07.002101 1 controller.go:172] Configuration changes detected, backend reload required., it does successfully load and populate the main nginx config

I've attached the full log for you whole-log.log

Let me know if there is anything else I can provide.

@Stono
Copy link
Contributor Author

Stono commented Dec 8, 2018

Hey @aledbf - don't suppose you have a build with this fix in it do you?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant