Changes CustomHTTPErrors annotation to use custom default backend #3730

jasongwartz · 2019-02-06T15:21:39Z

What this PR does / why we need it:

This PR makes it possible to use a default-backend annotation together with a custom-http-errors annotation. When the default-backend annotation is set, the upstream of the @custom error locations is set to the upstream name of the default-backend annotation's service.

Cases covered:

single ingress object for a hostname, both default-backend and custom-http-errors annotations set
multiple ingress objects for the same hostname, different paths, each with their own set of default-backend and custom-http-errors annotations (eg. capturing errors to one default-backend for a certain path vs another default-backend, or the global default-backend, for a different path - via separate ingress objects but the same hostname)

This involved making the name of the error-handling location @custom_<upstream-name>_<code>. This is needed for the case where, within one server block, different paths (locations) are capturing the same error code but sending it to different default-backends. This means that the new 'default' error location name is @custom_upstream-default-backend_<code>.

Which issue this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged):

This PR is a follow-up to #3344, specifically addressing comment: #3344 (comment). This also resolves #3700

Special notes:

I've had some trouble running the e2e-tests locally so I might need to revisit those. Thanks for reviewing!

jasongwartz · 2019-02-09T17:42:00Z

internal/ingress/controller/controller.go

@@ -591,11 +591,13 @@ func (n *NGINXController) getBackendServers(ingresses []*ingress.Ingress) ([]*in
 									location.Path, server.Hostname, location.DefaultBackend.Namespace, location.DefaultBackend.Name)

 								nb := upstream.DeepCopy()
-								name := fmt.Sprintf("custom-default-backend-%v", upstream.Name)
+								name := fmt.Sprintf("custom-default-backend-%v", location.DefaultBackend.GetName())


This change means that the backend name of custom-error locations is custom-default-backend-<service_name> where service_name is the name of the default-backend service. It used to be that the service_name would be the name of the upstream for which this default-backend is substituted. Eg. where it used to be custom-default-backend-my-app, it is now custom-default-backend-app-errorhandler.

This is necessary since multiple locations within a server block can share the same 'default-backend' - so now, the name of the error-handler-location clearly signifies which service it will route to.

Is this a necessary change? I wonder what was the rationale to design that way initially? Could it be maybe because it would explicitly mention what upstream it is replacing due to the fact that upstream has no endpoints?

well it's necessary if multiple ingress objects (ie locations) for a single upstream can have different error blocks. if I don't make this change, all locations that point to a given upstream have to share the same default-backend.

I assume it was that way initially since the default-backend was only for when the upstream had no endpoints, thus this upstream was replacing the regular upstream entirely (whereas now, if endpoints > 0, the default backend is just another upstream).

Given two ingresses one with location /one pointing to upstream X and another with location /two pointing to upstream X as well. And upstream X has no endpoints. Then without this change we can not use different default-backend annotation per location, right?

Given that this change is not necessarily about custom errors, do you think we should introduce it in a different PR with its own tests and documentation adjustments?

the case I was more targeting is two ingresses /one and /two, same upstream, both with custom error annotations but different default backends - but yeah, that case you mentioned is also true.

It might be tricky to pull it out into another PR - I'll look and see how tightly I integrated the rest. I don't think it will have any user-facing change, but we can definitely add tests and update the docs (https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#default-backend is out-of-date anyway)

jasongwartz · 2019-02-09T17:42:59Z

internal/ingress/controller/template/template.go

 	return struct {
-		ProxySetHeaders map[string]string


As far as I could see, the ProxySetHeaders map was unused in the template. If this is a bug, we can re-add it and fix.

jasongwartz · 2019-02-09T17:54:03Z

/assign bowei

aledbf · 2019-02-09T18:12:58Z

@jasongwartz please don't assign PRs manually

jasongwartz · 2019-02-09T18:18:15Z

@aledbf sorry, my bad. can we remove that line from the k8s-ci-robot message somehow?

aledbf · 2019-02-09T21:46:23Z

can we remove that line from the k8s-ci-robot message somehow?

No, because the bot reads the owners files and tries to suggest a name 😞

aledbf · 2019-02-09T21:46:35Z

@jasongwartz please squash the commits

jasongwartz · 2019-02-11T00:06:16Z

@aledbf cool, have done so

aledbf · 2019-02-12T13:50:49Z

/approve

aledbf · 2019-02-12T13:51:28Z

@ElvinEfendi please review (and lgtm)

ElvinEfendi · 2019-02-14T03:21:40Z

internal/ingress/controller/controller.go

@@ -580,7 +580,7 @@ func (n *NGINXController) getBackendServers(ingresses []*ingress.Ingress) ([]*in
 		for _, server := range servers {
 			for _, location := range server.Locations {
 				if upstream.Name == location.Backend {
-					if len(upstream.Endpoints) == 0 {
+					if len(upstream.Endpoints) == 0 || len(location.CustomHTTPErrors) != 0 {
 						klog.V(3).Infof("Upstream %q has no active Endpoint", upstream.Name)


I think this need to be replaced - it is not going to be true when len(upstream.Endpoints) != 0 and len(location.CustomHTTPErrors) != 0

it should be true in that case, right? maybe I'm not following, but if you mean false || true it should eval to true

ElvinEfendi · 2019-02-14T03:22:27Z

internal/ingress/controller/controller.go

@@ -591,11 +591,13 @@ func (n *NGINXController) getBackendServers(ingresses []*ingress.Ingress) ([]*in
 									location.Path, server.Hostname, location.DefaultBackend.Namespace, location.DefaultBackend.Name)


similar to https://github.com/kubernetes/ingress-nginx/pull/3730/files#r256678420, this log message can be lie, maybe move it to https://github.com/kubernetes/ingress-nginx/pull/3730/files#diff-86ac9ff9d75a0c5005c116e766a6127dR598?

done as well, good catch - I had missed that entirely

ElvinEfendi · 2019-02-14T04:58:23Z

internal/ingress/controller/controller.go

@@ -580,7 +580,7 @@ func (n *NGINXController) getBackendServers(ingresses []*ingress.Ingress) ([]*in
 		for _, server := range servers {
 			for _, location := range server.Locations {
 				if upstream.Name == location.Backend {
-					if len(upstream.Endpoints) == 0 {
+					if len(upstream.Endpoints) == 0 || len(location.CustomHTTPErrors) != 0 {
 						klog.V(3).Infof("Upstream %q has no active Endpoint", upstream.Name)
 						// check if the location contains endpoints and a custom default backend
 						if location.DefaultBackend != nil {


What do you think of merging everything until https://github.com/kubernetes/ingress-nginx/pull/3730/files#diff-86ac9ff9d75a0c5005c116e766a6127dR590 and extracting them all together into a helper function something called shouldCreateUpstreamForLocationDefaultBackend?

done, let me know if that's what you had in mind

ElvinEfendi · 2019-02-14T05:04:58Z

internal/ingress/controller/template/template.go

+	if location.DefaultBackend != nil {
+		return fmt.Sprintf("custom-default-backend-%v", location.DefaultBackend.GetName())
+	}
+	return "upstream-default-backend"


I think these values are used in controller.go as well, how hard would it be to put them into a helper function and constant respectively and reuse here?

I'd be happy to do that. package controller has a utils file already, but importing the controller package here doesn't quite seem right - is there somewhere with shared utils (across both controller and template)?

Maybe we can make this function a property of Location struct?

I don't mind leaving this as is for now.

I tried an implementation of making it a property of Location struct, let me know if it's what you had in mind?

internal/ingress/controller/template/template.go

ElvinEfendi · 2019-02-14T05:11:56Z

internal/ingress/controller/template/template.go

+
+	for upstream := range codesMap {
+		codesForUpstream := []int{}
+		for code := range codesMap[upstream] {


Could you rewrite it as

for code, _ := range codesMap[upstream] {

otherwise it's really confusing to read

yup, will do

have done so now, but gofmt throws a warning 'should omit 2nd value'

ElvinEfendi · 2019-02-14T05:13:08Z

internal/ingress/controller/template/template.go

-// and returns a unique set (for the template to create @custom_xxx locations)
-func collectCustomErrorsPerServer(input interface{}) []int {
+// and returns a set which is unique by default-upstream and error code. The resulting
+// data structure is a map of upstreamName => []errorCodes, so that sufficiently unique


If I read the code correctly this is an intermediary data structure you're using not what the function returns.

internal/ingress/controller/template/template.go

jasongwartz · 2019-02-23T20:35:47Z

@aledbf thanks for the reminder! I addressed the comments and am still working on the e2e tests (I haven't had a lot of time to dive into this but I'll keep at it)

ElvinEfendi · 2019-02-24T17:00:13Z

@jasongwartz just comparing DefaultBackend when diffing two locations should fix the test issue.

ElvinEfendi · 2019-02-24T17:07:56Z

I believe following patch should fix the issue:

diff --git a/internal/ingress/types_equals.go b/internal/ingress/types_equals.go
index ff6361faa..0dd6aafdd 100644
--- a/internal/ingress/types_equals.go
+++ b/internal/ingress/types_equals.go
@@ -479,6 +479,10 @@ func (l1 *Location) Equal(l2 *Location) bool {
                return false
        }

+       if l1.DefaultBackend.GetName() != l2.DefaultBackend.GetName() {
+               return false
+       }
+
        return true
 }

The reason why I'm not comparing the whole DefaultBackend is to avoid unnecessary Nginx reloads when something random other than name in DefaultBackend service changes - because we need a new Nginx configuration only when name changes.

jasongwartz · 2019-02-24T20:15:22Z

@ElvinEfendi I should be able to use the new .ocation.DefaultBackendUpstreamName for this as well, right? Otherwise I have to do nil-checks before doing location.DefaultBackend.GetName()?
I've been testing it locally, and the e2e test still fails, but I'll keep digging

jasongwartz · 2019-02-24T21:38:55Z

I'm thinking the failure of the e2e test is because the service of the custom-default-backend doesn't really exist and therefore has no endpoints (so the upstream doesn't get created). I'll try adding a deployment + service and see if that works

Updates e2e test Removes focus from e2e test Fixes renamed function Adds tests for new template funcs Addresses gofmt Updates e2e test, fixes custom-default-backend test by creating service Updates docs

jasongwartz · 2019-02-25T06:13:31Z

Looks like that was it. @aledbf @ElvinEfendi I think this is ready for review again

ElvinEfendi · 2019-02-25T14:45:26Z

internal/ingress/types_equals.go

@@ -479,6 +479,10 @@ func (l1 *Location) Equal(l2 *Location) bool {
 		return false
 	}

+	if l1.DefaultBackendUpstreamName != l2.DefaultBackendUpstreamName {
+		return false
+	}


Was this the only change that fixed e2e test failure?

This, and adding an actual deployment so that endpoints will exist (here: https://github.com/kubernetes/ingress-nginx/pull/3730/files#diff-e82a2942a511f5a12803acc64b3fc676R107)

ElvinEfendi · 2019-02-25T14:45:37Z

test/e2e/annotations/customhttperrors.go

 	extensions "k8s.io/api/extensions/v1beta1"

 	"k8s.io/ingress-nginx/test/e2e/framework"
 )

+const defaultBackendName = "upstream-default-backend"


this is not being used

I'll put up a quick follow-up to remove this, wasn't quick enough before merging. thanks for catching it

ElvinEfendi · 2019-02-25T14:49:37Z

/lgtm

I'll defer approve to @aledbf

Thank you @jasongwartz !

k8s-ci-robot · 2019-02-25T14:49:48Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: aledbf, ElvinEfendi, jasongwartz

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [ElvinEfendi,aledbf]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ElvinEfendi · 2019-02-25T14:50:30Z

/hold

oops the PR already has approved label

aledbf · 2019-02-25T15:19:40Z

/hold cancel

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Feb 6, 2019

k8s-ci-robot requested review from aledbf and ElvinEfendi February 6, 2019 15:22

jasongwartz commented Feb 9, 2019

View reviewed changes

k8s-ci-robot assigned bowei Feb 9, 2019

aledbf unassigned bowei Feb 9, 2019

jasongwartz force-pushed the jg-customerrors-defaultbackend branch from 8bb1700 to 22a89cf Compare February 10, 2019 23:22

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Feb 12, 2019

aledbf mentioned this pull request Feb 12, 2019

Unable to get custom errors working #3759

Closed

ElvinEfendi reviewed Feb 14, 2019

View reviewed changes

internal/ingress/controller/template/template.go Outdated Show resolved Hide resolved

ElvinEfendi reviewed Feb 14, 2019

View reviewed changes

internal/ingress/controller/template/template.go Outdated Show resolved Hide resolved

jasongwartz force-pushed the jg-customerrors-defaultbackend branch 3 times, most recently from 41ef917 to 3dedcd8 Compare February 14, 2019 21:19

jasongwartz force-pushed the jg-customerrors-defaultbackend branch 4 times, most recently from 930a557 to 19ca056 Compare February 24, 2019 06:20

jasongwartz force-pushed the jg-customerrors-defaultbackend branch 2 times, most recently from 55d41c2 to 8693fe8 Compare February 24, 2019 19:34

jasongwartz force-pushed the jg-customerrors-defaultbackend branch 2 times, most recently from eeb54be to cdd1245 Compare February 24, 2019 20:18

Changes CustomHTTPErrors annotation to use custom default backend

3865e30

Updates e2e test Removes focus from e2e test Fixes renamed function Adds tests for new template funcs Addresses gofmt Updates e2e test, fixes custom-default-backend test by creating service Updates docs

jasongwartz force-pushed the jg-customerrors-defaultbackend branch from cdd1245 to 3865e30 Compare February 24, 2019 21:49

ElvinEfendi reviewed Feb 25, 2019

View reviewed changes

k8s-ci-robot assigned ElvinEfendi Feb 25, 2019

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Feb 25, 2019

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Feb 25, 2019

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Feb 25, 2019

k8s-ci-robot merged commit d8ec96c into kubernetes:master Feb 25, 2019

jasongwartz mentioned this pull request Feb 25, 2019

Removes unused const from customhttperrors e2e test #3812

Merged

Stono mentioned this pull request Mar 9, 2019

Support default backends in other namespaces #3878

Closed

haoqing0110 mentioned this pull request Aug 30, 2019

SSL passthrough not working well on 0.23.0 #4512

Closed

		@@ -591,11 +591,13 @@ func (n NGINXController) getBackendServers(ingresses []ingress.Ingress) ([]*in
		location.Path, server.Hostname, location.DefaultBackend.Namespace, location.DefaultBackend.Name)

Changes CustomHTTPErrors annotation to use custom default backend #3730

Changes CustomHTTPErrors annotation to use custom default backend #3730

Conversation

jasongwartz commented Feb 6, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jasongwartz commented Feb 9, 2019

aledbf commented Feb 9, 2019

jasongwartz commented Feb 9, 2019

aledbf commented Feb 9, 2019

aledbf commented Feb 9, 2019

jasongwartz commented Feb 11, 2019

aledbf commented Feb 12, 2019

aledbf commented Feb 12, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jasongwartz commented Feb 23, 2019

ElvinEfendi commented Feb 24, 2019

ElvinEfendi commented Feb 24, 2019

jasongwartz commented Feb 24, 2019

jasongwartz commented Feb 24, 2019

jasongwartz commented Feb 25, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ElvinEfendi commented Feb 25, 2019

k8s-ci-robot commented Feb 25, 2019

ElvinEfendi commented Feb 25, 2019

aledbf commented Feb 25, 2019