[GLBC] Support backside re-encryption #519

nicksardo · 2017-03-29T18:40:09Z

Enables proxying traffic to HTTPS k8s services.

Fixes #494 - Support HTTPS backend
Fixes #475 - RPS per instance instead of instance group

Requires:

[GCE] Support legacy-https and generic health checks kubernetes#43644
update godeps in ingress repo (compute & k8s)

Surfacing Service Annotation

Ideally, the service annotation parsing function is moved here: https://github.com/kubernetes/kubernetes/blob/master/pkg/api/service/annotations.go . However, the ingress repo consumes the client-go api and k8s still consumes the local api. @.caesarxuchao mentions that all types will be moved out of k8s and client-go (kubernetes/kubernetes#44065) in a few weeks. He suggests we place the annotation function locally until things change.

Overview of Changes

Added ability for controller to create compute.HttpsHealthChecks
Abstract HTTP/HTTPSHealthCheck with a struct called HealthCheck
Modified healthchecks package and interface to separate the instantiation of a health check (New...) from creating it in the cloud (`Add...).
Moved the GCETranslator presence from healthchecks to backends
Addition/cleanup of fakes and tests for healthchecks
Plumbing of tuple (nodeport int64, encrypted bool) instead of nodeport int64
Modified backends to create HTTPS compute.BackendServices if encrypted=true

Test Cases

"Perfect setup": Create ingress with default backend pointing to a service (port 443), service has port https=443, service annotation has ["https"]
"Update ingress port": Create ingress pointing to service's port 80. Service already has port https=443 and annotation. Update ingress to point to 443
"Update service annotation":Create ingress pointing to service's port 443. Service already has port https=443 but NO annotation. Set annotation to ["https"]

k8s-reviewable · 2017-03-29T18:40:15Z

This change is

coveralls · 2017-03-29T20:11:06Z

Changes Unknown when pulling d7eaa26f0cb515d20f8357a61b4076d5611776d4 on nicksardo:backside-reencrypt into ** on kubernetes:master**.

aledbf · 2017-03-29T20:19:12Z

update godeps in ingress repo (compute & k8s)

I am doing this in #463 and also migrating to client-go.

nicksardo · 2017-03-31T16:11:47Z

kubernetes/kubernetes#43644 was merged after 1.6.0 release. After talking with some folks here, it may be possible to cherry pick into 1.6.1.

coveralls · 2017-04-05T22:32:25Z

Coverage increased (+0.5%) to 44.696% when pulling 13b2d7b7b1b97a1e8b7436b0fd65c0b68a8bf2ae on nicksardo:backside-reencrypt into ed6987e on kubernetes:master.

aledbf · 2017-04-05T22:36:52Z

@nicksardo why not adding this as a field in Ingress?
(this is already present as an annotation secureupstream in nginx)

nicksardo · 2017-04-05T22:48:52Z

@aledbf Makes more sense as a descriptor for the Service object. Probably not a common case, but there may be users with multiple ingress objs referencing a single service. It also touches the subject of service discovery which was a proposal from long ago. Entities other than ingress controllers may want descriptive information about services. kubernetes/kubernetes#41678

aledbf · 2017-04-05T22:56:35Z

Makes more sense as a descriptor for the Service object

👍

coveralls · 2017-04-06T21:45:41Z

Coverage increased (+0.5%) to 44.649% when pulling 407fe6339a26847421b685ad73a9a8cb628fe3f1 on nicksardo:backside-reencrypt into ed6987e on kubernetes:master.

coveralls · 2017-04-06T21:46:40Z

Coverage increased (+0.5%) to 44.649% when pulling 407fe6339a26847421b685ad73a9a8cb628fe3f1 on nicksardo:backside-reencrypt into ed6987e on kubernetes:master.

coveralls · 2017-04-06T21:47:27Z

Coverage increased (+0.5%) to 44.649% when pulling 407fe6339a26847421b685ad73a9a8cb628fe3f1 on nicksardo:backside-reencrypt into ed6987e on kubernetes:master.

coveralls · 2017-04-07T01:48:16Z

Coverage increased (+0.5%) to 44.7% when pulling 3479c42dbec34e0f641f639913c09c7a8857b739 on nicksardo:backside-reencrypt into ed6987e on kubernetes:master.

coveralls · 2017-04-07T16:19:12Z

Coverage increased (+0.5%) to 44.728% when pulling 5f53f0b on nicksardo:backside-reencrypt into 7c635a8 on kubernetes:master.

coveralls · 2017-04-07T16:40:11Z

Coverage increased (+0.8%) to 45.049% when pulling 5f53f0b on nicksardo:backside-reencrypt into 7c635a8 on kubernetes:master.

coveralls · 2017-04-07T16:46:21Z

Coverage increased (+0.8%) to 45.049% when pulling cba5bc6 on nicksardo:backside-reencrypt into 7c635a8 on kubernetes:master.

coveralls · 2017-04-07T17:08:48Z

Coverage increased (+0.9%) to 45.075% when pulling cba5bc6 on nicksardo:backside-reencrypt into 7c635a8 on kubernetes:master.

coveralls · 2017-04-07T19:17:18Z

Coverage increased (+0.9%) to 45.075% when pulling 81a1263 on nicksardo:backside-reencrypt into 7c635a8 on kubernetes:master.

coveralls · 2017-04-07T22:23:36Z

Coverage increased (+0.9%) to 45.075% when pulling 9bf3a9c on nicksardo:backside-reencrypt into 7c635a8 on kubernetes:master.

coveralls · 2017-04-12T22:35:48Z

Coverage increased (+0.9%) to 47.122% when pulling fe3ebc1157b3025a20ace99393ad77e7aad8854f on nicksardo:backside-reencrypt into 03cae88 on kubernetes:master.

coveralls · 2017-04-12T23:45:53Z

Coverage increased (+0.9%) to 47.151% when pulling b53061597c4531354e933c432833880e5910cebe on nicksardo:backside-reencrypt into 03cae88 on kubernetes:master.

bowei

Docs changes LGTM

bowei · 2017-04-11T07:26:16Z

controllers/gce/backends/backends.go

-		glog.Infof("Creating backend for %d instance groups, port %v named port %v",
-			len(igs), port, namedPort)
-		be, err = b.create(igs, namedPort, b.namer.BeName(port))
+		glog.Infof("Creating backend for %d instance groups, port %v named port %v", len(igs), p.Port, namedPort)


add V(1 or 2) I think?

bowei · 2017-04-11T07:26:31Z

controllers/gce/backends/backends.go

+
+	pProto := utils.GetHTTPScheme(p.Encrypted)
+	if be.Protocol != pProto {
+		glog.Infof("Updating backend protocol %v from %v to %v", pName, be.Protocol, pProto)


bowei · 2017-04-11T07:29:03Z

controllers/gce/backends/backends.go

 	}
 	pool := b.snapshotter.Snapshot()
 	for port := range pool {
-		p, err := strconv.Atoi(port)
+		p, err := strconv.ParseInt(port, 10, 64)


isn't the port 16-bit?

Yes, I believe existing code is following the compute library's use of int64 for Port. https://godoc.org/google.golang.org/api/compute/v1#NamedPort

Maybe I misunderstood - are you suggesting we assert that the port can fit in 16bits? We could but it would be unnecessary as the string is generated from within the same package.

discussed offline, probably should use strconv.ParseInt(port, 10, 16) to be consistent with port being uint16_t

bowei · 2017-04-11T07:29:51Z

controllers/gce/backends/backends.go

+	healthPath := p.Handler.HTTPGet.Path
+	// GCE requires a leading "/" for health check urls.
+	if !strings.HasPrefix(healthPath, "/") {
+		healthPath = fmt.Sprintf("/%v", healthPath)


healthPath = "/" + healthPath, seems heavyweight to use Sprintf

Agreed, was just moving, but I'll make the change.

bowei · 2017-04-11T07:31:31Z

controllers/gce/backends/backends.go

+	hc.RequestPath = healthPath
+	hc.Host = host
+	hc.Description = "kubernetes L7 health check from readiness probe."
+	// set a low health threshold and a high failure threshold.


this comment seems like it should be attached to healthchecks.DefaultHealthCheckInterval instead of here

Thanks for catching!

bowei

ignore LGTM -- thought change was only docs

coveralls · 2017-04-13T22:58:36Z

Coverage increased (+0.9%) to 47.105% when pulling b2a280d0b260906205deaa9886254e90402ae1f1 on nicksardo:backside-reencrypt into 03cae88 on kubernetes:master.

coveralls · 2017-04-14T17:57:17Z

Coverage increased (+1.0%) to 47.181% when pulling c42b7c0d9bfca5c6c2e9ec5c85f0487115c46373 on nicksardo:backside-reencrypt into 03cae88 on kubernetes:master.

…view comments; rename protocol constants

coveralls · 2017-04-15T01:43:20Z

Coverage increased (+0.9%) to 47.196% when pulling 6af52e7 on nicksardo:backside-reencrypt into 4817ddf on kubernetes:master.

bowei · 2017-04-17T18:15:14Z

controllers/gce/backends/backends.go

 	}
 	pool := b.snapshotter.Snapshot()
 	for port := range pool {
-		p, err := strconv.Atoi(port)
+		p, err := strconv.ParseInt(port, 10, 64)


discussed offline, probably should use strconv.ParseInt(port, 10, 16) to be consistent with port being uint16_t

bowei · 2017-04-17T18:24:26Z

controllers/gce/healthchecks/healthchecks.go

+	// borked, service level outages will get detected sooner
+	// by kube-proxy.
+	// DefaultHealthCheckInterval defines how frequently a probe runs
+	DefaultHealthCheckInterval = 60


Can we change this to 60 * time.Second, then the duration is more self-documenting?

I agree that this would be desirable. However, the compute api takes int64 of seconds. If we made this a time.Duration, we would need to int64(DefaultHealthCheckInterval.Seconds()) which isn't ideal.

Should we settle for renaming to DefaultHealthCheckIntervalSeconds?

bowei · 2017-04-17T18:24:33Z

controllers/gce/healthchecks/healthchecks.go

+	// DefaultUnhealthyThreshold defines the threshold of failure probes that declare a backend "unhealthy"
+	DefaultUnhealthyThreshold = 10
+	// DefaultTimeoutSeconds defines the timeout of each probe
+	DefaultTimeoutSeconds = 60


bowei · 2017-04-17T18:26:05Z

controllers/gce/healthchecks/healthchecks.go

-		wantHC.RequestPath = h.defaultPath
+
+	if existingHC.Protocol() != hc.Protocol() {
+		glog.Infof("Updating health check %v because it has protocol %v but need %v", existingHC.Name, existingHC.Type, hc.Type)


bowei · 2017-04-17T18:26:28Z

controllers/gce/healthchecks/healthchecks.go

 		// TODO: reconcile health checks, and compare headers interval etc.
 		// Currently Ingress doesn't expose all the health check params
 		// natively, so some users prefer to hand modify the check.
-		glog.Infof("Unexpected request path on health check %v, has %v want %v, NOT reconciling",
-			name, hc.RequestPath, wantHC.RequestPath)
+		glog.Infof("Unexpected request path on health check %v, has %v want %v, NOT reconciling", hc.Name, existingHC.RequestPath, hc.RequestPath)


bowei · 2017-04-17T18:57:07Z

controllers/gce/controller/utils.go

-
+		service = *as.(*api_v1.Service)
+		for _, sp := range service.Spec.Ports {
+			svcPort = sp


move down into the if

Let me explain my thought process on this styling choice. Ideally, svcPort is a pointer so nil can represent not found. However, service.Spec.Ports[N] is a struct which is why svcPort is a struct. You cannot take the address of sp because it's changing with every loop iteration. The other options are:

Declare a new struct for each iteration and instantiate with value of sp, then take address of that variable if port is matched. Then you can check the pointer for existence, but you're not saving any lines of codes.

Move svcPort=sp into if and check if ServicePort is a zero-value struct. Of course, if svcPort.Port != 0 is a bit of an eyesore.

IMO, the current implementation and first alternative are equally fine and a matter of preference.

bowei · 2017-04-17T19:01:47Z

controllers/gce/backends/backends.go

+			return "", err
+		}
+		if probe != nil {
+			glog.V(1).Infof("Applying httpGet settings of readinessProbe to health check on port %+v", sp)


bowei · 2017-04-17T19:10:26Z

controllers/gce/backends/backends.go

@@ -231,7 +286,7 @@ func (b *Backends) Add(port int64) error {
 // Delete deletes the Backend for the given port.
 func (b *Backends) Delete(port int64) (err error) {
 	name := b.namer.BeName(port)
-	glog.Infof("Deleting backend %v", name)
+	glog.Infof("Deleting backend service %v", name)


bowei · 2017-04-17T19:24:03Z

controllers/gce/backends/backends.go

-			len(igs), port, namedPort)
-		be, err = b.create(igs, namedPort, b.namer.BeName(port))
+		glog.V(1).Infof("Creating backend for %d instance groups, port %v named port %v", len(igs), p.Port, namedPort)
+		be, err = b.create(igs, namedPort, hcLink, p.Protocol, pName)
 		if err != nil {


Is it possible for be to be nil when it gets to here? Is this going to be a problem for the defer'd function above?

Yes, definitely possible be could be added to the snapshot as nil. However, this is as expected. I double-checked that all code fetching items from snapshotter are looking at the key and not at the backend-service value.

bowei · 2017-04-17T19:24:30Z

controllers/gce/backends/backends.go

+	}
+
+	if be.Protocol != string(p.Protocol) || existingHCLink != hcLink {
+		glog.Infof("Updating backend protocol %v (%v) for change in protocol (%v) or health check", pName, be.Protocol, string(p.Protocol))


coveralls · 2017-04-18T18:18:35Z

Coverage increased (+1.0%) to 47.224% when pulling 04af243 on nicksardo:backside-reencrypt into 4817ddf on kubernetes:master.

bowei

LGTM

coveralls · 2017-04-18T19:07:57Z

Coverage increased (+1.0%) to 47.224% when pulling 5fe199b on nicksardo:backside-reencrypt into 4817ddf on kubernetes:master.

ibawt · 2017-04-21T17:20:22Z

@nicksardo I see this is merged, how long until I see this in GKE? This will be quite relevant in our decision to either use the GCE load balancer or nginx for our soft launch.

nicksardo · 2017-04-21T17:31:15Z

#609 warrants a release of the GLBC and version bump in the manifest file for all release branches. Could land in 1.6.3 or 1.6.4 and GKE usually updates on patches about a week or two after release. I'll ping you from the version bump PR so you can track.

nicksardo · 2017-05-11T20:14:36Z

This feature was released in v0.9.3 which is included in K8s 1.6.3. @ibawt

ibawt · 2017-05-12T13:45:48Z

@nicksardo you are my favourite person in the world :O .

1.6.3 lines up nicely with our schedule

nicksardo · 2017-06-14T20:59:35Z

@ibawt and others,

Have any of you had a chance to use this feature yet? Any comments? Thanks

ibawt · 2017-06-14T21:07:52Z

@nicksardo yes, currently running HTTPS backends on two separate( read fairly large) deployments.
Everything worked as advertised, biggest pain was in the HC's updating in 1.6.4, so if we misplaced a readinessProbe and manually fixed, it went and updated it back to /. That's really our problem but did cause some goblins to come out.

I also tried to swap the service port, ie from 80 to 443 to indicate at least at a glance that it will be https, that part didn't work so hot. If I left it to 80 everything went fine, I even changed the targetPort. It looked something was doing an equality check, I was going deep dive into but haven't had a chance.

So tl;dr works great, thanks a bunch. The HC updating shook some goblins out that we should have done correctly anyways.

One thing I didn't find in the docs is what should I be using for the cert pair on the service backend? I ended up using the same cert as the ingress. Can we use different ones? How does the verification work? Can we use self signed if we provide the root?

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Mar 29, 2017

nicksardo self-assigned this Mar 29, 2017

nicksardo added the backend/gce label Mar 29, 2017

nicksardo force-pushed the backside-reencrypt branch 2 times, most recently from 994fda9 to 13b2d7b Compare April 5, 2017 22:06

nicksardo force-pushed the backside-reencrypt branch 2 times, most recently from b1851de to 5f53f0b Compare April 7, 2017 16:16

nicksardo changed the title ~~[GLBC] [WIP] Support backside re-encryption~~ [GLBC] Support backside re-encryption Apr 7, 2017

nicksardo assigned bowei and unassigned nicksardo Apr 7, 2017

nicksardo force-pushed the backside-reencrypt branch 2 times, most recently from 06f03c4 to 94da993 Compare April 12, 2017 05:31

bowei approved these changes Apr 13, 2017

View reviewed changes

bowei suggested changes Apr 13, 2017

View reviewed changes

Nick Sardo added 4 commits April 14, 2017 18:13

Support backside re-encryption

6c4fe22

modified docs

ce5257d

Updated more documentation

52bc74d

Fix: only take readinessProbe settings if protocols match; address re…

6af52e7

…view comments; rename protocol constants

nicksardo force-pushed the backside-reencrypt branch from c42b7c0 to 6af52e7 Compare April 15, 2017 01:14

bowei reviewed Apr 17, 2017

View reviewed changes

Nick Sardo added 2 commits April 18, 2017 10:53

Fix review comments

9b38164

Fix review comments

04af243

bowei approved these changes Apr 18, 2017

View reviewed changes

Minor fix to typing

5fe199b

nicksardo merged commit 642cb74 into kubernetes:master Apr 18, 2017

nicksardo deleted the backside-reencrypt branch April 18, 2017 19:44

nicksardo mentioned this pull request May 11, 2017

More ingress TLS modes kubernetes/kubernetes#32928

Closed

abstrctn mentioned this pull request Jun 12, 2017

GCP HealthCheck customizations lost upgrading to GLBC 0.9.3 #842

Closed

slaskawi mentioned this pull request May 8, 2020

KEYCLOAK-12307 TLS Termination configuration keycloak/keycloak-operator#186

Merged

4 tasks

[GLBC] Support backside re-encryption #519

[GLBC] Support backside re-encryption #519

Conversation

nicksardo commented Mar 29, 2017 • edited Loading

Requires:

Surfacing Service Annotation

Overview of Changes

Test Cases

k8s-reviewable commented Mar 29, 2017

coveralls commented Mar 29, 2017

aledbf commented Mar 29, 2017

nicksardo commented Mar 31, 2017

coveralls commented Apr 5, 2017

aledbf commented Apr 5, 2017

nicksardo commented Apr 5, 2017

aledbf commented Apr 5, 2017

coveralls commented Apr 6, 2017

coveralls commented Apr 6, 2017

coveralls commented Apr 6, 2017

coveralls commented Apr 7, 2017

coveralls commented Apr 7, 2017

coveralls commented Apr 7, 2017

coveralls commented Apr 7, 2017

coveralls commented Apr 7, 2017

coveralls commented Apr 7, 2017

coveralls commented Apr 7, 2017

coveralls commented Apr 12, 2017

coveralls commented Apr 12, 2017

bowei left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bowei left a comment

Choose a reason for hiding this comment

coveralls commented Apr 13, 2017

coveralls commented Apr 14, 2017

coveralls commented Apr 15, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

coveralls commented Apr 18, 2017

bowei left a comment

Choose a reason for hiding this comment

coveralls commented Apr 18, 2017

ibawt commented Apr 21, 2017

nicksardo commented Apr 21, 2017

nicksardo commented May 11, 2017

ibawt commented May 12, 2017

nicksardo commented Jun 14, 2017

ibawt commented Jun 14, 2017 • edited Loading

nicksardo commented Mar 29, 2017 •

edited

Loading

ibawt commented Jun 14, 2017 •

edited

Loading