pkg/metrics: Add Deployment ownerReference to metrics Service object #1037

lilic · 2019-01-31T12:26:12Z

Description of the change:
Add Deployment ownerReference to metrics Service object, by getting the Pod the operator is currently running in and via ReplicaSet getting the Deployment Object.

Motivation for the change:
When operator Deployment is deleted currently the metrics Service operator creates was not deleted.

Closes #459

shawn-hurley

Some concerns about corner cases, but overall approach LGTM

pkg/metrics/metrics.go

shawn-hurley · 2019-01-31T14:42:30Z

Re: the edge cases

What if a user just creates a pod for the operator. This might not be best practice but the metrics will fail to start up.
What if a user uses something other than a deployment? maybe they use a replica set or they use a stateful set? I think that will cause issues here.

I think that we probably just need to degrade gracefully IMO.

lilic · 2019-01-31T15:00:57Z

@shawn-hurley Good points!

I think that we probably just need to degrade gracefully IMO.

Right now we don't error out just log the errors coming from exposing the metrics port:
https://github.com/operator-framework/operator-sdk/blob/b1d5e62bcf750515d5b34fe9705844217373a5cf/pkg/scaffold/cmd.go

What if a user just creates a pod for the operator. This might not be best practice but the metrics will fail to start up.
What if a user uses something other than a deployment? maybe they use a replica set or they use a stateful set? I think that will cause issues here.

Instead erroring out if no Deployment or ReplicaSet exist, will retry with other possible objects mentioned above. And if no Owner exists then we error out but we log as metioned above. WDYT?

shawn-hurley · 2019-01-31T15:15:14Z

If it is just a pod, could we not just create a service to only select that pod, and set the owner ref to that pod?

I would just like the metrics to always be exposed by a service for all the ways that one could run an operator but if the above is not possible then this makes sense to me.

joelanford · 2019-01-31T15:40:34Z

Instead erroring out if no Deployment or ReplicaSet exist, will retry with other possible objects mentioned above. And if no Owner exists then we error out but we log as metioned above. WDYT?

It could also be a custom resource that owns the Pod (or the Pod's parent, grandparent, etc.), so this gets a bit tricky.

I haven't totally thought this through, but maybe we go up the owner chain looking for specific types (Deployments, DaemonSets, StatefulSets). If we find one of those, we stop and return that, but if not, we traverse until we find an object that doesn't have an owner reference and use that instead?

If the pod itself doesn't have an owner reference, I agree with Shawn that we could create the service to select and be owned by just the Pod.

lilic · 2019-01-31T15:44:40Z

If the pod itself doesn't have an owner reference, I agree with Shawn that we could create the service to select and be owned by just the Pod.

Yes, agree as well.

If we find one of those, we stop and return that, but if not, we traverse until we find an object that doesn't have an owner reference and use that instead?

"If we find one of those, we stop and return that" that part should be easy enough, already have a POC implemented locally, as for the other part guess lots of use of the Unstructured object will be used :D

shawn-hurley · 2019-01-31T16:07:13Z

I haven't totally thought this through, but maybe we go up the owner chain looking for specific types (Deployments, DaemonSets, StatefulSets). If we find one of those, we stop and return that, but if not, we traverse until we find an object that doesn't have an owner reference and use that instead?

Could we just use a Replica Set? that seems like the controller that actually manages the pods that the Deployments and DaemonSets and StatefulSets create correct? I think that a Custom one "should" use that as well. If the pod is not managed by a ReplicaSet, maybe just degrade to the pod? This would make the logic simpler and I think more understandable for a user who may be seeing some odd behavior and cover the 95% use cases?

joelanford · 2019-01-31T16:31:13Z

@shawn-hurley I'm pretty sure DaemonSets and StatefulSets manage Pods directly with no ReplicaSet involved.

Another problem with ReplicaSets (when used with Deployments at least) is that the Deployment scales them down to 0, but does not always delete them when a new rollout occurs. This is based on the Deployment's spec.revisionHistoryLimit.

I'm not sure how the Deployment control loop works, but I can envision a problem where the ReplicaSet that owns the Service is 3 or 4 revisions old. If it gets cleaned up by the Deployment controller after the current Pods for the current ReplicaSet have started, the Service will get garbage-collected and no new Pods would start to re-create it.

shawn-hurley · 2019-01-31T16:46:22Z

Another problem with ReplicaSets (when used with Deployments at least) is that the Deployment scales them down to 0, but does not always delete them when a new rollout occurs. This is based on the Deployment's spec.revisionHistoryLimit.

This makes sense; I was hoping because I think we need to figure out how to handle DeploymentConfigs as well. It looks like the correct thing to do no is what @lilic has already done locally :).

lilic · 2019-01-31T16:55:07Z

I think tracing up until a known last k8s native Object is found, e.g. Deployment or StatefulSet orDeamonSet or if none are found a Pod even if the last known is a ReplicaSet, due to the problem mentioned by @joelanford, and using that as the OwnerRef.

I don't like defaulting to a Pod as those get deleted more often and we would create a lot of Services throughout the life of the operator. But we can add a warning info log to the user that using Pod, if this is bothersome remove creation of Service altogether.

This would only happen if a user is deploying the operator using a CR, which I am not sure how often this would happen.

joelanford · 2019-01-31T20:40:24Z

I think tracing up until a known last k8s native Object is found, e.g. Deployment or StatefulSet orDeamonSet or if none are found a Pod even if the last known is a ReplicaSet, due to the problem mentioned by @joelanford, and using that as the OwnerRef.

@lilic that sounds reasonable to me.

@shawn-hurley to solve the problem with DeploymentConfig and other similar objects, could we have another stop condition if we find a ReplicaSet or ReplicationController that has an owner reference? In which case we would use the owner reference of the ReplicaSet or ReplicationController for the Service?

lilic · 2019-02-05T10:45:17Z

@shawn-hurley @joelanford this should be ready for another look. Thanks!

shawn-hurley · 2019-02-05T16:05:47Z

pkg/metrics/metrics.go

+}
+
+// findFinalOwnerRef tries to locate the final controller/owner based on the owner reference provided.
+func findFinalOwnerRef(ctx context.Context, client crclient.Client, ns string, ownerRef *metav1.OwnerReference) (*metav1.OwnerReference, error) {


I really think that this function has to handle DeploymentConfigs

I agree, but I think we should try to do it as generically as possible. I wonder if we could consolidate the switch statement cases for known types to be:

case "ReplicaSet", "ReplicationController": rsrc := &unstructured.Unstructured{} key := crclient.ObjectKey{Namespace: ns, Name: ownerRef.Name} if err := client.Get(ctx, key, rsrc); err != nil { return nil, err } rsrcOwner := metav1.GetControllerOf(rsrc) // If we find an owner for the RS/RC, return that. // Otherwise, just return the ownerRef directly. if rsrcOwner != nil { return rsrcOwner, nil } return ownerRef, nil case "DaemonSet", "StatefulSet", "Job": // I think we can just return this directly. // Any reason to fetch the object first? return ownerRef, nil

I think this would cover Deployments and DeploymentConfigs generically, and it would be a bit more future-proof since we're being less explicit about some of the types we care about.

I agree with @joelanford. A generic approach will reduce code significantly.

// Any reason to fetch the object first? return ownerRef, nil

Agreed! 🤦‍♀️ Done!

This would then return the RS/RC as the owner directly, above we said we want to avoid doing that? Did I misunderstand your suggestion above maybe #1037 (comment)

// If we find an owner for the RS/RC, return that. // Otherwise, just return the ownerRef directly. if rsrcOwner != nil { return rsrcOwner, nil }

The reason was to avoid returning RS/RC as the owner and to avoid returning non k8s controllers owners as well.

@lilic I think if the ReplicaSet or ReplicationController has an owner we should return its owner, regardless of the Kind of the owner. That would cover the cases for Deployment and DeploymentConfig implicitly and it would mean other custom or future native K8s or Openshift types that own and manage ReplicaSets and ReplicationControllers would be supported without any code changes.

If the ReplicaSet or ReplicationController does not have an owner, I think it makes the most sense to use the ReplicaSet or ReplicationController as the owner directly, since it would be the highest level Kind we found from our traversal of the owner references from the pod.

shawn-hurley

LGTM just depends on if we all agree on supporting DeploymentConfigs

estroz · 2019-02-07T20:18:26Z

pkg/k8sutil/k8sutil.go

+
+	pod := &corev1.Pod{
+		TypeMeta: metav1.TypeMeta{
+			APIVersion: "v1",


This suggestion isn't strictly necessary but using corev1.SchemaGroupVersion.Version is a better way of setting this value IMO, as it tracks dependency version changes. WDYT?

Is it necessary to set the TypeMeta for a concrete type when using the controller-runtime client? I thought this was only necessary with unstructured.Unstructured.

No idea why this was done this way. I can remove it, but it comes originally and is used still in the leader.go myOwnerRef function, so not sure if they need to set this maybe. I would leave it, as it doesn't hurt. https://github.com/operator-framework/operator-sdk/pull/1037/files#diff-a0eaeead6981b3b2f753c1ecbba711ccL145

so it can be used outside of leader package.

This allows the Service cleanup to happen when deployment is deleted.

Tracing up until a known last k8s native controller is found, e.g. Deployment or StatefulSet or DeamonSet or if none are found a Pod even if the last known is a ReplicaSet.

lilic · 2019-02-08T13:31:06Z

Also added updating the Service. We can have the case when it doesn't get deleted correctly by the GC and the owner controller is deleted, but when a new Pod gets created the OwnerRef will change and then we need to make sure to Update the existing Service.

When Service already exists but the OwnerRef might change, we must make sure to the propagate those changes to the Service object by updating the object.

lilic · 2019-02-13T11:38:03Z

@joelanford As the suggestions were already very close to this, I decided to implement a fully generic solution, which covers all the cases we talked about: core Kubernetes objects, OpenShift objects (like DeploymentConfigs), and potentially any custom controllers. We simply walk up the tree of OwnerReferences until we can't find an OwnerReference that is a controller anymore and then use that.

The question we had earlier about ReplicaSets that are scaled to 0 won't have an effect here either as that means there can never be Pods that have those ReplicaSets as owners, as there are none. Beyond that Kubernetes garbage collection removes OwnerReferences from a list of owners if there are still owners left and only truly garbage collects the objects if no existing owner is left (this behavior is shown and proven here in OLM: https://github.com/operator-framework/operator-lifecycle-manager/blob/master/test/e2e/gc_e2e_test.go#L22).

I believe this solves all the problems/concerns raised before and is even simpler code. Please have another look, thanks!

joelanford

Looks pretty good to me. I like the completely generic approach for finding the root owner reference 👍

A couple of nits and questions.

pkg/metrics/metrics.go

joelanford · 2019-02-14T22:51:27Z

pkg/metrics/metrics.go

+
+	// Get Owner that the Pod belongs to
+	ownerRef := metav1.GetControllerOf(pod)
+	finalOwnerRef, err := findFinalOwnerRef(ctx, client, ns, ownerRef)


Nit: We could simplify things further if we declare a new gvkObject interface and pass the object itself into findFinalOwnerRef:

func getPodOwnerRef(ctx context.Context, client crclient.Client, ns string) (*metav1.OwnerReference, error) { // Get current Pod the operator is running in pod, err := k8sutil.GetPod(ctx, client, ns) if err != nil { return nil, err } return findFinalOwnerRef(ctx, client, ns, pod) } type gvkObject interface { metav1.Object GroupVersionKind() schema.GroupVersionKind } func findFinalOwnerRef(ctx context.Context, client crclient.Client, ns string, obj gvkObject) (*metav1.OwnerReference, error) { ownerRef := metav1.GetControllerOf(obj) if ownerRef == nil { log.V(1).Info("Pods owner found", "Kind", ownerRef.Kind, "Name", ownerRef.Name, "Namespace", ns) return metav1.NewControllerRef(obj, obj.GroupVersionKind()), nil } ownerObj := &unstructured.Unstructured{} ownerObj.SetAPIVersion(ownerRef.APIVersion) ownerObj.SetKind(ownerRef.Kind) err := client.Get(ctx, types.NamespacedName{Namespace: ns, Name: ownerRef.Name}, ownerObj) if err != nil { return nil, err } return findFinalOwnerRef(ctx, client, ns, ownerObj) }

I would prefer to leave the function as is, as it has a clear entry point and you can see from the declaration what it does.

joelanford · 2019-02-14T23:02:15Z

pkg/k8sutil/k8sutil.go

+
+	pod := &corev1.Pod{
+		TypeMeta: metav1.TypeMeta{
+			APIVersion: "v1",


Is it necessary to set the TypeMeta for a concrete type when using the controller-runtime client? I thought this was only necessary with unstructured.Unstructured.

pkg/k8sutil/k8sutil.go

joelanford · 2019-02-14T23:25:40Z

pkg/metrics/metrics.go

@@ -97,11 +103,14 @@ func initOperatorService(port int32, portName string) (*v1.Service, error) {
 	if err != nil {
 		return nil, err
 	}
+
+	label := map[string]string{"name": operatorName}
+
 	service := &v1.Service{
 		ObjectMeta: metav1.ObjectMeta{
 			Name:      operatorName,


Hmm. So I'm wondering if your original instinct to give the operator developer a chance to tweak the service before submitting it to the cluster was a good one.

As is, if an operator developer wants to expose any other ports via a service, they will either have to create a separate service with a different name, or won't be able to enable metrics.

Another option could be to have ExposeMetricsPort take a service as an input parameter, and if its nil initialize it with the metrics port. WDYT?

I would agree, my only fear is that it will break the API. Can open an issue to discuss this, as it's not strictly related to this PR anyways?

Co-Authored-By: LiliC <cosiclili@gmail.com>

lilic · 2019-02-18T15:45:15Z

Think this is ready for another look @joelanford, thanks!

joelanford

LGTM 👍

lilic requested review from estroz and shawn-hurley and removed request for estroz January 31, 2019 12:26

openshift-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Jan 31, 2019

lilic requested review from estroz, joelanford, hasbro17 and AlexNPavel January 31, 2019 12:26

shawn-hurley requested changes Jan 31, 2019

View reviewed changes

pkg/metrics/metrics.go Outdated Show resolved Hide resolved

lilic changed the title ~~pkg/metrics: Add Deployment ownerReference to metrics Service object~~ WIP: pkg/metrics: Add Deployment ownerReference to metrics Service object Jan 31, 2019

openshift-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jan 31, 2019

lilic mentioned this pull request Feb 1, 2019

pkg/leader: Switch to controller-runtime config pkg #1045

Merged

lilic force-pushed the lili/owner-ref-service branch 3 times, most recently from 297f85f to 5489c40 Compare February 4, 2019 19:10

lilic changed the title ~~WIP: pkg/metrics: Add Deployment ownerReference to metrics Service object~~ pkg/metrics: Add Deployment ownerReference to metrics Service object Feb 4, 2019

shawn-hurley reviewed Feb 5, 2019

View reviewed changes

shawn-hurley approved these changes Feb 5, 2019

View reviewed changes

estroz reviewed Feb 7, 2019

View reviewed changes

lilic force-pushed the lili/owner-ref-service branch from 3047f78 to 3860b22 Compare February 8, 2019 13:08

lilic added 6 commits February 8, 2019 14:12

pkg/*: Move getting current pod to k8sutil

6c771ea

so it can be used outside of leader package.

pkg/metrics: Add deployment ownerRef to Service

366b2e9

This allows the Service cleanup to happen when deployment is deleted.

pkg/metrics: Use controller-runtime GetConfig

6c77b55

pkg/metrics: Expand logic for owner retrival

1fa02a0

Tracing up until a known last k8s native controller is found, e.g. Deployment or StatefulSet or DeamonSet or if none are found a Pod even if the last known is a ReplicaSet.

Gopkg.lock: Rerun dep ensure

ab00257

pkg/metrics: Skip unnecessarily getting OwnerRef

23b0946

lilic force-pushed the lili/owner-ref-service branch from 3860b22 to 23b0946 Compare February 8, 2019 13:15

lilic changed the title ~~pkg/metrics: Add Deployment ownerReference to metrics Service object~~ WIP: pkg/metrics: Add Deployment ownerReference to metrics Service object Feb 8, 2019

pkg/metrics: Update Service if it exists

814be07

When Service already exists but the OwnerRef might change, we must make sure to the propagate those changes to the Service object by updating the object.

lilic force-pushed the lili/owner-ref-service branch from e4c3198 to 814be07 Compare February 8, 2019 14:41

lilic changed the title ~~WIP: pkg/metrics: Add Deployment ownerReference to metrics Service object~~ pkg/metrics: Add Deployment ownerReference to metrics Service object Feb 8, 2019

openshift-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Feb 8, 2019

lilic mentioned this pull request Feb 8, 2019

doc/user: Add metrics and service-monitor docs #719

Merged

pkg/metrics: Make retrieval of owner ref more generic

98731bf

joelanford reviewed Feb 14, 2019

View reviewed changes

joelanford and others added 3 commits February 15, 2019 10:45

Update pkg/k8sutil/k8sutil.go

f1924d7

Co-Authored-By: LiliC <cosiclili@gmail.com>

Update pkg/k8sutil/k8sutil.go

eec9bfa

Co-Authored-By: LiliC <cosiclili@gmail.com>

pkg/metrics: Use method to get ownerRef

b2ec0f3

lilic force-pushed the lili/owner-ref-service branch from 34954b3 to b2ec0f3 Compare February 15, 2019 10:23

lilic mentioned this pull request Feb 18, 2019

pkg/metrics: Return Service so user can make changes #1107

Closed

joelanford approved these changes Feb 19, 2019

View reviewed changes

lilic merged commit 21e899c into operator-framework:master Feb 20, 2019

lilic deleted the lili/owner-ref-service branch February 20, 2019 08:53

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pkg/metrics: Add Deployment ownerReference to metrics Service object #1037

pkg/metrics: Add Deployment ownerReference to metrics Service object #1037

lilic commented Jan 31, 2019

shawn-hurley left a comment

shawn-hurley commented Jan 31, 2019

lilic commented Jan 31, 2019

shawn-hurley commented Jan 31, 2019

joelanford commented Jan 31, 2019

lilic commented Jan 31, 2019

shawn-hurley commented Jan 31, 2019

joelanford commented Jan 31, 2019

shawn-hurley commented Jan 31, 2019

lilic commented Jan 31, 2019

joelanford commented Jan 31, 2019

lilic commented Feb 5, 2019

shawn-hurley Feb 5, 2019

joelanford Feb 5, 2019

estroz Feb 7, 2019

lilic Feb 8, 2019 •

edited

Loading

lilic Feb 8, 2019

joelanford Feb 12, 2019

shawn-hurley left a comment

estroz Feb 7, 2019 •

edited

Loading

joelanford Feb 14, 2019

lilic Feb 15, 2019

lilic commented Feb 8, 2019

lilic commented Feb 13, 2019

joelanford left a comment

joelanford Feb 14, 2019

lilic Feb 15, 2019

joelanford Feb 14, 2019

joelanford Feb 14, 2019

lilic Feb 15, 2019

lilic Feb 18, 2019

lilic commented Feb 18, 2019

joelanford left a comment

pkg/metrics: Add Deployment ownerReference to metrics Service object #1037

pkg/metrics: Add Deployment ownerReference to metrics Service object #1037

Conversation

lilic commented Jan 31, 2019

shawn-hurley left a comment

Choose a reason for hiding this comment

shawn-hurley commented Jan 31, 2019

lilic commented Jan 31, 2019

shawn-hurley commented Jan 31, 2019

joelanford commented Jan 31, 2019

lilic commented Jan 31, 2019

shawn-hurley commented Jan 31, 2019

joelanford commented Jan 31, 2019

shawn-hurley commented Jan 31, 2019

lilic commented Jan 31, 2019

joelanford commented Jan 31, 2019

lilic commented Feb 5, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lilic Feb 8, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shawn-hurley left a comment

Choose a reason for hiding this comment

estroz Feb 7, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lilic commented Feb 8, 2019

lilic commented Feb 13, 2019

joelanford left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lilic commented Feb 18, 2019

joelanford left a comment

Choose a reason for hiding this comment

lilic Feb 8, 2019 •

edited

Loading

estroz Feb 7, 2019 •

edited

Loading