Creating mutating webhook for daemonsets for auto-annotation #65

mitali-salvi · 2024-01-26T22:35:50Z

Description of changes:
Creating a new mutating webhook for daemonsets that will:

Add annotations to a daemonset if it doesnt already have them and is provided to the operator via the args
Removes annotations from a daemonset if it has them and is no longer provided to the operator via the args

Testing
Testing while watching the default namespace

{"level":"info","ts":"2024-01-26T22:03:37Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/mutate-v1-daemonset"}

--- Creating DS in default namespace ---
kubectl apply -f ds.yaml 
daemonset.apps/prometheus-daemonset created

kubectl get ds 
NAME                   DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
prometheus-daemonset   2         2         2       2            2           <none>          15s

kubectl describe ds prometheus-daemonset
Name:           prometheus-daemonset
Selector:       name=prometheus-exporter,tier=monitoring
Node-Selector:  <none>
Labels:         <none>
Annotations:    auto-annotation: true
                deprecated.daemonset.template.generation: 1
Desired Number of Nodes Scheduled: 2
Current Number of Nodes Scheduled: 2
Number of Nodes Scheduled with Up-to-date Pods: 2
Number of Nodes Scheduled with Available Pods: 2
Number of Nodes Misscheduled: 0
Pods Status:  2 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:       name=prometheus-exporter
                tier=monitoring
  Annotations:  instrumentation.opentelemetry.io/inject-java: true
  Containers:
   prometheus:
    Image:        prom/node-exporter
    Port:         80/TCP
    Host Port:    0/TCP
    Environment:  <none>
    Mounts:       <none>
  Volumes:        <none>
Events:
  Type    Reason            Age   From                  Message
  ----    ------            ----  ----                  -------
  Normal  SuccessfulCreate  23s   daemonset-controller  Created pod: prometheus-daemonset-k6t4n
  Normal  SuccessfulCreate  23s   daemonset-controller  Created pod: prometheus-daemonset-86f8s



kubectl describe pod prometheus-daemonset-86f8s
Name:             prometheus-daemonset-86f8s
Namespace:        default
Priority:         0
Service Account:  default
Node:             ip-192-168-63-8.ec2.internal/192.168.63.8
Start Time:       Fri, 26 Jan 2024 22:04:18 +0000
Labels:           controller-revision-hash=7b67f6b656
                  name=prometheus-exporter
                  pod-template-generation=1
                  tier=monitoring
Annotations:      instrumentation.opentelemetry.io/inject-java: true
Status:           Running
IP:               192.168.55.158
IPs:
  IP:           192.168.55.158
Controlled By:  DaemonSet/prometheus-daemonset
Init Containers:
  opentelemetry-auto-instrumentation-java:
    Container ID:  containerd://4f9b922f5160903162b6d8c121cc6d790827bb72511cd4bd7567de3d7542e0dc
    Image:         public.ecr.aws/aws-observability/adot-autoinstrumentation-java:v1.31.1
    Image ID:      public.ecr.aws/aws-observability/adot-autoinstrumentation-java@sha256:08d05d413361ea2a11e27b4392f6fe856e384136e8d28f995cfc98438aa8a3f2
    Port:          <none>
    Host Port:     <none>
    Command:
      cp
      /javaagent.jar
      /otel-auto-instrumentation-java/javaagent.jar
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Fri, 26 Jan 2024 22:04:19 +0000
      Finished:     Fri, 26 Jan 2024 22:04:19 +0000
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /otel-auto-instrumentation-java from opentelemetry-auto-instrumentation-java (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-59nwj (ro)
Containers:
  prometheus:
    Container ID:   containerd://1ca98d078d0dda32f665283536400fa38038c354201dbede8a6b5edf23d444fb
    Image:          prom/node-exporter
    Image ID:       docker.io/prom/node-exporter@sha256:4cb2b9019f1757be8482419002cb7afe028fdba35d47958829e4cfeaf6246d80
    Port:           80/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Fri, 26 Jan 2024 22:04:20 +0000
    Ready:          True
    Restart Count:  0
    Environment:
      OTEL_SMP_ENABLED:                    true
      OTEL_TRACES_SAMPLER_ARG:             endpoint=http://cloudwatch-agent.amazon-cloudwatch:2000
      OTEL_TRACES_SAMPLER:                 xray
      OTEL_EXPORTER_OTLP_TRACES_ENDPOINT:  http://cloudwatch-agent.amazon-cloudwatch:4315
      OTEL_AWS_SMP_EXPORTER_ENDPOINT:      http://cloudwatch-agent.amazon-cloudwatch:4315
      OTEL_METRICS_EXPORTER:               none
      JAVA_TOOL_OPTIONS:                    -javaagent:/otel-auto-instrumentation-java/javaagent.jar
      OTEL_SERVICE_NAME:                   prometheus-daemonset
      OTEL_RESOURCE_ATTRIBUTES_POD_NAME:   prometheus-daemonset-86f8s (v1:metadata.name)
      OTEL_RESOURCE_ATTRIBUTES_NODE_NAME:   (v1:spec.nodeName)
      OTEL_PROPAGATORS:                    tracecontext,baggage,b3,xray
      OTEL_RESOURCE_ATTRIBUTES:            k8s.container.name=prometheus,k8s.daemonset.name=prometheus-daemonset,k8s.namespace.name=default,k8s.node.name=$(OTEL_RESOURCE_ATTRIBUTES_NODE_NAME),k8s.pod.name=$(OTEL_RESOURCE_ATTRIBUTES_POD_NAME)
    Mounts:
      /otel-auto-instrumentation-java from opentelemetry-auto-instrumentation-java (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-59nwj (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  kube-api-access-59nwj:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
  opentelemetry-auto-instrumentation-java:
    Type:        EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:      
    SizeLimit:   200Mi
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/disk-pressure:NoSchedule op=Exists
                 node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                 node.kubernetes.io/not-ready:NoExecute op=Exists
                 node.kubernetes.io/pid-pressure:NoSchedule op=Exists
                 node.kubernetes.io/unreachable:NoExecute op=Exists
                 node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  57s   default-scheduler  Successfully assigned default/prometheus-daemonset-86f8s to ip-192-168-63-8.ec2.internal
  Normal  Pulled     56s   kubelet            Container image "public.ecr.aws/aws-observability/adot-autoinstrumentation-java:v1.31.1" already present on machine
  Normal  Created    56s   kubelet            Created container opentelemetry-auto-instrumentation-java
  Normal  Started    56s   kubelet            Started container opentelemetry-auto-instrumentation-java
  Normal  Pulling    55s   kubelet            Pulling image "prom/node-exporter"
  Normal  Pulled     55s   kubelet            Successfully pulled image "prom/node-exporter" in 124.459204ms (124.467572ms including waiting)
  Normal  Created    55s   kubelet            Created container prometheus
  Normal  Started    55s   kubelet            Started container prometheus

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

mitali-salvi · 2024-01-26T22:37:10Z

internal/config/main.go

@@ -15,6 +15,22 @@ const (
 	defaultCollectorConfigMapEntry = "cwagentconfig.json"
 )

+// AnnotationConfig details the resources that have enabled
+// auto-annotation for each instrumentation type.
+type AnnotationConfig struct {


this is just to create the necessary functions to test the webhook, this is subject to change

mitali-salvi · 2024-01-26T22:38:05Z

pkg/annotation/annotationutil.go

+}
+
+// getAutoAnnotatedLang returns the list of languages to be auto-annotated for the given kubernetes workload (deployment, daemon-set, stateful-set)
+// TODO can this function be made generic for supporting all workloads instead of just daemon-set


I am open to suggestions if there is a better way to handle this functionality

sky333999 · 2024-01-29T15:20:10Z

internal/webhook/daemonsetmutation/webhookhandler.go

+)
+
+// +kubebuilder:webhook:path=/mutate-v1-daemonset,mutating=true,failurePolicy=ignore,groups="apps",resources=daemonsets,verbs=create;update,versions=v1,name=mdaemonset.kb.io,sideEffects=none,admissionReviewVersions=v1
+// +kubebuilder:rbac:groups="",resources=namespaces,verbs=list;watch


We dont need to list/watch namespaces do we?

Yeah well I think this is used to watch namespaces incase the resource is not created yet, I think the pod mutator does something similar - https://github.com/aws/amazon-cloudwatch-agent-operator/blob/main/internal/webhook/podmutation/webhookhandler.go#L66

sky333999 · 2024-01-29T15:23:07Z

internal/webhook/daemonsetmutation/webhookhandler.go

+	}
+
+	// we use the req.Namespace here because the pod might have not been created yet
+	ns := corev1.Namespace{}


Dont think we need ns here.

Ties in to the previous comment, upstream has this change incase the resource is not created yet

okankoAMZ

This looks great! Added some minor comments.

go.mod

pkg/annotation/annotationutil.go

pkg/annotation/daemonset_test.go

Creating mutating webhook for daemonsets for auto-annotation

a2865c0

mitali-salvi self-assigned this Jan 26, 2024

mitali-salvi commented Jan 26, 2024

View reviewed changes

sky333999 reviewed Jan 29, 2024

View reviewed changes

mitali-salvi marked this pull request as ready for review January 29, 2024 16:11

okankoAMZ reviewed Jan 29, 2024

View reviewed changes

go.mod Show resolved Hide resolved

pkg/annotation/annotationutil.go Show resolved Hide resolved

pkg/annotation/daemonset_test.go Show resolved Hide resolved

okankoAMZ approved these changes Jan 30, 2024

View reviewed changes

mitali-salvi closed this Jan 31, 2024

mitali-salvi deleted the daemon-set branch February 1, 2024 17:44

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Creating mutating webhook for daemonsets for auto-annotation #65

Creating mutating webhook for daemonsets for auto-annotation #65

mitali-salvi commented Jan 26, 2024

mitali-salvi Jan 26, 2024

mitali-salvi Jan 26, 2024

sky333999 Jan 29, 2024

mitali-salvi Jan 29, 2024

sky333999 Jan 29, 2024

mitali-salvi Jan 29, 2024

okankoAMZ left a comment

Creating mutating webhook for daemonsets for auto-annotation #65

Creating mutating webhook for daemonsets for auto-annotation #65

Conversation

mitali-salvi commented Jan 26, 2024

mitali-salvi Jan 26, 2024

Choose a reason for hiding this comment

mitali-salvi Jan 26, 2024

Choose a reason for hiding this comment

sky333999 Jan 29, 2024

Choose a reason for hiding this comment

mitali-salvi Jan 29, 2024

Choose a reason for hiding this comment

sky333999 Jan 29, 2024

Choose a reason for hiding this comment

mitali-salvi Jan 29, 2024

Choose a reason for hiding this comment

okankoAMZ left a comment

Choose a reason for hiding this comment