Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Network policy incorrectly applied due to mismatch between pod selector and agent pod labels #1490

Open
bogatuadrian opened this issue Oct 28, 2024 · 1 comment

Comments

@bogatuadrian
Copy link

Context

We're using the Datadog Operator to deploy a Datadog agent with Kubernetes network policies enabled, but the operator doesn't use the correct pod selectors when creating the NetworkPolicy resources to target the deployed agent and cluster agent pods.

Problem

This might lead to either a false sense of security if your network plugin allows traffic by default, or Datadog simply not working if your network plugin disallows traffic by default. We fall in the second category, having multiple network issues in both the agent and the cluster agent because traffic is denied by default and the network policies set up by the operator do not seem to work. For us, this leads to losing observability data.

Setup

We configure our Datadog agent something like this:

kind: DatadogAgent
apiVersion: datadoghq.com/v2alpha1
metadata:
  name: datadog
spec:
  global:
    networkPolicy:
      create: true
      flavor: kubernetes

This leads to the creation of agents and cluster agents with the following spec (some labels omitted for brevity):

apiVersion: v1
kind: Pod
metadata:
  generateName: datadog-agent-
  labels:
    agent.datadoghq.com/component: agent
    agent.datadoghq.com/name: datadog
    agent.datadoghq.com/provider: ""
    app.kubernetes.io/component: agent
    app.kubernetes.io/instance: datadog-agent # Notice how this is prefixed with "datadog" (likely from the `DatadogAgent` resource)
    app.kubernetes.io/managed-by: datadog-operator
    app.kubernetes.io/name: datadog-agent-deployment
    app.kubernetes.io/part-of: datadog-datadog

However the network policies created by the operator use the following pod selectors:

➜ kubectl get networkpolicies.networking.k8s.io                                                 
NAME                            POD-SELECTOR                                                                                 AGE
datadog-agent                   app.kubernetes.io/instance=agent,app.kubernetes.io/part-of=datadog-datadog                   77m
datadog-cluster-agent           app.kubernetes.io/instance=cluster-agent,app.kubernetes.io/part-of=datadog-datadog           77m
datadog-cluster-checks-runner   app.kubernetes.io/instance=cluster-checks-runner,app.kubernetes.io/part-of=datadog-datadog   77m

Notice how the pods use datadog-agent and the policies use agent as value for the app.kubernetes.io/instance label.

Code breadcrumbs

I took a look at the code and I might have found where the discrepancy might occur.

On one hand you have the Network Policy being created with the hard-coded agent value in

suffix = apicommon.DefaultAgentResourceSuffix

DefaultAgentResourceSuffix = "agent"

On the other hand, the app.kubernetes.io/instance label seems to be set here
labels[kubernetes.AppKubernetesInstanceLabelKey] = instanceName

with the value from here
// GetAgentName return the Agent name based on the DatadogAgent info
func GetAgentName(dda metav1.Object) string {
return fmt.Sprintf("%s-%s", dda.GetName(), apicommon.DefaultAgentResourceSuffix)
}

It looks like in our case the value would be datadog-agent, while the network policy has the hardcoded agent, hence the mismatch in pod selector, leading to the network policies not targeting the correct pods.

Potential solution

I'm guessing this discrepancy appeared when the Datadog Operator added support for running multiple DD agents at the same time, and probably migrated to using <agent-name>-agent, <agent-name>-cluster-agent etc. as the app.kubernetes.io/instance value, while not updating the network policy accordingly.

The solution would be to correctly set the pod selectors for the created network policies, for each Datadog agent created by the operator.

Environment

We are using the latest versions for both the operator and the agent, available at the time of writing.

Operator helm chart version: 2.1.0
Operator version: 1.9.0
Agent version: 7.58.1
Kubernetes Distribution: EKS

@bogatuadrian
Copy link
Author

One more note: not only that the pod selectors used to attach the network policies are not correct, the pod selectors used in from and to in the ingress/egress allowlists are also not correct, e.g.:

    - from:
      - podSelector:
          matchLabels:
            app.kubernetes.io/instance: agent # no agent pod exists with this label
            app.kubernetes.io/part-of: datadog-datadog

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant