[Metricbeat] Manifest referenced in docs for k8 deployment results in metricbeat being reported as host

[gingerwizard]

• Version: 6.2.4
• Operating System: Linux K8 on GCP

The docs reference this manifest showing how to deploy metricbeat on k8 as a deployment - in order to poll the kube-state-metrics service for the kubernetes metricset. This configuration results in the metricbeat pod being reported as a host i.e. beat.name becomes the container name on all docs. This can be confusing in some visualizations.

To force it to set its hostname and beat.name to the underlying host i would suggest mounting proc and adding the switch -system.hostfs=/hostfs.
In order for the kube-state-metrics to still be contactable via DNS the following is also required:

dnsPolicy: ClusterFirstWithHostNet

e.g.

---
# Deploy singleton instance in the whole cluster for some unique data sources, like kube-state-metrics
apiVersion: apps/v1
kind: Deployment
metadata:
  name: metricbeat
  namespace: kube-system
  labels:
    k8s-app: metricbeat
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: metricbeat
  template:
    metadata:
      labels:
        k8s-app: metricbeat
    spec:
      serviceAccountName: metricbeat
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      containers:
      - name: metricbeat
        image: docker.elastic.co/beats/metricbeat:6.2.4
        imagePullPolicy: Always
        args: [
          "-c", "/etc/metricbeat.yml",
          "-e",
          "-system.hostfs=/hostfs",
        ]
        env:
        - name: CLOUD_ID
          valueFrom:
            secretKeyRef:
              name: elastic
              key: CLOUD_ID
        - name: CLOUD_AUTH
          valueFrom:
            secretKeyRef:
              name: elastic
              key: CLOUD_AUTH
        securityContext:
          runAsUser: 0
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 100Mi
        volumeMounts:
        - name: config
          mountPath: /etc/metricbeat.yml
          readOnly: true
          subPath: metricbeat.yml
        #we do this so the container isn't reported as a host and thus appears as a node
        - name: proc
          mountPath: /hostfs/proc
          readOnly: true
      volumes:
      - name: proc
        hostPath:
          path: /proc
      - name: config
        configMap:
          defaultMode: 0600
          name: metricbeat-deployment-config

[ph]

@exekias or @jsoriano mind to take a look?

[praseodym]

When using kube-state-metrics the name of the host on which the Metricbeat pod is running is not very useful information either, is it?

[gingerwizard]

It’s not but it atleast means the container doesn’t show up in host listings. Ideally it would send an empty field, but the beat could be running other modules.

[exekias]

Agreed, this more an aesthetic fix, as we show beat hostname in the Hosts overview dashboard, we don't want to introduce noise by reporting metricbeat container name as a host.