Filebeat keeps sending monitoring to "Standalone Cluster", metricbeat works with exact same config

[sgreszcz]

For confirmed bugs, please report:

• Version: 7.9.2
• Operating System: kubernetes docker container on Ubuntu 18.04
• Steps to Reproduce:

Using the recommended templates for metricbeat and filebeat for kubernetes:
https://raw.githubusercontent.com/elastic/beats/master/deploy/kubernetes/filebeat-kubernetes.yaml
https://raw.githubusercontent.com/elastic/beats/master/deploy/kubernetes/metricbeat-kubernetes.yaml

Metricbeat sends the data and monitoring to the same cluster as Filebeat. The metrics and logs as well as monitoring all get to that server from both beats, but filebeat puts the monitoring under "Standalone Cluster" rather than the expected "docker-cluster". Metricbeats from the k8s servers does show the monitoring on the expected "docker-cluster" along with all the other 7.8.1 filebeat/metricbeat agents running on other servers (not kubernetes) from docker containers.

Filebeat config:

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: kube-system
  labels:
    k8s-app: filebeat
data:
  filebeat.yml: |-
    filebeat.autodiscover:
      providers:
        - type: kubernetes
          node: ${NODE_NAME}
          hints.enabled: true
          hints.default_config:
            type: container
            paths:
              - /var/log/containers/*${data.kubernetes.container.id}.log

    processors:
      - add_host_metadata:
      - add_kubernetes_metadata:

    monitoring:
      enabled: true

    output.elasticsearch:
      hosts: '${ELASTICSEARCH_HOST}'
      username: ${ELASTICSEARCH_USERNAME}
      password: ${ELASTICSEARCH_PASSWORD}

Metricbeat config:

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: metricbeat-daemonset-config
  namespace: kube-system
  labels:
    k8s-app: metricbeat
data:
  metricbeat.yml: |-
    metricbeat.config.modules:
      # Mounted `metricbeat-daemonset-modules` configmap:
      path: ${path.config}/modules.d/*.yml
      # Reload module configs as they change:
      reload.enabled: false

    metricbeat.autodiscover:
      providers:
        - type: kubernetes
          scope: cluster
          node: ${NODE_NAME}
          unique: true
          templates:
            - config:
                - module: kubernetes
                  hosts: ["kube-state-metrics:8080"]
                  period: 10s
                  add_metadata: true
                  metricsets:
                    - state_node
                    - state_deployment
                    - state_daemonset
                    - state_replicaset
                    - state_pod
                    - state_container
                    - state_cronjob
                    - state_resourcequota
                    - state_statefulset
                - module: kubernetes
                  metricsets:
                    - apiserver
                  hosts: ["https://${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT}"]
                  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
                  ssl.certificate_authorities:
                    - /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
                  period: 30s
                # Uncomment this to get k8s events:
                #- module: kubernetes
                #  metricsets:
                #    - event
        # To enable hints based autodiscover uncomment this:
        - type: kubernetes
          node: ${NODE_NAME}
          hints.enabled: true

    processors:
      - add_host_metadata:
      - add_kubernetes_metadata:

    monitoring:
      enabled: true

    output.elasticsearch:
      hosts: '${ELASTICSEARCH_HOST}'
      username: ${ELASTICSEARCH_USERNAME}
      password: ${ELASTICSEARCH_PASSWORD}

k8s Filebeat monitoring (6 agents) on "Standalone Cluster"

k8s Metricbeat and the other Docker server Metricbeat and Filebeat monitoring (24 agents) on "docker-cluster":

[elasticmachine]

Pinging @elastic/stack-monitoring (Stack monitoring)

[elasticmachine]

Pinging @elastic/integrations-services (Team:Services)

[sgreszcz]

The root cause to this issue was that filebeat was not working correctly and shipping logs from the k8s clusters. When the problem with data shipping was resolved as per issue #11834 the monitoring "joined up" with the actual ES cluster name and is no longer in "Standalone Cluster"