Enable beats monitoring

[anyasabo]

You can use Beats to monitor other beats (for instance, Metricbeat monitoring Filebeat). I think it's worth considering doing some of the setup by default. Off the top of my head:

• Expose port 5066 on the container
• Configure certificates
  • The SSL docs don't mention it is configurable, but the "use metricbeat to monitor filebeat" docs do
  • Presumably this allows authn/authz going by that same doc?
• Enable the web server
• Listen on all addrs

I was wondering if the certificates and authentication are actually necessary, but it looks like the stats show a lot of info: https://gist.github.com/anyasabo/a53875b6735208ed7d513e29120fe0d9

The state endpoint also provides a decent amount of info:

{
  "beat": {
    "name": "gke-sabo-dev-cluster-default-pool-c24f3bc2-ggb1"
  },
  "host": {
    "architecture": "x86_64",
    "containerized": "containerized",
    "hostname": "gke-sabo-dev-cluster-default-pool-c24f3bc2-ggb1",
    "id": "83a8f1f835d84a9a9bf5417cecaf0c8e",
    "os": {
      "codename": "Core",
      "family": "redhat",
      "kernel": "4.19.112+",
      "name": "CentOS Linux",
      "platform": "centos",
      "version": "7 (Core)"
    }
  },
  "input": {
    "count": 23,
    "names": [
      "container"
    ]
  },
  "management": {
    "enabled": false
  },
  "module": {
    "count": 2,
    "names": [
      "elasticsearch"
    ]
  },
  "output": {
    "name": "elasticsearch"
  },
  "outputs": {
    "elasticsearch": {
      "cluster_uuid": "1-M3lRkMTXmmEnIq6V50sg"
    }
  },
  "queue": {
    "name": "mem"
  },
  "service": {
    "id": "086e9713-c1df-4309-a3ea-ae7c946f6809",
    "name": "filebeat",
    "version": "7.8.0"
  }

And it also is a daemonset with host access so it seems like securing it is a good idea.

That said, I was still struggling to get this to work as part of Stack Monitoring even without the operator's assistance.

I tried modifying the stack monitoring recipe, but it seemed that it was having issues associating the beats with the monitored cluster (or any cluster) even though the data ended up in the .monitoring indices. I thought maybe setting the cluster UUID to the monitored cluster manually (as described here: elastic/beats#13182) might do it but I was not that lucky.

It may also be worth just using internal collection for beats, but I did not get much of a chance to try it yet.

[david-kow]

We currently don't have plans to introduce any new mechanism around enabling Stack Monitoring, so I'm closing this issue. For anyone looking, the basic guidance for this topic can be found in a blog post that we've published.