metricbeat. elasticsearch-xpack module failed to start.

[gray380]

Hi,

I've tried to implement cluster monitoring according this manual.

Collection settings:

$ curl -k -X GET https://elk-es-00-int:9200/_cluster/settings?pretty
{
  "persistent" : {
    "xpack" : {
      "monitoring" : {
        "elasticsearch" : {
          "collection" : {
            "enabled" : "false"
          }
        },
        "collection" : {
          "enabled" : "true"
        }
      }
    }
  },
  "transient" : { }
}

metricbeat version:

$ metricbeat version
metricbeat version 7.5.0 (amd64), libbeat 7.5.0 [6d0d0ae079e5cb1d4f224801ac6df926dfb1594c built 2019-11-26 00:09:41 +0000 UTC]

enabled modules:

$ sudo metricbeat modules list
Enabled:
elasticsearch-xpack

/etc/metricbeat/modules.d/elasticsearch-xpack.yml:

- module: elasticsearch
  metricsets:
    - ccr
    - cluster_stats
    - index
    - index_recovery
    - index_summary
    - ml_job
    - node_stats
    - shard
  period: 10s
  hosts: ["https://elk-es-02-int:9200"]
  username: "remote_monitoring_user"
  password: "SuperPassword"
  xpack.enabled: true

faced error:

Dec 06 14:42:14 elk-es-02 metricbeat[4294]: 2019-12-06T14:42:14.148+0200        ERROR        instance/beat.go:916        Exiting: The elasticsearch module with xpack.enabled: true must have metricsets: [ccr enrich cluster_stats index index_recovery index_summary ml_job node_stats shard]
Dec 06 14:42:14 elk-es-02 metricbeat[4294]: Exiting: The elasticsearch module with xpack.enabled: true must have metricsets: [ccr enrich cluster_stats index index_recovery index_summary ml_job node_stats shard]

[kaiyan-sheng]

Hmm seems like you are missing enrich metricset?

- module: elasticsearch
  metricsets:
    - ccr
    - cluster_stats
    - enrich
    - index
    - index_recovery
    - index_summary
    - ml_job
    - node_stats
    - shard
  period: 10s
  hosts: ["https://elk-es-02-int:9200"]
  username: "remote_monitoring_user"
  password: "SuperPassword"
  xpack.enabled: true

[gray380]

Brilliant!
Thanks a lot!

[gray380]

Next level, one of three nodes shows the following INFO messages:

Error fetching data for metricset elasticsearch.enrich: HTTP error 403 in : 403 Forbidden
Error fetching data for metricset elasticsearch.ml_job: HTTP error 400 in : 400 Bad Request

[djb000m]

@gray380 I'm also getting a similar error on one of my master nodes.
All Metricbeat configs are identical across nodes but I'm only getting errors on one.

INFO module/wrapper.go:252 Error fetching data for metricset elasticsearch.enrich: HTTP error 403 in : 403 Forbidden

Have you had any luck with solving your errors?

My elasticsearch module is the same as yours, however I'm using the beats_system user credentials.

Update: I've fixed this by using the built-in elastic user. So it seems it was a permission issue, but I still don't know why errors were only generated on one of the nodes.

[jsermer]

Having the same issue as @djb000m and @gray380 on only one of my master nodes (happens to be the current master). I'm using the built-in remote_monitoring_user user (with a password set) which has the recommended roles from the docs:
remote_monitoring_agent, remote_monitoring_collector

Update: the elasticsearch.index_summary and elasticsearch.index errors I was receiving on only the master node were because of a couple closed indices. After opening them, the only erroor I receive is from the elasticsearch.enrich metricset. I discovered the issue by trying to curl against the _stats endpoint which told me about the closed index.

Looks like this change will address the enrich monitoring capabilities: elastic/elasticsearch#52027

[ycombinator]

Looks like this change will address the enrich monitoring capabilities: elastic/elasticsearch#52027

Indeed. Consequently, I'm going to close this issue here for now. Feel free to open a new one if there are other issues with the elasticsearch-xpack module.