Elasticsearch CCR metricset returns 405 errors when run against ES in Cloud

[bczifra]

When using metricbeat-7.6.0-darwin-x86_64, commenting out any of the metricsets in /etc/metricbeat/modules.d/elasticsearch-xpack.yml, such as:

- module: elasticsearch
  metricsets:
    - cluster_stats
    - index
    - index_recovery
    - index_summary
    - node_stats
    - shard
#    - ccr
    - enrich
    - ml_job

results in an error message:

2020-02-24T12:38:35.193+0100	ERROR	instance/beat.go:933	Exiting: The elasticsearch module with xpack.enabled: true must have metricsets: [ccr enrich cluster_stats index index_recovery index_summary ml_job node_stats shard]
Exiting: The elasticsearch module with xpack.enabled: true must have metricsets: [ccr enrich cluster_stats index index_recovery index_summary ml_job node_stats shard]

Related issues:

• [6.8] Difficulty configuring Elasticsearch module in Metricbeat  #14963
• metricbeat. elasticsearch-xpack module failed to start. #14984

For some context, this came up while trying to use the module with an Elastic Cloud deployment which apparently doesn't support the CCR stats endpoint. Instead, metricbeat logs:

2020-02-24T12:47:19.888+0100	INFO	module/wrapper.go:252	Error fetching data for metricset elasticsearch.ccr: HTTP error 405 in : 405 Method Not Allowed

Interestingly, I do see documents created in the metricbeat index for this document with:

{
  "@timestamp" : "2020-02-24T11:33:47.592Z",
  "agent" : {
    "type" : "metricbeat",
    "ephemeral_id" : "$$$",
    "hostname" : "$$$",
    "id" : "c912fcc4-90fa-4b77-8d12-0587ec040ed5",
    "version" : "7.6.0"
  },
  "event" : {
    "duration" : 730521798,
    "dataset" : "elasticsearch.ccr",
    "module" : "elasticsearch"
  },
  "metricset" : {
    "name" : "ccr",
    "period" : 10000
  },
  "error" : {
    "message" : "HTTP error 405 in : 405 Method Not Allowed"
  },
  "service" : {
    "address" : "$$$",
    "type" : "elasticsearch"
  },
  "ecs" : {
    "version" : "1.4.0"
  },
  "host" : {
    "name" : "$$$",
    "architecture" : "x86_64",
    "os" : {
      "build" : "17G10021",
      "platform" : "darwin",
      "version" : "10.13.6",
      "family" : "darwin",
      "name" : "Mac OS X",
      "kernel" : "17.7.0"
    },
    "id" : "$$$",
    "hostname" : "$$$"
  }
}

However, I don't see documents generated for any of the other metricsets.

GET metricbeat-7.6.0/_search?filter_path=aggregations.history
{
  "size": 0,
  "query": {
    "range": {
      "@timestamp": {
        "gte": "now-10m",
        "lte": "now/m"
      }
    }
  }, 
  "aggs": {
    "history": {
      "date_histogram": {
        "field": "@timestamp",
        "fixed_interval": "1m"
      },
      "aggs": {
        "metricset": {
          "terms": {
            "field": "metricset.name",
            "size": 10
          }
        }
      }
    }
  }
}

results in:

{
  "aggregations" : {
    "history" : {
      "buckets" : [
        {
          "key_as_string" : "2020-02-24T11:47:00.000Z",
          "key" : 1582544820000,
          "doc_count" : 6,
          "metricset" : {
            "doc_count_error_upper_bound" : 0,
            "sum_other_doc_count" : 0,
            "buckets" : [
              {
                "key" : "ccr",
                "doc_count" : 6
              }
            ]
          }
        },
        {
          "key_as_string" : "2020-02-24T11:48:00.000Z",
          "key" : 1582544880000,
          "doc_count" : 6,
          "metricset" : {
            "doc_count_error_upper_bound" : 0,
            "sum_other_doc_count" : 0,
            "buckets" : [
              {
                "key" : "ccr",
                "doc_count" : 6
              }
            ]
          }
        },
        {
          "key_as_string" : "2020-02-24T11:49:00.000Z",
          "key" : 1582544940000,
          "doc_count" : 6,
          "metricset" : {
            "doc_count_error_upper_bound" : 0,
            "sum_other_doc_count" : 0,
            "buckets" : [
              {
                "key" : "ccr",
                "doc_count" : 6
              }
            ]
          }
        },
        {
          "key_as_string" : "2020-02-24T11:50:00.000Z",
          "key" : 1582545000000,
          "doc_count" : 6,
          "metricset" : {
            "doc_count_error_upper_bound" : 0,
            "sum_other_doc_count" : 0,
            "buckets" : [
              {
                "key" : "ccr",
                "doc_count" : 6
              }
            ]
          }
        },
        {
          "key_as_string" : "2020-02-24T11:51:00.000Z",
          "key" : 1582545060000,
          "doc_count" : 6,
          "metricset" : {
            "doc_count_error_upper_bound" : 0,
            "sum_other_doc_count" : 0,
            "buckets" : [
              {
                "key" : "ccr",
                "doc_count" : 6
              }
            ]
          }
        },
        {
          "key_as_string" : "2020-02-24T11:52:00.000Z",
          "key" : 1582545120000,
          "doc_count" : 6,
          "metricset" : {
            "doc_count_error_upper_bound" : 0,
            "sum_other_doc_count" : 0,
            "buckets" : [
              {
                "key" : "ccr",
                "doc_count" : 6
              }
            ]
          }
        },
        {
          "key_as_string" : "2020-02-24T11:53:00.000Z",
          "key" : 1582545180000,
          "doc_count" : 6,
          "metricset" : {
            "doc_count_error_upper_bound" : 0,
            "sum_other_doc_count" : 0,
            "buckets" : [
              {
                "key" : "ccr",
                "doc_count" : 6
              }
            ]
          }
        },
        {
          "key_as_string" : "2020-02-24T11:54:00.000Z",
          "key" : 1582545240000,
          "doc_count" : 4,
          "metricset" : {
            "doc_count_error_upper_bound" : 0,
            "sum_other_doc_count" : 0,
            "buckets" : [
              {
                "key" : "ccr",
                "doc_count" : 4
              }
            ]
          }
        }
      ]
    }
  }
}

I wonder if the failure collecting one metricset prevents the reporting of the other metricsets.

[ycombinator]

Chatted with @bczifra off-issue. Summarizing:

• HTTP 405 errors should be ignored, and the metricset should return early.
• We should probably do this for all ES APIs across all metricsets, but definitely any non-OSS APIs, which are more likely to be absent in certain environments/distributions.

[elasticmachine]

Pinging @elastic/stack-monitoring (Stack monitoring)

[elasticmachine]

Pinging @elastic/integrations-services (Team:Services)

[ycombinator]

Thinking about this some more, I don't think we should silently ignore 405 errors. At the same time, I don't think we want to spam our logs either. So perhaps we should ignore such errors and log about it at the DEBUG level.

[tophercullen]

This issue is marked as closed. However, as far as I can tell, this is still broken (tested 7.6.2) and prevents collection of all metricsets from Elastic's own cloud service. @bczifra were you able to resolve the collection problem?

[ycombinator]

@tophercullen This issue has been fixed by #17073. We label the PRs to show which versions those changes become available in. In this case, the fix will be available in 7.7.0.

[tophercullen]

Ok thanks!