[Metricbeat] RabbitMQ module stopped reporting rabbitmq.queue.consumers.utilisation.pct

[kaiyan-sheng]

Tested with Metricbeat 7.2.0 BC2, Elasticsearch 7.2.0 BC2 and Kibana 7.2.0 BC2.
I think this bug is introduced by #12089. When testing metricbeat rabbitmq module with 7.2.0 BC2, I cannot see any rabbitmq.queue.consumers.utilisation.pct data points from the metrics. For example, from Kibana I can see rabbitmq.queue metric:

{
  "_index": "metricbeat-7.2.0-2019.05.29-000001",
  "_type": "_doc",
  "_id": "B5ymBmsBE7Iz_p254_wJ",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2019-05-30T02:52:07.834Z",
    "ecs": {
      "version": "1.0.0"
    },
    "host": {
      "name": "KaiyanMacBookPro",
      "id": "9C7FAB7B-29D1-5926-8E84-158A9CA3E25D",
      "hostname": "KaiyanMacBookPro",
      "architecture": "x86_64",
      "os": {
        "platform": "darwin",
        "version": "10.13.6",
        "family": "darwin",
        "name": "Mac OS X",
        "kernel": "17.7.0",
        "build": "17G7024"
      }
    },
    "agent": {
      "hostname": "KaiyanMacBookPro",
      "id": "db57fc22-27fe-4f66-8fdc-3dc45bc3f024",
      "version": "7.2.0",
      "type": "metricbeat",
      "ephemeral_id": "b4aeb3bd-dbd0-4d66-9b64-d513988194c3"
    },
    "metricset": {
      "name": "queue"
    },
    "rabbitmq": {
      "vhost": "/",
      "node": {
        "name": "rabbit@localhost"
      },
      "queue": {
        "name": "hello",
        "auto_delete": false,
        "state": "running",
        "exclusive": false,
        "memory": {
          "bytes": 11188
        },
        "durable": false,
        "disk": {
          "writes": {},
          "reads": {}
        },
        "consumers": {
          "utilisation": {},
          "count": 1
        },
        "arguments": {},
        "messages": {
          "total": {
            "count": 0,
            "details": {
              "rate": 0
            }
          },
          "ready": {
            "count": 0,
            "details": {
              "rate": 0
            }
          },
          "unacknowledged": {
            "details": {
              "rate": 0
            },
            "count": 0
          },
          "persistent": {
            "count": 0
          }
        }
      }
    },
    "service": {
      "address": "localhost:15672",
      "type": "rabbitmq"
    },
    "event": {
      "dataset": "rabbitmq.queue",
      "module": "rabbitmq",
      "duration": 4250014
    }
  },
  "fields": {
    "@timestamp": [
      "2019-05-30T02:52:07.834Z"
    ]
  },
  "highlight": {
    "event.dataset": [
      "@kibana-highlighted-field@rabbitmq.queue@/kibana-highlighted-field@"
    ]
  },
  "sort": [
    1559184727834
  ]
}

But querying directly from rabbitmq shows:

kaiyansheng@KaiyanMacBookPro:/usr/local/Cellar/rabbitmq/3.7.15/sbin$ ./rabbitmqctl list_queues name messages messages_ready state consumer_utilisation
Timeout: 60.0 seconds ...
Listing queues for vhost / ...
name	messages	messages_ready	state	consumer_utilisation
hello	0	0	running	1.0

Based on rabbitmqctl result, rabbitmq.queue.consumers.utilisation.pct should be 1.0 instead.

I removed the `` from the code and rebuilt/ran metricbeat:

diff --git a/metricbeat/module/rabbitmq/queue/data.go b/metricbeat/module/rabbitmq/queue/data.go
index 2ebae73a1..7a072a492 100644
--- a/metricbeat/module/rabbitmq/queue/data.go
+++ b/metricbeat/module/rabbitmq/queue/data.go
@@ -45,7 +45,7 @@ var (
                "consumers": s.Object{
                        "count": c.Int("consumers"),
                        "utilisation": s.Object{
-                               "pct": c.Int("consumer_utilisation", s.IgnoreAllErrors),
+                               "pct": c.Int("consumer_utilisation"),
                        },
                },
                "messages": s.Object{

After that consumer_utilisation data showed up:

{
  "_index": "metricbeat-8.0.0-2019.05.29-000001",
  "_type": "_doc",
  "_id": "O5ywBmsBE7Iz_p25GPwI",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2019-05-30T03:02:11.227Z",
    "event": {
      "dataset": "rabbitmq.queue",
      "module": "rabbitmq",
      "duration": 3296567
    },
    "metricset": {
      "name": "queue"
    },
    "service": {
      "address": "localhost:15672",
      "type": "rabbitmq"
    },
    "rabbitmq": {
      "vhost": "/",
      "node": {
        "name": "rabbit@localhost"
      },
      "queue": {
        "auto_delete": false,
        "exclusive": false,
        "messages": {
          "persistent": {
            "count": 0
          },
          "total": {
            "count": 0,
            "details": {
              "rate": 0
            }
          },
          "ready": {
            "count": 0,
            "details": {
              "rate": 0
            }
          },
          "unacknowledged": {
            "count": 0,
            "details": {
              "rate": 0
            }
          }
        },
        "durable": false,
        "consumers": {
          "count": 1,
          "utilisation": {
            "pct": 1
          }
        },
        "name": "hello",
        "memory": {
          "bytes": 34988
        },
        "state": "running",
        "disk": {
          "writes": {},
          "reads": {}
        },
        "arguments": {}
      }
    },
    "host": {
      "name": "KaiyanMacBookPro",
      "id": "9C7FAB7B-29D1-5926-8E84-158A9CA3E25D",
      "hostname": "KaiyanMacBookPro",
      "architecture": "x86_64",
      "os": {
        "build": "17G7024",
        "platform": "darwin",
        "version": "10.13.6",
        "family": "darwin",
        "name": "Mac OS X",
        "kernel": "17.7.0"
      }
    },
    "agent": {
      "id": "5a6a982d-6218-43a1-9c80-ccf6a3c2549d",
      "version": "8.0.0",
      "type": "metricbeat",
      "ephemeral_id": "e8bf96fd-5809-415b-9285-3f957acaa2d0",
      "hostname": "KaiyanMacBookPro"
    },
    "ecs": {
      "version": "1.0.0"
    }
  },
  "fields": {
    "@timestamp": [
      "2019-05-30T03:02:11.227Z"
    ]
  },
  "highlight": {
    "event.dataset": [
      "@kibana-highlighted-field@rabbitmq.queue@/kibana-highlighted-field@"
    ]
  },
  "sort": [
    1559185331227
  ]
}

[kaiyan-sheng]

Never mind 😬 It's just timing issue. I was able to get rabbitmq.queue.consumers.utilisation.pct right after I created this issue... Closing it!