[8.15](backport #40836) [metricbeat] meraki module with device_health…

… metricset (#40862)

* [metricbeat] meraki module with device_health metricset    (#40836)

* inital meraki module and metricsets

* initial refactor for single meraki metricset device_health

* added tunnel support aka VPN support by Device

* adding interfaces aka switch ports and switch port status

* processing review comments

* refactored for comments

* fixing default metricset

* Removing unused variables and adding text to required variables

* add go module deps

* fixing for loop and starting on fields yml

* remove %d from loop condition

* fixing review comments and loss latency

* update dashboard-api lib to version with MIT licensed ratelimiter

* remove unused fields in metricset struct

* remove device details for now

* Major refactor to improve overall readability and device status/metric associations.

This commit removes a couple of data points from the original code, namely:
 - network appliance port settings - these settings are not defined per-device,
   but per-network, and as such don't fit into the current naming taxonomy.
 - site-to-site VPN settings - similarly to the above, these are not per-device attibutes.

In addition there is a limitation included on including only a single metric bucket (as returned
from the API) per metricbeat collection loop. There needs to be some better logging/config
validation on this, which will come later; for now, just run with a maximum 5 minute collection
interval.

This commit includes the field mappings too, but it currently causes events to get dropped due
to the index field limit getting hit (max 10k fields). You can get around this by setting
'setup.template.settings.index.mapping.total_fields.limit' to a higher value. We maybe can't
include all these mappings in the metricbeat module, but for now they are here to allow testing
with standalone metricbeat.

* Add maximum collection interval check to avoid missing uplink loss and latency metrics.

Remove 'percent' units, because the data does not fit the 0-1 elastic interpretation of percent.

* revert accidental changes to reference config

* add example in data.json

* review comments: license headers, missing/incorrect mapping

* result of 'go mod tidy' ... trying to unblock the CI builds

* fix linter issues

* update NOTICE.txt

* add release tag to fields.yml, update docs

* review feedback: keyword -> dat

* commit results of 'make update' in x-pack metricbeat for CI

* remove mappings to avoid hitting field limits in global metricbeat index template

* add one field back to please the docs CI

* more linter errors guarding against shared loop var pointers

* fix error introduced in previous commit

* add exception for golangci-lint G601 which is no longer required in go1.22+

* remove the guards against taking addresses of loop vars

* i can't get the golanglint-ci config to get picked up, so i'm reverting that change and putting back the annoying index loop code

* Update go.mod to match toolchain version to .go-version

Co-authored-by: Craig MacKenzie <craig.mackenzie@elastic.co>

---------

Co-authored-by: Dan Hiebert <dan@semplicityinc.com>
Co-authored-by: Craig MacKenzie <craig.mackenzie@elastic.co>
(cherry picked from commit ae71c67)

# Conflicts:
#	NOTICE.txt
#	go.mod
#	go.sum
#	x-pack/metricbeat/metricbeat.reference.yml

* fix go.mod merge conflict

* fix merge errors in NOTICE.txt

* fix merge errors in metricbeat.reference.yml

* add result of running 'mage update'

---------

Co-authored-by: Tom Myers <106530686+tommyers-elastic@users.noreply.github.com>

go.mod

@@ -1,6 +1,8 @@
 module github.com/elastic/beats/v7
 
-go 1.22
+go 1.22.0
+
+toolchain go1.22.6
 
 require (
 	cloud.google.com/go/bigquery v1.59.1
@@ -144,16 +146,16 @@ require (
 	go.uber.org/atomic v1.11.0 // indirect
 	go.uber.org/multierr v1.11.0
 	go.uber.org/zap v1.27.0
-	golang.org/x/crypto v0.25.0
+	golang.org/x/crypto v0.27.0
 	golang.org/x/lint v0.0.0-20210508222113-6edffad5e616
-	golang.org/x/mod v0.18.0
-	golang.org/x/net v0.27.0
+	golang.org/x/mod v0.21.0
+	golang.org/x/net v0.29.0
 	golang.org/x/oauth2 v0.18.0
-	golang.org/x/sync v0.7.0
-	golang.org/x/sys v0.22.0
-	golang.org/x/text v0.16.0
+	golang.org/x/sync v0.8.0
+	golang.org/x/sys v0.25.0
+	golang.org/x/text v0.18.0
 	golang.org/x/time v0.5.0
-	golang.org/x/tools v0.21.1-0.20240508182429-e35e4ccd0d2d
+	golang.org/x/tools v0.25.0
 	google.golang.org/api v0.169.0
 	google.golang.org/genproto v0.0.0-20240213162025-012b6fc9bca9 // indirect
 	google.golang.org/grpc v1.64.0
@@ -214,6 +216,7 @@ require (
 	github.com/gorilla/mux v1.8.0
 	github.com/gorilla/websocket v1.5.0
 	github.com/icholy/digest v0.1.22
+	github.com/meraki/dashboard-api-go/v3 v3.0.9
 	github.com/otiai10/copy v1.12.0
 	github.com/pierrec/lz4/v4 v4.1.18
 	github.com/pkg/xattr v0.4.9
@@ -263,6 +266,7 @@ require (
 	github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.17.5 // indirect
 	github.com/aws/aws-sdk-go-v2/service/sso v1.20.5 // indirect
 	github.com/aws/aws-sdk-go-v2/service/ssooidc v1.23.4 // indirect
+	github.com/benbjohnson/clock v1.3.0 // indirect
 	github.com/beorn7/perks v1.0.1 // indirect
 	github.com/cespare/xxhash v1.1.0 // indirect
 	github.com/cilium/ebpf v0.13.2 // indirect
@@ -292,6 +296,7 @@ require (
 	github.com/go-openapi/jsonpointer v0.19.6 // indirect
 	github.com/go-openapi/jsonreference v0.20.2 // indirect
 	github.com/go-openapi/swag v0.22.3 // indirect
+	github.com/go-resty/resty/v2 v2.11.0 // indirect
 	github.com/go-stack/stack v1.8.0 // indirect
 	github.com/gobuffalo/here v0.6.7 // indirect
 	github.com/goccy/go-json v0.10.2 // indirect
@@ -302,6 +307,7 @@ require (
 	github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
 	github.com/golang/protobuf v1.5.4 // indirect
 	github.com/google/gnostic-models v0.6.8 // indirect
+	github.com/google/go-querystring v1.1.0 // indirect
 	github.com/google/licenseclassifier v0.0.0-20221004142553-c1ed8fcf4bab // indirect
 	github.com/google/s2a-go v0.1.7 // indirect
 	github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510 // indirect
@@ -372,8 +378,9 @@ require (
 	go.opentelemetry.io/otel/metric v1.28.0 // indirect
 	go.opentelemetry.io/otel/trace v1.28.0 // indirect
 	go.opentelemetry.io/proto/otlp v1.3.1 // indirect
+	go.uber.org/ratelimit v0.3.1 // indirect
 	golang.org/x/exp v0.0.0-20231127185646-65229373498e // indirect
-	golang.org/x/term v0.22.0 // indirect
+	golang.org/x/term v0.24.0 // indirect
 	golang.org/x/xerrors v0.0.0-20231012003039-104605ab7028 // indirect
 	google.golang.org/appengine v1.6.8 // indirect
 	google.golang.org/genproto/googleapis/rpc v0.0.0-20240513163218-0867130af1f8 // indirect
@@ -414,6 +421,7 @@ replace (
 	github.com/golang/glog => github.com/elastic/glog v1.0.1-0.20210831205241-7d8b5c89dfc4
 	github.com/google/gopacket => github.com/elastic/gopacket v1.1.20-0.20211202005954-d412fca7f83a
 	github.com/insomniacslk/dhcp => github.com/elastic/dhcp v0.0.0-20200227161230-57ec251c7eb3 // indirect
+	github.com/meraki/dashboard-api-go/v3 => github.com/tommyers-elastic/dashboard-api-go/v3 v3.0.0-20240913150833-a945473a8f25
 	github.com/snowflakedb/gosnowflake => github.com/snowflakedb/gosnowflake v1.6.19
 	github.com/tonistiigi/fifo => github.com/containerd/fifo v0.0.0-20190816180239-bda0ff6ed73c
 	k8s.io/kubernetes v1.13.0 => k8s.io/kubernetes v1.24.15

metricbeat/docs/fields.asciidoc

@@ -60,6 +60,7 @@ grouped in the following categories:
 * <<exported-fields-linux>>
 * <<exported-fields-logstash>>
 * <<exported-fields-memcached>>
+* <<exported-fields-meraki>>
 * <<exported-fields-mongodb>>
 * <<exported-fields-mssql>>
 * <<exported-fields-munin>>
@@ -50453,6 +50454,17 @@ type: long
 
 --
 
+[[exported-fields-meraki]]
+
+
+
+*`meraki.device.serial`*::
++
+--
+type: keyword
+
+--
+
 [[exported-fields-mongodb]]
 == MongoDB fields
 

metricbeat/docs/modules/meraki.asciidoc

@@ -0,0 +1,47 @@
+////
+This file is generated! See scripts/mage/docs_collector.go
+////
+
+:modulename: meraki
+:edit_url: https://github.com/elastic/beats/edit/main/x-pack/metricbeat/module/meraki/_meta/docs.asciidoc
+
+
+[[metricbeat-module-meraki]]
+[role="xpack"]
+== Cisco Meraki module
+
+beta[]
+
+This is the meraki module.
+
+
+
+:edit_url:
+
+[float]
+=== Example configuration
+
+The Cisco Meraki module supports the standard configuration options that are described
+in <<configuration-metricbeat>>. Here is an example configuration:
+
+[source,yaml]
+----
+metricbeat.modules:
+- module: meraki
+  metricsets: ["device_health"]
+  enabled: true
+  period: 300s
+  apiKey: "Meraki dashboard API key"
+  organizations: ["Meraki organization ID"]  
+----
+
+[float]
+=== Metricsets
+
+The following metricsets are available:
+
+* <<metricbeat-metricset-meraki-device_health,device_health>>
+
+include::meraki/device_health.asciidoc[]
+
+:edit_url!:

metricbeat/docs/modules/meraki/device_health.asciidoc

@@ -0,0 +1,29 @@
+////
+This file is generated! See scripts/mage/docs_collector.go
+////
+:edit_url: https://github.com/elastic/beats/edit/main/x-pack/metricbeat/module/meraki/device_health/_meta/docs.asciidoc
+
+
+[[metricbeat-metricset-meraki-device_health]]
+[role="xpack"]
+=== Cisco Meraki device_health metricset
+
+beta[]
+
+include::../../../../x-pack/metricbeat/module/meraki/device_health/_meta/docs.asciidoc[]
+
+
+:edit_url:
+
+==== Fields
+
+For a description of each field in the metricset, see the
+<<exported-fields-meraki,exported fields>> section.
+
+Here is an example document generated by this metricset:
+
+[source,json]
+----
+include::../../../../x-pack/metricbeat/module/meraki/device_health/_meta/data.json[]
+----
+:edit_url!:

metricbeat/docs/modules_list.asciidoc

@@ -213,6 +213,8 @@ This file is generated! See scripts/mage/docs_collector.go
 |<<metricbeat-metricset-logstash-node_stats,node_stats>>   
 |<<metricbeat-module-memcached,Memcached>>     |image:./images/icon-no.png[No prebuilt dashboards]    |  
 .1+| .1+|  |<<metricbeat-metricset-memcached-stats,stats>>   
+|<<metricbeat-module-meraki,Cisco Meraki>>  beta[]   |image:./images/icon-no.png[No prebuilt dashboards]    |  
+.1+| .1+|  |<<metricbeat-metricset-meraki-device_health,device_health>> beta[]  
 |<<metricbeat-module-mongodb,MongoDB>>     |image:./images/icon-yes.png[Prebuilt dashboards are available]    |  
 .5+| .5+|  |<<metricbeat-metricset-mongodb-collstats,collstats>>   
 |<<metricbeat-metricset-mongodb-dbstats,dbstats>>   
@@ -367,6 +369,7 @@ include::modules/kvm.asciidoc[]
 include::modules/linux.asciidoc[]
 include::modules/logstash.asciidoc[]
 include::modules/memcached.asciidoc[]
+include::modules/meraki.asciidoc[]
 include::modules/mongodb.asciidoc[]
 include::modules/mssql.asciidoc[]
 include::modules/munin.asciidoc[]

x-pack/metricbeat/metricbeat.reference.yml

@@ -1103,6 +1103,14 @@ metricbeat.modules:
   hosts: ["localhost:11211"]
   enabled: true
 
+#----------------------------- Cisco Meraki Module -----------------------------
+- module: meraki
+  metricsets: ["device_health"]
+  enabled: true
+  period: 300s
+  apiKey: "Meraki dashboard API key"
+  organizations: ["Meraki organization ID"]  
+
 #------------------------------- MongoDB Module -------------------------------
 - module: mongodb
   metricsets: ["dbstats", "status", "collstats", "metrics", "replstatus"]
@@ -1311,9 +1319,11 @@ metricbeat.modules:
   # Password to use when connecting to PostgreSQL. Empty by default.
   #password: pass
 
-#----------------------- Prometheus Typed Metrics Module -----------------------
+#------------------------------ Prometheus Module ------------------------------
+# Metrics collected from a Prometheus endpoint
 - module: prometheus
   period: 10s
+  metricsets: ["collector"]
   hosts: ["localhost:9090"]
   metrics_path: /metrics
   #metrics_filters:
@@ -1327,12 +1337,6 @@ metricbeat.modules:
   #ssl.certificate_authorities:
   #  - /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt
 
-  # Use Elasticsearch histogram type to store histograms (beta, default: false)
-  # This will change the default layout and put metric type in the field name
-  #use_types: true
-
-  # Store counter rates instead of original cumulative counters (experimental, default: false)
-  #rate_counters: true
 
 # Metrics sent by a Prometheus server using remote_write option
 #- module: prometheus
@@ -1344,18 +1348,6 @@ metricbeat.modules:
   #ssl.certificate: "/etc/pki/server/cert.pem"
   #ssl.key: "/etc/pki/server/cert.key"
 
-  # Use Elasticsearch histogram type to store histograms (beta, default: false)
-  # This will change the default layout and put metric type in the field name
-  #use_types: true
-
-  # Store counter rates instead of original cumulative counters (experimental, default: false)
-  #rate_counters: true
-
-  # Define patterns for counter and histogram types so as to identify metrics' types according to these patterns
-  #types_patterns:
-  #  counter_patterns: []
-  #  histogram_patterns: []
-
 # Metrics that will be collected using a PromQL
 #- module: prometheus
 #  metricsets: ["query"]
@@ -1382,11 +1374,9 @@ metricbeat.modules:
 #    params:
 #      query: "some_value"
 
-#------------------------------ Prometheus Module ------------------------------
-# Metrics collected from a Prometheus endpoint
+#----------------------- Prometheus Typed Metrics Module -----------------------
 - module: prometheus
   period: 10s
-  metricsets: ["collector"]
   hosts: ["localhost:9090"]
   metrics_path: /metrics
   #metrics_filters:
@@ -1400,6 +1390,12 @@ metricbeat.modules:
   #ssl.certificate_authorities:
   #  - /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt
 
+  # Use Elasticsearch histogram type to store histograms (beta, default: false)
+  # This will change the default layout and put metric type in the field name
+  #use_types: true
+
+  # Store counter rates instead of original cumulative counters (experimental, default: false)
+  #rate_counters: true
 
 # Metrics sent by a Prometheus server using remote_write option
 #- module: prometheus
@@ -1411,6 +1407,18 @@ metricbeat.modules:
   #ssl.certificate: "/etc/pki/server/cert.pem"
   #ssl.key: "/etc/pki/server/cert.key"
 
+  # Use Elasticsearch histogram type to store histograms (beta, default: false)
+  # This will change the default layout and put metric type in the field name
+  #use_types: true
+
+  # Store counter rates instead of original cumulative counters (experimental, default: false)
+  #rate_counters: true
+
+  # Define patterns for counter and histogram types so as to identify metrics' types according to these patterns
+  #types_patterns:
+  #  counter_patterns: []
+  #  histogram_patterns: []
+
 # Metrics that will be collected using a PromQL
 #- module: prometheus
 #  metricsets: ["query"]

x-pack/metricbeat/module/meraki/_meta/config.yml

@@ -0,0 +1,6 @@
+- module: meraki
+  metricsets: ["device_health"]
+  enabled: true
+  period: 300s
+  apiKey: "Meraki dashboard API key"
+  organizations: ["Meraki organization ID"]  

x-pack/metricbeat/module/meraki/_meta/docs.asciidoc

@@ -0,0 +1,2 @@
+This is the meraki module.
+

x-pack/metricbeat/module/meraki/_meta/fields.yml

@@ -0,0 +1,7 @@
+- key: meraki
+  title: Cisco Meraki
+  release: beta
+  fields:
+   - name: meraki
+     type: group
+     fields: