[Metricbeat][Istio] Add pilot metricset

CHANGELOG.next.asciidoc

@@ -117,6 +117,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Add a `system/network_summary` metricset {pull}15196[15196]
 - Add mesh metricset for Istio Metricbeat module {pull}15535[15535]
 - Add mixer metricset for Istio Metricbeat module {pull}15696[15696]
+- Add pilot metricset for Istio Metricbeat module {pull}15761[15761]
 - Make the `system/cpu` metricset collect normalized CPU metrics by default. {issue}15618[15618] {pull}15729[15729]
 - Add `key/value` mode for SQL module. {issue}15770[15770] {pull]15845[15845]
 - Add STAN dashboard {pull}15654[15654]

metricbeat/docs/fields.asciidoc

@@ -17138,6 +17138,203 @@ type: long
 
 --
 
+[float]
+=== pilot
+
+Contains statistics related to the Istio pilot service
+
+
+
+*`istio.pilot.xds.count`*::
++
+--
+Count of concurrent xDS client connections for Pilot.
+
+
+type: integer
+
+--
+
+*`istio.pilot.xds.pushes`*::
++
+--
+Count of xDS messages sent, as well as errors building or sending xDS messages for lds, rds, cds and eds.
+
+
+type: integer
+
+--
+
+*`istio.pilot.xds.push.time.sec`*::
++
+--
+Total time in seconds Pilot takes to push lds, rds, cds and eds.
+
+
+type: long
+
+--
+
+*`istio.pilot.xds.eds.instances`*::
++
+--
+Instances for each cluster, as of last push. Zero instances is an error.
+
+
+type: object
+
+--
+
+*`istio.pilot.xds.push.context.errors`*::
++
+--
+Number of errors (timeouts) initiating push context.
+
+
+type: integer
+
+--
+
+*`istio.pilot.xds.internal.errors`*::
++
+--
+Total number of internal XDS errors in pilot.
+
+
+type: integer
+
+--
+
+*`istio.pilot.conflict.listener.inbound`*::
++
+--
+Number of conflicting inbound listeners.
+
+
+type: integer
+
+--
+
+*`istio.pilot.conflict.listener.outbound.http.over.current.tcp`*::
++
+--
+Number of conflicting wildcard http listeners with current wildcard tcp listener.
+
+
+type: integer
+
+--
+
+*`istio.pilot.conflict.listener.outbound.http.over.https`*::
++
+--
+Number of conflicting HTTP listeners with well known HTTPS ports.
+
+
+type: integer
+
+--
+
+*`istio.pilot.conflict.listener.outbound.tcp.over.current.http`*::
++
+--
+Number of conflicting wildcard tcp listeners with current wildcard http listener.
+
+
+type: integer
+
+--
+
+*`istio.pilot.conflict.listener.outbound.tcp.over.current.tcp`*::
++
+--
+Number of conflicting tcp listeners with current tcp listener.
+
+
+type: integer
+
+--
+
+*`istio.pilot.proxy.conv.ms.bucket.*`*::
++
+--
+Time needed by Pilot to push Envoy configurations, histogram buckets in milliseconds.
+
+
+type: object
+
+--
+
+*`istio.pilot.proxy.conv.ms.sum`*::
++
+--
+Time needed by Pilot to push Envoy configurations, histogram sum of times in milliseconds.
+
+
+type: long
+
+--
+
+*`istio.pilot.services`*::
++
+--
+Total services known to pilot.
+
+
+type: integer
+
+--
+
+*`istio.pilot.virt.services`*::
++
+--
+Total virtual services known to pilot.
+
+
+type: integer
+
+--
+
+*`istio.pilot.proxy.conv.ms.count`*::
++
+--
+Time needed by Pilot to push Envoy configurations, histogram count of times.
+
+
+type: long
+
+--
+
+*`istio.pilot.no.ip`*::
++
+--
+Pods not found in the endpoint table, possibly invalid.
+
+
+type: object
+
+--
+
+*`istio.pilot.cluster`*::
++
+--
+The instance FQDN.
+
+
+type: text
+
+--
+
+*`istio.pilot.type`*::
++
+--
+The Envoy proxy configuration type.
+
+
+type: text
+
+--
+
 [[exported-fields-jolokia]]
 == Jolokia fields
 

metricbeat/docs/modules/istio.asciidoc

@@ -11,7 +11,7 @@ beta[]
 This is the Istio module. The Istio module collects metrics from the
 Istio https://istio.io/docs/tasks/observability/metrics/querying-metrics/#about-the-prometheus-add-on[prometheus exporters endpoints].
 
-The default metricsets are `mesh`, `mixer`.
+The default metricsets are `mesh`, `mixer`, `pilot`.
 
 [float]
 === Compatibility
@@ -41,6 +41,13 @@ metricbeat.modules:
  period: 10s
  # use istio-telemetry.istio-system:15014, when deploying Metricbeat in a kuberentes cluster as Pod or Daemonset
  hosts: ["localhost:15014"]
+
+# Istio pilot. To collect all Pilot-generated metrics.
+- module: istio
+ metricsets: ["pilot"]
+ period: 10s
+ # use istio-pilot.istio-system:15014, when deploying Metricbeat in a kuberentes cluster as Pod or Daemonset
+ hosts: ["localhost:15014"]
 ----
 
 [float]
@@ -52,7 +59,11 @@ The following metricsets are available:
 
 * <<metricbeat-metricset-istio-mixer,mixer>>
 
+* <<metricbeat-metricset-istio-pilot,pilot>>
+
 include::istio/mesh.asciidoc[]
 
 include::istio/mixer.asciidoc[]
 
+include::istio/pilot.asciidoc[]
+

metricbeat/docs/modules/istio/pilot.asciidoc

@@ -0,0 +1,23 @@
+////
+This file is generated! See scripts/mage/docs_collector.go
+////
+
+[[metricbeat-metricset-istio-pilot]]
+=== istio pilot metricset
+
+beta[]
+
+include::../../../../x-pack/metricbeat/module/istio/pilot/_meta/docs.asciidoc[]
+
+
+==== Fields
+
+For a description of each field in the metricset, see the
+<<exported-fields-istio,exported fields>> section.
+
+Here is an example document generated by this metricset:
+
+[source,json]
+----
+include::../../../../x-pack/metricbeat/module/istio/pilot/_meta/data.json[]
+----

metricbeat/docs/modules_list.asciidoc

@@ -104,8 +104,9 @@ This file is generated! See scripts/mage/docs_collector.go
 |<<metricbeat-module-ibmmq,IBM MQ>> beta[] |image:./images/icon-yes.png[Prebuilt dashboards are available] | 
 .1+| .1+| |<<metricbeat-metricset-ibmmq-qmgr,qmgr>> beta[] 
 |<<metricbeat-module-istio,istio>> beta[] |image:./images/icon-no.png[No prebuilt dashboards] | 
-.2+| .2+| |<<metricbeat-metricset-istio-mesh,mesh>> beta[] 
+.3+| .3+| |<<metricbeat-metricset-istio-mesh,mesh>> beta[] 
 |<<metricbeat-metricset-istio-mixer,mixer>> beta[] 
+|<<metricbeat-metricset-istio-pilot,pilot>> beta[] 
 |<<metricbeat-module-jolokia,Jolokia>> |image:./images/icon-no.png[No prebuilt dashboards] | 
 .1+| .1+| |<<metricbeat-metricset-jolokia-jmx,jmx>> 
 |<<metricbeat-module-kafka,Kafka>> |image:./images/icon-yes.png[Prebuilt dashboards are available] | 

x-pack/metricbeat/metricbeat.reference.yml

@@ -552,6 +552,13 @@ metricbeat.modules:
  # use istio-telemetry.istio-system:15014, when deploying Metricbeat in a kuberentes cluster as Pod or Daemonset
  hosts: ["localhost:15014"]
 
+# Istio pilot. To collect all Pilot-generated metrics.
+- module: istio
+ metricsets: ["pilot"]
+ period: 10s
+ # use istio-pilot.istio-system:15014, when deploying Metricbeat in a kuberentes cluster as Pod or Daemonset
+ hosts: ["localhost:15014"]
+
 #------------------------------- Jolokia Module -------------------------------
 - module: jolokia
  #metricsets: ["jmx"]

x-pack/metricbeat/module/istio/_meta/config.reference.yml

@@ -11,3 +11,10 @@
  period: 10s
  # use istio-telemetry.istio-system:15014, when deploying Metricbeat in a kuberentes cluster as Pod or Daemonset
  hosts: ["localhost:15014"]
+
+# Istio pilot. To collect all Pilot-generated metrics.
+- module: istio
+ metricsets: ["pilot"]
+ period: 10s
+ # use istio-pilot.istio-system:15014, when deploying Metricbeat in a kuberentes cluster as Pod or Daemonset
+ hosts: ["localhost:15014"]

x-pack/metricbeat/module/istio/_meta/config.yml

@@ -11,3 +11,10 @@
  period: 10s
  # use istio-telemetry.istio-system:15014, when deploying Metricbeat in a kuberentes cluster
  hosts: ["localhost:15014"]
+
+# Istio pilot. To collect all Pilot-generated metrics.
+- module: istio
+ metricsets: ["pilot"]
+ period: 10s
+ # use istio-pilot.istio-system:15014, when deploying Metricbeat in a kuberentes cluster
+ hosts: ["localhost:15014"]

x-pack/metricbeat/module/istio/_meta/docs.asciidoc

@@ -1,7 +1,7 @@
 This is the Istio module. The Istio module collects metrics from the
 Istio https://istio.io/docs/tasks/observability/metrics/querying-metrics/#about-the-prometheus-add-on[prometheus exporters endpoints].
 
-The default metricsets are `mesh`, `mixer`.
+The default metricsets are `mesh`, `mixer`, `pilot`.
 
 [float]
 === Compatibility

x-pack/metricbeat/module/istio/pilot/_meta/data.json

@@ -0,0 +1,26 @@
+{
+ "@timestamp": "2019-03-01T08:05:34.853Z",
+ "event": {
+ "dataset": "istio.pilot",
+ "duration": 115000,
+ "module": "istio"
+ },
+ "istio": {
+ "pilot": {
+ "cluster": "outbound_.15031_._.istio-ingressgateway.istio-system.svc.cluster.local",
+ "xds": {
+ "eds": {
+ "instances": 1
+ }
+ }
+ }
+ },
+ "metricset": {
+ "name": "pilot",
+ "period": 10000
+ },
+ "service": {
+ "address": "127.0.0.1:55555",
+ "type": "istio"
+ }
+}

x-pack/metricbeat/module/istio/pilot/_meta/docs.asciidoc

@@ -0,0 +1 @@
+This is the pilot metricset of the module istio. This metricset collects all Pilot-generated metrics.