Skip to content

Commit

Permalink
[vSphere] Add new Cluster metricset (#40536)
Browse files Browse the repository at this point in the history
* add cluster metricset

* update changelog

* fix lint errors

* minor changes

* add unit test

* update fields.yml

* mage update

* minor changes

* minor changes

* mage update

* Address review comment

* Address review comments

* address review comments

* update data.json

* update fields.yml

* update data.json

* lint changes

* mage update

* mage update for x-pack

* Mage update after conflicts

* update fields.go

* Address review comments

---------

Co-authored-by: Niraj Rathod <niraj.rathod@crestdatasys.com>
Co-authored-by: subham sarkar <subham.sarkar@elastic.co>
  • Loading branch information
3 people authored and ishleenk17 committed Sep 14, 2024
1 parent 7c12246 commit db72602
Show file tree
Hide file tree
Showing 26 changed files with 788 additions and 116 deletions.
34 changes: 34 additions & 0 deletions CHANGELOG.next.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -183,6 +183,40 @@ https://github.com/elastic/beats/compare/v8.8.1\...main[Check the HEAD diff]
- Parse more fields from Elasticsearch slowlogs {pull}38295[38295]
- added benchmark input {pull}37437[37437]
- added benchmark input and discard output {pull}37437[37437]
- Ensure all responses sent by HTTP Endpoint are HTML-escaped. {pull}39329[39329]
- Update CEL mito extensions to v1.11.0 to improve type checking. {pull}39460[39460]
- Improve logging of request and response with request trace logging in error conditions. {pull}39455[39455]
- Implement Elastic Agent status and health reporting for CEL Filebeat input. {pull}39209[39209]
- Add HTTP metrics to CEL input. {issue}39501[39501] {pull}39503[39503]
- Add default user-agent to CEL HTTP requests. {issue}39502[39502] {pull}39587[39587]
- Improve reindexing support in security module pipelines. {issue}38224[38224] {pull}39588[39588]
- Make HTTP Endpoint input GA. {issue}38979[38979] {pull}39410[39410]
- Update CEL mito extensions to v1.12.2. {pull}39755[39755]
- Add support for base64-encoded HMAC headers to HTTP Endpoint. {pull}39655[39655]
- Add user group membership support to Okta entity analytics provider. {issue}39814[39814] {pull}39815[39815]
- Add request trace support for Okta and EntraID entity analytics providers. {pull}39821[39821]
- Fix handling of infinite rate values in CEL rate limit handling logic. {pull}39940[39940]
- Allow elision of set and append failure logging. {issue}34544[34544] {pull}39929[39929]
- Add ability to remove request trace logs from CEL input. {pull}39969[39969]
- Add ability to remove request trace logs from HTTPJSON input. {pull}40003[40003]
- Update CEL mito extensions to v1.13.0. {pull}40035[40035]
- Add Jamf entity analytics provider. {pull}39996[39996]
- Add ability to remove request trace logs from http_endpoint input. {pull}40005[40005]
- Add ability to remove request trace logs from entityanalytics input. {pull}40004[40004]
- Relax constraint on Base DN in entity analytics Active Directory provider. {pull}40054[40054]
- Implement Elastic Agent status and health reporting for Netflow Filebeat input. {pull}40080[40080]
- Enhance input state reporting for CEL evaluations that return a single error object in events. {pull}40083[40083]
- Allow absent credentials when using GCS with Application Default Credentials. {issue}39977[39977] {pull}40072[40072]
- Add SSL and username support for Redis input, now the input includes support for Redis 6.0+. {pull}40111[40111]
- Add scaling up support for Netflow input. {issue}37761[37761] {pull}40122[40122]
- Update CEL mito extensions to v1.15.0. {pull}40294[40294]
- Allow cross-region bucket configuration in s3 input. {issue}22161[22161] {pull}40309[40309]
- Improve logging in Okta Entity Analytics provider. {issue}40106[40106] {pull}40347[40347]
- Document `winlog` input. {issue}40074[40074] {pull}40462[40462]
- Added retry logic to websocket connections in the streaming input. {issue}40271[40271] {pull}40601[40601]
- Add new metricset cluster for the vSphere module. {pull}40536[40536]
- Disable event normalization for netflow input {pull}40635[40635]
- Allow attribute selection in the Active Directory entity analytics provider. {issue}40482[40482] {pull}40662[40662]

*Auditbeat*

Expand Down
101 changes: 101 additions & 0 deletions metricbeat/docs/fields.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -66854,6 +66854,107 @@ vSphere module



[float]
=== cluster

Cluster information.




*`vsphere.cluster.datastore.names`*::
+
--
List of all the Datastore names associated with the cluster.


type: keyword

--

*`vsphere.cluster.datastore.count`*::
+
--
Number of Datastores associated with the cluster.


type: long

--


*`vsphere.cluster.das_config.admission.control.enabled`*::
+
--
Indicates whether strict admission control is enabled.


type: boolean

--

*`vsphere.cluster.das_config.enabled`*::
+
--
Indicates whether vSphere HA feature is enabled.


type: boolean

--


*`vsphere.cluster.host.count`*::
+
--
Number of Hosts associated with the cluster.


type: long

--

*`vsphere.cluster.host.names`*::
+
--
List of all the Host names associated with the cluster.


type: keyword

--

*`vsphere.cluster.name`*::
+
--
Cluster name.


type: keyword

--


*`vsphere.cluster.network.count`*::
+
--
Number of Networks associated with the cluster.


type: long

--

*`vsphere.cluster.network.names`*::
+
--
List of all the Network names associated with the cluster.


type: keyword

--

[float]
=== datastore

Expand Down
8 changes: 6 additions & 2 deletions metricbeat/docs/modules/vsphere.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ This file is generated! See scripts/mage/docs_collector.go

The vSphere module uses the https://github.com/vmware/govmomi[Govmomi] library to collect metrics from any Vmware SDK URL (ESXi/VCenter). This library is built for and tested against ESXi and vCenter 5.5, 6.0 and 6.5.

By default it enables the metricsets `network`, `resourcepool`, `datastore`, `host` and `virtualmachine`.
By default it enables the metricsets `cluster`, `network`, `resourcepool`, `datastore`, `host` and `virtualmachine`.

[float]
=== Dashboard
Expand All @@ -35,7 +35,7 @@ in <<configuration-metricbeat>>. Here is an example configuration:
metricbeat.modules:
- module: vsphere
enabled: true
metricsets: ["datastore", "host", "virtualmachine", "network", "resourcepool"]
metricsets: ["cluster", "datastore", "host", "virtualmachine", "network", "resourcepool"]
# Real-time data collection – An ESXi Server collects data for each performance counter every 20 seconds.
period: 20s
hosts: ["https://localhost/sdk"]
Expand All @@ -53,6 +53,8 @@ metricbeat.modules:

The following metricsets are available:

* <<metricbeat-metricset-vsphere-cluster,cluster>>

* <<metricbeat-metricset-vsphere-datastore,datastore>>

* <<metricbeat-metricset-vsphere-host,host>>
Expand All @@ -63,6 +65,8 @@ The following metricsets are available:

* <<metricbeat-metricset-vsphere-virtualmachine,virtualmachine>>

include::vsphere/cluster.asciidoc[]

include::vsphere/datastore.asciidoc[]

include::vsphere/host.asciidoc[]
Expand Down
29 changes: 29 additions & 0 deletions metricbeat/docs/modules/vsphere/cluster.asciidoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
////
This file is generated! See scripts/mage/docs_collector.go
////
:edit_url: https://github.com/elastic/beats/edit/main/metricbeat/module/vsphere/cluster/_meta/docs.asciidoc


[[metricbeat-metricset-vsphere-cluster]]
=== vSphere cluster metricset

beta[]

include::../../../module/vsphere/cluster/_meta/docs.asciidoc[]

This is a default metricset. If the host module is unconfigured, this metricset is enabled by default.

:edit_url:

==== Fields

For a description of each field in the metricset, see the
<<exported-fields-vsphere,exported fields>> section.

Here is an example document generated by this metricset:

[source,json]
----
include::../../../module/vsphere/cluster/_meta/data.json[]
----
:edit_url!:
3 changes: 2 additions & 1 deletion metricbeat/docs/modules_list.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -311,7 +311,8 @@ This file is generated! See scripts/mage/docs_collector.go
|<<metricbeat-module-uwsgi,uWSGI>> |image:./images/icon-yes.png[Prebuilt dashboards are available] |
.1+| .1+| |<<metricbeat-metricset-uwsgi-status,status>>
|<<metricbeat-module-vsphere,vSphere>> |image:./images/icon-yes.png[Prebuilt dashboards are available] |
.5+| .5+| |<<metricbeat-metricset-vsphere-datastore,datastore>>
.6+| .6+| |<<metricbeat-metricset-vsphere-cluster,cluster>> beta[]
|<<metricbeat-metricset-vsphere-datastore,datastore>>
|<<metricbeat-metricset-vsphere-host,host>>
|<<metricbeat-metricset-vsphere-network,network>> beta[]
|<<metricbeat-metricset-vsphere-resourcepool,resourcepool>> beta[]
Expand Down
1 change: 1 addition & 0 deletions metricbeat/include/list_common.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion metricbeat/metricbeat.reference.yml
Original file line number Diff line number Diff line change
Expand Up @@ -1007,7 +1007,7 @@ metricbeat.modules:
#------------------------------- VSphere Module -------------------------------
- module: vsphere
enabled: true
metricsets: ["datastore", "host", "virtualmachine", "network", "resourcepool"]
metricsets: ["cluster", "datastore", "host", "virtualmachine", "network", "resourcepool"]
# Real-time data collection – An ESXi Server collects data for each performance counter every 20 seconds.
period: 20s
hosts: ["https://localhost/sdk"]
Expand Down
1 change: 1 addition & 0 deletions metricbeat/module/vsphere/_meta/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ Now setup your metricbeat config to connect to Govcsim:
```
- module: vsphere
metricsets:
- cluster
- datastore
- host
- virtualmachine
Expand Down
2 changes: 1 addition & 1 deletion metricbeat/module/vsphere/_meta/config.reference.yml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
- module: vsphere
enabled: true
metricsets: ["datastore", "host", "virtualmachine", "network", "resourcepool"]
metricsets: ["cluster", "datastore", "host", "virtualmachine", "network", "resourcepool"]
# Real-time data collection – An ESXi Server collects data for each performance counter every 20 seconds.
period: 20s
hosts: ["https://localhost/sdk"]
Expand Down
1 change: 1 addition & 0 deletions metricbeat/module/vsphere/_meta/config.yml
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
- module: vsphere
#metricsets:
# - cluster
# - datastore
# - host
# - virtualmachine
Expand Down
2 changes: 1 addition & 1 deletion metricbeat/module/vsphere/_meta/docs.asciidoc
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
The vSphere module uses the https://github.com/vmware/govmomi[Govmomi] library to collect metrics from any Vmware SDK URL (ESXi/VCenter). This library is built for and tested against ESXi and vCenter 5.5, 6.0 and 6.5.

By default it enables the metricsets `network`, `resourcepool`, `datastore`, `host` and `virtualmachine`.
By default it enables the metricsets `cluster`, `network`, `resourcepool`, `datastore`, `host` and `virtualmachine`.

[float]
=== Dashboard
Expand Down
47 changes: 47 additions & 0 deletions metricbeat/module/vsphere/cluster/_meta/data.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
{
"@timestamp": "2017-10-12T08:05:34.853Z",
"event": {
"dataset": "vsphere.cluster",
"duration": 115000,
"module": "vsphere"
},
"metricset": {
"name": "cluster",
"period": 10000
},
"service": {
"address": "127.0.0.1:33365",
"type": "vsphere"
},
"vsphere": {
"cluster": {
"name": "Cluster_1",
"das_config": {
"enabled": false,
"admission": {
"control": {
"enabled": true
}
}
},
"host": {
"count": 1,
"names": [
"Host_1"
]
},
"datastore": {
"count": 1,
"names": [
"Datastore_1"
]
},
"network": {
"count": 1,
"names": [
"Network_1"
]
}
}
}
}
1 change: 1 addition & 0 deletions metricbeat/module/vsphere/cluster/_meta/docs.asciidoc
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
This is the `cluster` metricset of the vSphere module.
54 changes: 54 additions & 0 deletions metricbeat/module/vsphere/cluster/_meta/fields.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
- name: cluster
type: group
release: beta
description: >
Cluster information.
fields:
- name: datastore
type: group
fields:
- name: names
type: keyword
description: >
List of all the Datastore names associated with the cluster.
- name: count
type: long
description: >
Number of Datastores associated with the cluster.
- name: das_config
type: group
fields:
- name: admission.control.enabled
type: boolean
description: >
Indicates whether strict admission control is enabled.
- name: enabled
type: boolean
description: >
Indicates whether vSphere HA feature is enabled.
- name: host
type: group
fields:
- name: count
type: long
description: >
Number of Hosts associated with the cluster.
- name: names
type: keyword
description: >
List of all the Host names associated with the cluster.
- name: name
type: keyword
description: >
Cluster name.
- name: network
type: group
fields:
- name: count
type: long
description: >
Number of Networks associated with the cluster.
- name: names
type: keyword
description: >
List of all the Network names associated with the cluster.
Loading

0 comments on commit db72602

Please sign in to comment.