Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add metricbeat benchmark module #41801

Merged
merged 4 commits into from
Dec 4, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.next.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -400,6 +400,7 @@ https://github.com/elastic/beats/compare/v8.8.1\...main[Check the HEAD diff]
- Add support for location label as an optional configuration parameter in GCP metrics metricset. {issue}41550[41550] {pull}41626[41626]

*Metricbeat*
- Add benchmark module {pull}41801[41801]


*Osquerybeat*
Expand Down
31 changes: 31 additions & 0 deletions metricbeat/docs/fields.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ grouped in the following categories:
* <<exported-fields-azure>>
* <<exported-fields-beat-common>>
* <<exported-fields-beat>>
* <<exported-fields-benchmark>>
* <<exported-fields-ceph>>
* <<exported-fields-cloud>>
* <<exported-fields-cloudfoundry>>
Expand Down Expand Up @@ -10472,6 +10473,36 @@ type: long

--

[[exported-fields-benchmark]]
== Benchmark fields

benchmark module



[float]
=== benchmark




[float]
=== info

info



*`benchmark.info.counter`*::
+
--
The nth info metric emitted by the benchmark module


type: keyword
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be a numeric type?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably not. Even though it is numeric "data", it is unlikely we would use range searches, it is much more likely we want fast retrieval. ie you search to make sure that benchmark metric number 13 was actually indexed.

Here is the tip from elasticsearch docs I'm basing this off of.

Mapping numeric identifiers

Not all numeric data should be mapped as a numeric field data type. Elasticsearch optimizes numeric fields, such as integer or long, for range queries. However, keyword fields are better for term and other term-level queries.

Identifiers, such as an ISBN or a product ID, are rarely used in range queries. However, they are often retrieved using term-level queries.

Consider mapping a numeric identifier as a keyword if:

  • You don’t plan to search for the identifier data using range queries.
  • Fast retrieval is important. term query searches on keyword fields are often faster than term searches on numeric fields.

If you’re unsure which to use, you can use a multi-field to map the data as both a keyword and a numeric data type.


--

[[exported-fields-ceph]]
== Ceph fields

Expand Down
76 changes: 76 additions & 0 deletions metricbeat/docs/modules/benchmark.asciidoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
////
This file is generated! See scripts/mage/docs_collector.go
////

:modulename: benchmark
:edit_url: https://github.com/elastic/beats/edit/main/x-pack/metricbeat/module/benchmark/_meta/docs.asciidoc


[[metricbeat-module-benchmark]]
[role="xpack"]
== Benchmark module

beta[]

include::{libbeat-dir}/shared/integration-link.asciidoc[]

:modulename!:

The `benchmark` module is used to generate synthetic metrics at a predictable rate. This can be useful when you want to test output settings or test system sizing without using real data.

The `benchmark` module metricset is `info`.

[source,yaml]
----
- module: benchmark
metricsets:
- info
enabled: true
period: 10s
----

[float]
== Metricsets

[float]
=== `info`
A metric that includes a `counter` field which is used to keep the metric unique.

[float]
=== Module-specific configuration notes

`count`:: number, the number of metrics to emit per fetch.





:edit_url:

[float]
=== Example configuration

The Benchmark module supports the standard configuration options that are described
in <<configuration-metricbeat>>. Here is an example configuration:

[source,yaml]
----
metricbeat.modules:
- module: benchmark
metricsets:
- info
enabled: false
period: 10s

----

[float]
=== Metricsets

The following metricsets are available:

* <<metricbeat-metricset-benchmark-info,info>>

include::benchmark/info.asciidoc[]

:edit_url!:
29 changes: 29 additions & 0 deletions metricbeat/docs/modules/benchmark/info.asciidoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
////
This file is generated! See scripts/mage/docs_collector.go
////
:edit_url: https://github.com/elastic/beats/edit/main/x-pack/metricbeat/module/benchmark/info/_meta/docs.asciidoc


[[metricbeat-metricset-benchmark-info]]
[role="xpack"]
=== Benchmark info metricset

beta[]

include::../../../../x-pack/metricbeat/module/benchmark/info/_meta/docs.asciidoc[]


:edit_url:

==== Fields

For a description of each field in the metricset, see the
<<exported-fields-benchmark,exported fields>> section.

Here is an example document generated by this metricset:

[source,json]
----
include::../../../../x-pack/metricbeat/module/benchmark/info/_meta/data.json[]
----
:edit_url!:
3 changes: 3 additions & 0 deletions metricbeat/docs/modules_list.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,8 @@ This file is generated! See scripts/mage/docs_collector.go
|<<metricbeat-module-beat,Beat>> |image:./images/icon-no.png[No prebuilt dashboards] |
.2+| .2+| |<<metricbeat-metricset-beat-state,state>>
|<<metricbeat-metricset-beat-stats,stats>>
|<<metricbeat-module-benchmark,Benchmark>> beta[] |image:./images/icon-no.png[No prebuilt dashboards] |
.1+| .1+| |<<metricbeat-metricset-benchmark-info,info>> beta[]
|<<metricbeat-module-ceph,Ceph>> |image:./images/icon-yes.png[Prebuilt dashboards are available] |
.13+| .13+| |<<metricbeat-metricset-ceph-cluster_disk,cluster_disk>>
|<<metricbeat-metricset-ceph-cluster_health,cluster_health>>
Expand Down Expand Up @@ -344,6 +346,7 @@ include::modules/aws.asciidoc[]
include::modules/awsfargate.asciidoc[]
include::modules/azure.asciidoc[]
include::modules/beat.asciidoc[]
include::modules/benchmark.asciidoc[]
include::modules/ceph.asciidoc[]
include::modules/cloudfoundry.asciidoc[]
include::modules/cockroachdb.asciidoc[]
Expand Down
2 changes: 2 additions & 0 deletions x-pack/metricbeat/include/list.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

76 changes: 42 additions & 34 deletions x-pack/metricbeat/metricbeat.reference.yml
Original file line number Diff line number Diff line change
Expand Up @@ -404,6 +404,14 @@ metricbeat.modules:
# Monitoring instead of metricbeat-* indices.
#xpack.enabled: false

#------------------------------ Benchmark Module ------------------------------
- module: benchmark
metricsets:
- info
enabled: false
period: 10s


#--------------------------------- Ceph Module ---------------------------------
# Metricsets depending on the Ceph REST API (default port: 5000)
- module: ceph
Expand Down Expand Up @@ -1356,9 +1364,11 @@ metricbeat.modules:
# Password to use when connecting to PostgreSQL. Empty by default.
#password: pass

#----------------------- Prometheus Typed Metrics Module -----------------------
#------------------------------ Prometheus Module ------------------------------
# Metrics collected from a Prometheus endpoint
- module: prometheus
period: 10s
metricsets: ["collector"]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this need to change as part of this PR?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, in that it isn't part of this change. But Yes in that mage config generates it. I don't know why #40411 didn't already pull this change in.

hosts: ["localhost:9090"]
metrics_path: /metrics
#metrics_filters:
Expand All @@ -1367,45 +1377,27 @@ metricbeat.modules:
#username: "user"
#password: "secret"

# Count number of metrics present in Elasticsearch document (default: false)
#metrics_count: false

# This can be used for service account based authorization:
#bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
#ssl.certificate_authorities:
# - /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt

# Count number of metrics present in Elasticsearch document (default: false)
#metrics_count: false

# Use Elasticsearch histogram type to store histograms (beta, default: false)
# This will change the default layout and put metric type in the field name
#use_types: true

# Store counter rates instead of original cumulative counters (experimental, default: false)
#rate_counters: true

# Metrics sent by a Prometheus server using remote_write option
#- module: prometheus
# metricsets: ["remote_write"]
# host: "localhost"
# port: "9201"

# Secure settings for the server using TLS/SSL:
#ssl.certificate: "/etc/pki/server/cert.pem"
#ssl.key: "/etc/pki/server/cert.key"

# Count number of metrics present in Elasticsearch document (default: false)
#metrics_count: false

# Use Elasticsearch histogram type to store histograms (beta, default: false)
# This will change the default layout and put metric type in the field name
#use_types: true

# Store counter rates instead of original cumulative counters (experimental, default: false)
#rate_counters: true

# Define patterns for counter and histogram types so as to identify metrics' types according to these patterns
#types_patterns:
# counter_patterns: []
# histogram_patterns: []
# Secure settings for the server using TLS/SSL:
#ssl.certificate: "/etc/pki/server/cert.pem"
#ssl.key: "/etc/pki/server/cert.key"

# Metrics that will be collected using a PromQL
#- module: prometheus
Expand Down Expand Up @@ -1433,11 +1425,9 @@ metricbeat.modules:
# params:
# query: "some_value"

#------------------------------ Prometheus Module ------------------------------
# Metrics collected from a Prometheus endpoint
#----------------------- Prometheus Typed Metrics Module -----------------------
- module: prometheus
period: 10s
metricsets: ["collector"]
hosts: ["localhost:9090"]
metrics_path: /metrics
#metrics_filters:
Expand All @@ -1446,28 +1436,46 @@ metricbeat.modules:
#username: "user"
#password: "secret"

# Count number of metrics present in Elasticsearch document (default: false)
#metrics_count: false

# This can be used for service account based authorization:
#bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
#ssl.certificate_authorities:
# - /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt

# Count number of metrics present in Elasticsearch document (default: false)
#metrics_count: false

# Use Elasticsearch histogram type to store histograms (beta, default: false)
# This will change the default layout and put metric type in the field name
#use_types: true

# Store counter rates instead of original cumulative counters (experimental, default: false)
#rate_counters: true

# Metrics sent by a Prometheus server using remote_write option
#- module: prometheus
# metricsets: ["remote_write"]
# host: "localhost"
# port: "9201"

# Count number of metrics present in Elasticsearch document (default: false)
#metrics_count: false

# Secure settings for the server using TLS/SSL:
#ssl.certificate: "/etc/pki/server/cert.pem"
#ssl.key: "/etc/pki/server/cert.key"

# Count number of metrics present in Elasticsearch document (default: false)
#metrics_count: false

# Use Elasticsearch histogram type to store histograms (beta, default: false)
# This will change the default layout and put metric type in the field name
#use_types: true

# Store counter rates instead of original cumulative counters (experimental, default: false)
#rate_counters: true

# Define patterns for counter and histogram types so as to identify metrics' types according to these patterns
#types_patterns:
# counter_patterns: []
# histogram_patterns: []

# Metrics that will be collected using a PromQL
#- module: prometheus
# metricsets: ["query"]
Expand Down
6 changes: 6 additions & 0 deletions x-pack/metricbeat/module/benchmark/_meta/config.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
- module: benchmark
metricsets:
- info
enabled: false
period: 10s

31 changes: 31 additions & 0 deletions x-pack/metricbeat/module/benchmark/_meta/docs.asciidoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
include::{libbeat-dir}/shared/integration-link.asciidoc[]

:modulename!:

The `benchmark` module is used to generate synthetic metrics at a predictable rate. This can be useful when you want to test output settings or test system sizing without using real data.

The `benchmark` module metricset is `info`.

[source,yaml]
----
- module: benchmark
metricsets:
- info
enabled: true
period: 10s
----

[float]
== Metricsets

[float]
=== `info`
A metric that includes a `counter` field which is used to keep the metric unique.

[float]
=== Module-specific configuration notes

`count`:: number, the number of metrics to emit per fetch.



10 changes: 10 additions & 0 deletions x-pack/metricbeat/module/benchmark/_meta/fields.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
- key: benchmark
title: "Benchmark"
release: beta
description: >
benchmark module
fields:
- name: benchmark
type: group
description: >
fields:
6 changes: 6 additions & 0 deletions x-pack/metricbeat/module/benchmark/doc.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
// Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
// or more contributor license agreements. Licensed under the Elastic License;
// you may not use this file except in compliance with the Elastic License.

// Package benchmark is a Metricbeat module that contains MetricSets.
package benchmark
Loading
Loading