Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Make elasticsearch/index_recovery metricset work for Stack Monitoring without xpack.enabled flag #20810

Merged
merged 23 commits into from
Dec 1, 2020
Merged

Make elasticsearch/index_recovery metricset work for Stack Monitoring without xpack.enabled flag #20810

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
b1903d4
Merge branch 'master' into feature-stack-monitoring-mb-ecs
sayden Nov 5, 2020
abc4f69
Make update after rebasing and fixing conflicts in fields.yml
sayden Nov 5, 2020
58e2a7d
Make elasticsearch/index_summary metricset work for Stack Monitoring …
sayden Nov 10, 2020
dbaefcd
Force push new atomic commit. 1 shard == 1 event
sayden Nov 4, 2020
aa574dd
Fix test that was failing
sayden Nov 6, 2020
f9e65e2
Add missing field
sayden Nov 10, 2020
5f8560d
Rebase and `make update`
sayden Nov 10, 2020
6c00189
Add comment to test
sayden Nov 10, 2020
1f26899
Add missing fields to mapping
sayden Nov 11, 2020
a1a5209
Merge branch 'feature-stack-monitoring-mb-ecs' into feature/mb/elasti…
sayden Nov 19, 2020
083ecf8
Rebase and update fields
sayden Nov 19, 2020
bb16dc2
Skip broken test
sayden Nov 19, 2020
384faa4
Create TestData test
sayden Nov 19, 2020
6739ec4
Maybe fix lint?
sayden Nov 20, 2020
f164738
Merge branch 'feature-stack-monitoring-mb-ecs' into feature/mb/elasti…
sayden Nov 20, 2020
f198cc0
Fix testdata test
sayden Nov 20, 2020
77c3fa3
Merge branch 'feature-stack-monitoring-mb-ecs' into feature/mb/elasti…
sayden Nov 20, 2020
e9fc22f
Merge branch 'feature-stack-monitoring-mb-ecs' into feature/mb/elasti…
sayden Nov 20, 2020
36332e3
Fix fields.go
sayden Nov 20, 2020
4d110c3
Merge branch 'feature-stack-monitoring-mb-ecs' into feature/mb/elasti…
sayden Nov 20, 2020
d57b430
Merge branch 'feature-stack-monitoring-mb-ecs' into feature/mb/elasti…
sayden Dec 1, 2020
5bb0551
Modify fields.go
sayden Dec 1, 2020
eca8d7b
Fix linting
sayden Dec 1, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
131 changes: 131 additions & 0 deletions metricbeat/docs/fields.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19999,6 +19999,13 @@ index



*`elasticsearch.index.recovery.total_time.ms`*::
+
--
type: long

--

*`elasticsearch.index.recovery.id`*::
+
--
Expand Down Expand Up @@ -20099,6 +20106,130 @@ type: keyword

--

*`elasticsearch.index.recovery.stop_time.ms`*::
+
--
type: long

--

*`elasticsearch.index.recovery.start_time.ms`*::
+
--
type: long

--


*`elasticsearch.index.recovery.index.total_time.ms`*::
+
--
type: long

--

*`elasticsearch.index.recovery.index.throttle_time.source.ms`*::
+
--
type: long

--

*`elasticsearch.index.recovery.index.throttle_time.target.ms`*::
+
--
type: long

--


*`elasticsearch.index.recovery.index.files.count`*::
+
--
type: long

--

*`elasticsearch.index.recovery.index.files.reused`*::
+
--
type: long

--

*`elasticsearch.index.recovery.index.files.recovered`*::
+
--
type: long

--


*`elasticsearch.index.recovery.index.size.recovered.bytes`*::
+
--
type: long

--

*`elasticsearch.index.recovery.index.size.total.bytes`*::
+
--
type: long

--

*`elasticsearch.index.recovery.index.size.reused.bytes`*::
+
--
type: long

--


*`elasticsearch.index.recovery.translog.total_time.ms`*::
+
--
type: long

--

*`elasticsearch.index.recovery.translog.count`*::
+
--
type: long

--

*`elasticsearch.index.recovery.translog.total_on_start`*::
+
--
type: long

--

*`elasticsearch.index.recovery.translog.recovered`*::
+
--
type: long

--


*`elasticsearch.index.recovery.verify_index.check_index_time.ms`*::
+
--
type: long

--

*`elasticsearch.index.recovery.verify_index.total_time.ms`*::
+
--
type: long

--

[float]
=== index.summary

Expand Down
5 changes: 4 additions & 1 deletion metricbeat/module/elasticsearch/elasticsearch_integration_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -129,7 +129,7 @@ func TestXPackEnabled(t *testing.T) {
"ccr": []string{}, // no longer indexed into .monitoring-es-*
"cluster_stats": []string{}, // no longer indexed into .monitoring-es-*
"enrich": []string{}, // no longer indexed into .monitoring-es-*
"index_recovery": []string{"index_recovery"},
"index_recovery": []string{}, // no longer indexed into .monitoring-es-*
"index_summary": []string{}, // no longer indexed into .monitoring-es-*
"ml_job": []string{"job_stats"},
"node_stats": []string{}, // no longer indexed into .monitoring-es-*
Expand Down Expand Up @@ -261,6 +261,9 @@ func getXPackConfig(host string) map[string]interface{} {
"metricsets": xpackMetricSets,
"hosts": []string{host},
"xpack.enabled": true,
// index_recovery.active_only is part of the config of the index_recovery Metricset and it is required during the
// test of that particular metricset to get some data from the ES node (instead of an empty JSON if set to true)
"index_recovery.active_only": false,
sayden marked this conversation as resolved.
Show resolved Hide resolved
}
}

Expand Down
2 changes: 1 addition & 1 deletion metricbeat/module/elasticsearch/fields.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

84 changes: 69 additions & 15 deletions metricbeat/module/elasticsearch/index_recovery/_meta/data.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,38 +1,92 @@
{
"@timestamp": "2017-10-12T08:05:34.853Z",
"agent": {
"hostname": "host.example.com",
"name": "host.example.com"
},
"elasticsearch": {
"cluster": {
"id": "3LbUkLkURz--FR-YO0wLNA",
"name": "es1"
"id": "8l_zoGznQRmtoX9iSC-goA",
"name": "docker-cluster"
},
"index": {
"name": ".monitoring-es-6-2018.11.20",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are removing this field, which would be a breaking change. We should add it back.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bump

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Never mind, I see this has been updated now.

"name": ".kibana-event-log-8.0.0-000001",
"recovery": {
"id": 0,
"index": {
"files": {
"count": 0,
"recovered": 0,
"reused": 0
},
"size": {
"recovered": {
"bytes": 0
},
"reused": {
"bytes": 0
},
"total": {
"bytes": 0
}
},
"throttle_time": {
"source": {
"ms": 0
},
"target": {
"ms": 0
}
},
"total_time": {
"ms": 1600
}
},
"primary": true,
"source": {},
"stage": "DONE",
"start_time": {
"ms": 1605819056123
},
"stop_time": {
"ms": 1605819058696
},
"target": {
"host": "127.0.0.1",
"id": "FMRmkE3HTU6xxxoFK-06Ww",
"name": "es1_1"
"id": "Fkj12lAFQOex0DwK0HMwHw",
"name": "082618b4bb36"
},
"type": "EMPTY_STORE"
"total_time": {
"ms": 2573
},
"translog": {
"count": 0,
"recovered": 0,
"total_on_start": 0,
"total_time": {
"ms": 951
}
},
"type": "EMPTY_STORE",
"verify_index": {
"check_index_time": {
"ms": 0
},
"total_time": {
"ms": 0
}
}
}
}
},
"event": {
"dataset": "elasticsearch.index.recovery",
"duration": 115000,
"module": "elasticsearch"
},
"metricset": {
"host": "127.0.0.1:9200",
"module": "elasticsearch",
"name": "index_recovery",
"namespace": "elasticsearch.index.recovery",
"rtt": 115
"period": 10000
},
"service": {
"name": "elasticsearch"
"address": "127.0.0.1:50581",
"name": "elasticsearch",
"type": "elasticsearch"
}
}
Loading