Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Metricbeat] Migrate Ceph monitor_health to use ReporterV2 interface #11012

Merged
merged 5 commits into from
Apr 3, 2019
Merged

[Metricbeat] Migrate Ceph monitor_health to use ReporterV2 interface #11012

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
da61c26
Atomic commit
sayden Mar 26, 2019
998264a
Update asciidoc
sayden Mar 26, 2019
534ab53
Change extension of example files
sayden Apr 1, 2019
6cdd1c0
Rename test files
sayden Apr 2, 2019
0c3d1ac
Update data.json file
sayden Apr 2, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions metricbeat/module/ceph/monitor_health/_meta/testdata/config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
type: http
url: "/api/v0.1/health"
suffix: plain
1 change: 1 addition & 0 deletions metricbeat/module/ceph/monitor_health/_meta/testdata/input.plain
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"status": "OK", "output": {"detail": [], "timechecks": {"round_status": "finished", "epoch": 3, "round": 0}, "health": {"health_services": [{"mons": [{"last_updated": "2019-03-13 11:21:24.667025", "name": "26c372192772", "avail_percent": 82, "kb_total": 936145620, "kb_avail": 773013080, "health": "HEALTH_OK", "kb_used": 115509168, "store_stats": {"bytes_total": 4195407, "bytes_log": 4128768, "last_updated": "0.000000", "bytes_misc": 65552, "bytes_sst": 1087}}]}]}, "overall_status": "HEALTH_OK", "summary": []}}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is the suffix of this .plain? Looks like json to me.

If you rename this to docs.json and remove the suffix config in config.yml and then run -generate, this should directly als update the data.json file.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sayden I see you went now with input.json instead of docs.json as recommended above to have the data.json generated. Could you elaborate?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought that you mean docs.json like foo.json and that name didn't matter 😅 In any case, input or doc, it won't generate a data.json named file but a docs.json-expected.json, right? Now I'm not sure if I'm missing something else

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes: https://github.com/elastic/beats/blob/master/metricbeat/mb/testing/data/data_test.go#L179

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah! Now I understand, I misunderstood completely at the beginning. Thanks for the clarification :)

48 changes: 48 additions & 0 deletions metricbeat/module/ceph/monitor_health/_meta/testdata/input.plain-expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
[
{
"ceph": {
"monitor_health": {
"available": {
"kb": 773013080,
"pct": 82
},
"health": "HEALTH_OK",
"last_updated": "2019-03-13T11:21:24.667025Z",
"name": "26c372192772",
"store_stats": {
"last_updated": "0.000000",
"log": {
"bytes": 4128768
},
"misc": {
"bytes": 65552
},
"sst": {
"bytes": 1087
},
"total": {
"bytes": 4195407
}
},
"total": {
"kb": 936145620
},
"used": {
"kb": 115509168
}
}
},
"event": {
"dataset": "ceph.monitor_health",
"duration": 115000,
"module": "ceph"
},
"metricset": {
"name": "monitor_health"
},
"service": {
"address": "127.0.0.1:55555",
"type": "ceph"
}
}
]
9 changes: 5 additions & 4 deletions metricbeat/module/ceph/monitor_health/data.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,8 +21,9 @@ import (
"encoding/json"
"time"

"github.com/pkg/errors"

"github.com/elastic/beats/libbeat/common"
"github.com/elastic/beats/libbeat/logp"
)

type Tick struct {
Expand Down Expand Up @@ -85,11 +86,11 @@ type HealthRequest struct {
Output Output `json:"output"`
}

func eventsMapping(content []byte) []common.MapStr {
func eventsMapping(content []byte) ([]common.MapStr, error) {
var d HealthRequest
err := json.Unmarshal(content, &d)
if err != nil {
logp.Err("Error: %+v", err)
return nil, errors.Wrapf(err, "could not get HealthRequest data")
}

events := []common.MapStr{}
Expand Down Expand Up @@ -131,5 +132,5 @@ func eventsMapping(content []byte) []common.MapStr {
}
}

return events
return events, nil
}
19 changes: 15 additions & 4 deletions metricbeat/module/ceph/monitor_health/monitor_health.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,6 @@
package monitor_health

import (
"github.com/elastic/beats/libbeat/common"
"github.com/elastic/beats/metricbeat/helper"
"github.com/elastic/beats/metricbeat/mb"
"github.com/elastic/beats/metricbeat/mb/parse"
Expand Down Expand Up @@ -61,11 +60,23 @@ func New(base mb.BaseMetricSet) (mb.MetricSet, error) {
}, nil
}

func (m *MetricSet) Fetch() ([]common.MapStr, error) {
// Fetch methods implements the data gathering and data conversion to the right
// format. It publishes the event which is then forwarded to the output. In case
// of an error set the Error field of mb.Event or simply call report.Error().
func (m *MetricSet) Fetch(reporter mb.ReporterV2) error {
content, err := m.HTTP.FetchContent()
if err != nil {
return nil, err
return err
}

events, err := eventsMapping(content)
if err != nil {
return err
}

for _, event := range events {
reporter.Event(mb.Event{MetricSetFields: event})
}

return eventsMapping(content), nil
return nil
}
65 changes: 0 additions & 65 deletions metricbeat/module/ceph/monitor_health/monitor_health_integration_test.go
View file
Open in desktop

This file was deleted.

12 changes: 7 additions & 5 deletions metricbeat/module/ceph/monitor_health/monitor_health_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@ import (

func TestFetchEventContents(t *testing.T) {
absPath, err := filepath.Abs("../_meta/testdata/")
assert.NoError(t, err)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍


response, err := ioutil.ReadFile(absPath + "/sample_response.json")
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
Expand All @@ -47,12 +48,13 @@ func TestFetchEventContents(t *testing.T) {
"hosts": []string{server.URL},
}

f := mbtest.NewEventsFetcher(t, config)
events, err := f.Fetch()
if err != nil {
t.Fatal(err)
f := mbtest.NewReportingMetricSetV2Error(t, config)
events, errs := mbtest.ReportingFetchV2Error(f)
if len(errs) > 0 {
t.Fatalf("Expected 0 error, had %d. %v\n", len(errs), errs)
}
event := events[0]
assert.NotEmpty(t, events)
event := events[0].MetricSetFields

t.Logf("%s/%s event: %+v", f.Module().Name(), f.Name(), event.StringToPrint())

Expand Down