Add ml_job metricset to Elasticsearch module #7196

ruflin · 2018-05-29T12:46:01Z

This adds a basic ml_job metricset to the Elasticsearch module.

The testing was enhanced to enabled to also test non x-pack Basic features by enabling the trial license as part of the test run.

houndci-bot · 2018-05-29T12:46:15Z

metricbeat/module/elasticsearch/ml_job/ml.go

+	jobPath = "/_xpack/ml/anomaly_detectors/_all/_stats"
+)
+
+type MetricSet struct {


exported type MetricSet should have comment or be unexported

houndci-bot · 2018-05-29T12:46:15Z

metricbeat/module/elasticsearch/ml_job/ml.go

@@ -0,0 +1,46 @@
+package ml_job


don't use an underscore in package name

houndci-bot · 2018-05-29T12:46:15Z

metricbeat/module/elasticsearch/ml_job/data.go

@@ -0,0 +1,50 @@
+package ml_job


don't use an underscore in package name

ruflin · 2018-05-31T13:11:28Z

This PR depends on the release of ES 6.3.

jsoriano · 2018-07-05T14:52:44Z

metricbeat/module/elasticsearch/elasticsearch_integration_test.go

+		"time_field":"timestamp",
+		"time_format": "epoch_ms"
+		}
+	}`


Indentation? And/or move this to a file?

I will use the file that I already created for the system tests ...

jsoriano · 2018-07-05T14:53:25Z

metricbeat/module/elasticsearch/elasticsearch_integration_test.go

@@ -48,7 +52,14 @@ var metricSets = []string{
 func TestFetch(t *testing.T) {
 	compose.EnsureUp(t, "elasticsearch")

-	err := createIndex(getEnvHost() + ":" + getEnvPort())
+	host := getEnvHost() + ":" + getEnvPort()


net.JoinHostPort()?

jsoriano · 2018-07-05T15:00:52Z

metricbeat/module/elasticsearch/ml_job/data.go

+		r.Error(err)
+		return []error{err}
+	}
+


Return error also if jobsData.Jobs is nil?

I would say in case it's empty no error should be returned.

jsoriano · 2018-07-05T15:03:42Z

metricbeat/module/elasticsearch/test_elasticsearch.py

+        try:
+            es.transport.perform_request('POST', "/_xpack/license/start_trial?acknowledge=true")
+        except:
+            print "Trial already enabled"


We could print the error instead, in case it is caused by other reason.

jsoriano · 2018-07-05T15:05:32Z

metricbeat/module/elasticsearch/test_elasticsearch.py

+        if response["count"] > 0:
+            return
+
+        file = os.path.join(self.beat_path, "module", "elasticsearch", "ml_job", "_meta", "test", "test_job.json")


Could we use this file also for the integration test?

jsoriano · 2018-07-05T15:07:55Z

metricbeat/module/elasticsearch/ml_job/_meta/test/xpack.job.stats.json

+                "total_partition_field_count": 0,
+                "bucket_allocation_failures_count": 0,
+                "memory_status": "ok",
+                "log_time": 1525847933184


Are we going to collect more metrics?

Yes. This PR mainly creates the foundation.

ycombinator · 2018-07-05T15:13:32Z

metricbeat/module/elasticsearch/ml_job/ml.go

+
+	// Not master, no event sent
+	if !isMaster {
+		logp.Debug("elasticsearch", "Trying to fetch index summary stats from a none master node.")


Spelling: "none master" should be "non-master"

Also, "index summary stats" should probably be "machine learning job stats".

Both fixed.

ycombinator · 2018-07-05T16:48:33Z

metricbeat/module/elasticsearch/ml_job/ml.go

+
+	content, err := m.HTTP.FetchContent()
+	if err != nil {
+		r.Error(err)


When X-Pack isn't installed in ES, this if condition becomes true because a 400 error code is returned by the call to the /_xpack/ml/anomaly_detectors/_all/_stats API. This then results in error documents being indexed in the metricbeat index that look like this:

{ "@timestamp":"2018-07-05T16:35:40.258Z", "metricset":{ "rtt":6264, "namespace":"elasticsearch.ml.job", "name":"ml_job", "module":"elasticsearch", "host":"localhost:9200" }, "error":{ "message":"HTTP error 400 in ml_job: 400 Bad Request" }, "beat":{ "version":"7.0.0-alpha1", "name":"Shaunaks-MBP-2", "hostname":"Shaunaks-MBP-2" }, "host":{ "name":"Shaunaks-MBP-2" } }

Just want to confirm that this is the behavior we want? Another option could be to log an error (but not index anything) if this API returns 400?

[EDIT] Obviously, whatever approach we take here should be consistent with all X-Pack APIs/metricsets.

Alternatively, could we somehow make use of the xpack config setting here?

If someone enables the ml_job metricset and X-Pack is not enabled I think an error document should be returned. What we probably should improve is the error message above and have something inside that it is not enabled.

I think our xpack config is not related to this.

ycombinator · 2018-07-05T19:43:18Z

metricbeat/module/elasticsearch/ml_job/data.go

+		"state": c.Str("state"),
+		"data_counts": c.Dict("data_counts", s.Schema{
+			"processed_record_count": c.Int("processed_record_count"),
+		}),


Might be worth including invalid_date_count as well. From https://www.elastic.co/guide/en/elasticsearch/reference/6.2/ml-jobstats.html#ml-datacounts:

invalid_date_count
(long) The number of records with either a missing date field or a date that could not be parsed.

Added. In general I think we will have to add more fields but I also would like to involve the ML team here in a follow up. I don't think we should send all the stats but only the ones that are most relevant which is a tricky question to answer.

ycombinator · 2018-07-05T19:45:21Z

metricbeat/module/elasticsearch/ml_job/_meta/docs.asciidoc

@@ -0,0 +1,3 @@
+=== elasticsearch ml MetricSet


Should this say ml_job instead of ml?

ycombinator · 2018-07-05T19:47:50Z

metricbeat/module/elasticsearch/elasticsearch_integration_test.go

+}
+
+// createIndex creates and elasticsearch index in case it does not exit yet
+func enableLicense(host string) error {


Nit: Name this enableTrialLicense just to be more specific about what it actually does?

ycombinator · 2018-07-05T19:54:29Z

metricbeat/module/elasticsearch/ml_job/data.go

+	Jobs []map[string]interface{} `json:"jobs"`
+}
+
+func eventsMapping(r mb.ReporterV2, content []byte) []error {


It seems like the return value is being ignored anyway so my following comment my be moot:

Looking at other metricsets in the elasticsearch module, it looks like their eventsMapping functions return error instead of []error. Further, to build this error return object they use var errs multierror.Errors to collect multiple errors and then return errs.Err() at the end of the function. For consistency maybe the eventsMapping function here should do the same?

Yes, need to change it. This changed recently with the schema change from Jaime. Will adapt it accordingly.

This adds a basic ml_job metricset to the Elasticsearch module. The testing was enhanced to enabled to also test non x-pack Basic features by enabling the trial license as part of the test run.

ycombinator · 2018-07-10T15:16:25Z

metricbeat/module/elasticsearch/elasticsearch_integration_test.go

@@ -191,7 +180,7 @@ func createMLJob(host string) error {
 		return nil
 	}

-	req, err := http.NewRequest("PUT", "http://"+host+jobURL, strings.NewReader(mlJob))
+	req, err := http.NewRequest("PUT", "http://"+host+jobURL, bytes.NewReader(mlJob))


This is more of question for my learning, not a review suggestion: why is bytes.NewReader more appropriate here than strings.NewReader?

The main difference is that one takes a []byte array and the other on a string as param. ReadFile above returns a []byte array and previously it was a string (mlJob var).

ycombinator

LGTM.

(I asked a question but it's incidental to the review).

ruflin added in progress Pull request is currently in progress. module Metricbeat Metricbeat labels May 29, 2018

houndci-bot reviewed May 29, 2018

View reviewed changes

ruflin mentioned this pull request May 29, 2018

Monitor the Elastic Stack with Metricbeat #7035

Closed

39 tasks

ruflin force-pushed the ml-stats branch from d3d567c to ba12460 Compare June 15, 2018 08:29

ruflin force-pushed the ml-stats branch 2 times, most recently from cb784cc to b3bb590 Compare June 26, 2018 12:15

ruflin force-pushed the ml-stats branch from b3bb590 to ea896b6 Compare July 5, 2018 08:21

ruflin added review and removed in progress Pull request is currently in progress. labels Jul 5, 2018

ruflin force-pushed the ml-stats branch from 5fb54af to a469a13 Compare July 5, 2018 11:00

jsoriano reviewed Jul 5, 2018

View reviewed changes

ycombinator reviewed Jul 5, 2018

View reviewed changes

ruflin added 4 commits July 10, 2018 10:10

Add ml_job metricset to Elasticsearch module

017f335

This adds a basic ml_job metricset to the Elasticsearch module. The testing was enhanced to enabled to also test non x-pack Basic features by enabling the trial license as part of the test run.

refactoring

d5cb2eb

apply review

d4edb11

rebase on master

fe5307d

ruflin force-pushed the ml-stats branch from 64701ca to fe5307d Compare July 10, 2018 08:12

rename function

d704c13

ycombinator reviewed Jul 10, 2018

View reviewed changes

ycombinator approved these changes Jul 10, 2018

View reviewed changes

jsoriano merged commit 73751e0 into elastic:master Jul 11, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add ml_job metricset to Elasticsearch module #7196

Add ml_job metricset to Elasticsearch module #7196

ruflin commented May 29, 2018 •

edited

Loading

houndci-bot May 29, 2018

houndci-bot May 29, 2018

houndci-bot May 29, 2018

ruflin commented May 31, 2018

jsoriano Jul 5, 2018

ruflin Jul 9, 2018

jsoriano Jul 5, 2018

jsoriano Jul 5, 2018

ruflin Jul 10, 2018

jsoriano Jul 5, 2018

ruflin Jul 10, 2018

jsoriano Jul 5, 2018

ruflin Jul 10, 2018

jsoriano Jul 5, 2018

ruflin Jul 9, 2018

ycombinator Jul 5, 2018

ycombinator Jul 5, 2018

ruflin Jul 10, 2018

ycombinator Jul 5, 2018 •

edited

Loading

ycombinator Jul 5, 2018

ruflin Jul 6, 2018

ycombinator Jul 5, 2018

ruflin Jul 10, 2018

ycombinator Jul 5, 2018

ruflin Jul 10, 2018

ycombinator Jul 5, 2018

ycombinator Jul 5, 2018 •

edited

Loading

ruflin Jul 6, 2018

ycombinator Jul 10, 2018

ruflin Jul 11, 2018

ycombinator left a comment •

edited

Loading

Add ml_job metricset to Elasticsearch module #7196

Add ml_job metricset to Elasticsearch module #7196

Conversation

ruflin commented May 29, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ruflin commented May 31, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ycombinator Jul 5, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ycombinator Jul 5, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ycombinator left a comment • edited Loading

Choose a reason for hiding this comment

ruflin commented May 29, 2018 •

edited

Loading

ycombinator Jul 5, 2018 •

edited

Loading

ycombinator Jul 5, 2018 •

edited

Loading

ycombinator left a comment •

edited

Loading