New module windows for metricbeat #3758

martinscholz83 · 2017-03-16T08:03:04Z

This PR implements a new module called windows for metricbeat. The first metricset within this module is perfmonbeat which is able to collect data from windows performance counters.

elasticmachine · 2017-03-16T08:07:02Z

Jenkins standing by to test this. If you aren't a maintainer, you can ignore this comment. Someone with commit access, please review this and clear it for Jenkins to run.

elasticmachine · 2017-03-16T08:07:06Z

Jenkins standing by to test this. If you aren't a maintainer, you can ignore this comment. Someone with commit access, please review this and clear it for Jenkins to run.

martinscholz83 · 2017-03-16T09:05:31Z

Is there some flag which says this module is only for windows?

andrewkroh

Thanks for opening up a PR for this. 👍

I left some minor feedback about error handling. After you address that I'll probably take the code for a test drive.

Could you please add some tests (like a pdh_windows_test.go file) that exercises the code by querying some counters. And maybe a negative test case where a counter does not exist.

It looks like you accidentally committed a copy of libbeat/dashboards/import_dashboards. Please remove that.

andrewkroh · 2017-03-16T21:44:09Z

metricbeat/module/windows/perfmon/pdh.go

@@ -0,0 +1,83 @@
+package perfmon


You can use // +build windows to make files like this compile only for Windows. Or you can rename them to include a _windows.go suffix. This behavior is described here: https://golang.org/pkg/go/build/

andrewkroh · 2017-03-16T22:04:18Z

metricbeat/module/windows/perfmon/perfmon.go

+	}
+
+	for _, v := range config.CounterConfig {
+		if len(v.Name) <= 0 {


ucfg can apply this sort of validation automatically if you annotate a the fields with validation info. (see docs) For example,

type CounterConfig struct { Name string `config:"group" validate:"required"` Group []CounterConfigGroup `config:"collectors" validate:"required"` }

andrewkroh · 2017-03-16T22:06:09Z

metricbeat/module/windows/perfmon/perfmon.go

+
+	if err != 0 {
+		logp.Err("%v", err)
+		return nil, errors.New("Initialization fails with error: " + strconv.Itoa(err))


It's not a best practice to log errors and return them. I would just return the error and let the caller decide if it is supposed to be logged or not. Please review the whole PR and make the appropriate changes.

andrewkroh · 2017-03-16T22:17:31Z

metricbeat/module/windows/perfmon/pdh.go

+
+//go:generate go run $GOROOT/src/syscall/mksyscall_windows.go -output pdh_windows.go pdh.go
+// Windows API calls
+//sys   _PdhOpenQuery(dataSource uintptr, userData uintptr, query *uintptr) (err int) = pdh.PdhOpenQuery


I recommend changing the method signature of all these functions to return (err error). The newly generated code will then check the int return value and return the appropriate error. This make all the code that uses theses function conform to the standard Go conventions.

I have change the signature to error. Now it generate this function

func _PdhOpenQuery(dataSource uintptr, userData uintptr, query *uintptr) (err error) { r1, _, e1 := syscall.Syscall(procPdhOpenQuery.Addr(), 3, uintptr(dataSource), uintptr(userData), uintptr(unsafe.Pointer(query))) if r1 == 0 { if e1 != 0 { err = error(e1) } else { err = syscall.EINVAL } } return }

Now i always get Invalid argument error. How should i handle this error. Is it not better to return nil?

andrewkroh · 2017-03-16T22:25:01Z

metricbeat/module/windows/perfmon/perfmon.go

+
+	for _, v := range config.CounterConfig {
+		if len(v.Name) <= 0 {
+			err := errors.New("Group cannot be empty")


Error strings should not be capitalized. explanation

andrewkroh · 2017-03-16T22:35:23Z

metricbeat/module/windows/_meta/config.yml

+    - group: "processor"  
+      collectors:
+        - alias: "processor_perfomance"
+          query: "\\Processor Information(_Total)\\% Processor Performance"


If you use single quotes here, can the double backslashes be replaced with a single backslash? I would prefer that, unless of course using double backslashes is common in the perfmon world.

ruflin · 2017-03-17T10:02:06Z

metricbeat/module/windows/_meta/config.yml

+  metricsets: ["perfmon"]
+  enabled: true
+  period: 1s
+  counters:


For metricset specific configs we (recently) created the convention to have it under a namespace. Means here it would be perfmon.counters:

In source i then have to write this config:"perfmon.counters" validate:"required"?

ruflin

Thanks a lot for taking this on. Having platform specific modules is something new in metricbeat.

I'm not too happy about the metricset name yet, but I couldn't come up with an alternative name yet.

Any chance you could share some example outputs for the example config you created. I'm curious to see how the events will look.

I left you a few comments which are mostly related to the assumption, that this is a dynamic metricset.

ruflin · 2017-03-17T15:05:14Z

metricbeat/module/windows/perfmon/perfmon.go

+// descriptive error must be returned.
+func (m *MetricSet) Fetch() (common.MapStr, error) {
+
+	data, err := m.handle.readData()


As far as I understand, this is a "dynamic" metricset. If yes, this would need also a namespace. See https://github.com/elastic/beats/blob/master/metricbeat/module/jolokia/jmx/jmx.go#L93 for an example.

With dynamic metricset i suppose you mean that there can be multiple groups of counters, right?

I mean that the data structure / output depends on the user configuration. In most metricsets we know in advance the exact data structure. With metricsets like jmx it depends on the user configuration and we can't ship a predefined mapping.

Ok, then it's definitely a dynamic metricset.

Yeah, but I need to find a better name for the dynamic part, as I already confused @andrewkroh with that ...

ruflin · 2017-03-17T15:07:33Z

metricbeat/module/windows/perfmon/_meta/fields.yml

@@ -0,0 +1,26 @@
+- name: perfmon


For dynamic metricsets, this will change so what we nomrally do is just define windows (module) part and leave the rest empty. Check the jmx metricset as an example.

ruflin · 2017-03-17T15:09:43Z

metricbeat/module/windows/_meta/config.yml

+  metricsets: ["perfmon"]
+  enabled: true
+  period: 1s
+  perfmon.counters:


It's great to have these example coutners which can be used. The way we handled it in the jmx metricset is that we defined only perfmon.counters: but left it empty (https://github.com/elastic/beats/blob/master/metricbeat/module/jolokia/_meta/config.yml#L8) and move the examples to the docs: https://github.com/elastic/beats/blob/master/metricbeat/module/jolokia/jmx/_meta/docs.asciidoc

ruflin · 2017-03-17T15:10:20Z

metricbeat/module/windows/_meta/fields.yml

@@ -0,0 +1,9 @@
+- key: windows
+  title: "Windows"
+  description: >


Please mark this as beta: https://github.com/elastic/beats/blob/master/metricbeat/module/jolokia/_meta/fields.yml#L4

ruflin · 2017-03-17T15:11:04Z

metricbeat/module/windows/perfmon/perfmon.go

+// Part of new is also setting up the configuration by processing additional
+// configuration entries if needed.
+func New(base mb.BaseMetricSet) (mb.MetricSet, error) {
+


A log message should be added to make it clear it is beta: https://github.com/elastic/beats/blob/master/metricbeat/module/jolokia/jmx/jmx.go#L49 We do this for all new modules / metricsets to test them first.

ruflin · 2017-03-17T15:13:00Z

metricbeat/module/windows/_meta/config.yml

+- module: windows
+  metricsets: ["perfmon"]
+  enabled: true
+  period: 1s


Our default is 10s

martinscholz83 · 2017-03-19T11:00:50Z

Hi @ruflin, thanks for your review. I'm also not happy with the naming, but i think i get something better.

martinscholz83 · 2017-03-20T08:28:24Z

Hi @ruflin, here is an example output

 {
  "@timestamp": "2017-03-20T08:26:55.241Z",
  "beat": {
    "hostname": "4201halwsd00001",
    "name": "4201halwsd00001",
    "version": "6.0.0-alpha1"
  },
  "metricset": {
    "host": "localhost",
    "module": "windows",
    "name": "perfmon",
    "rtt": 0
  },
  "type": "metricsets",
  "windows": {
    "perfmon": {
      "disk": {
        "bytes_written": 289881.870059
      },
      "processor": {
        "processor_performance": 90.531335,
        "processor_time": 100.000000
      }
    }
  }
}
{
  "@timestamp": "2017-03-20T08:26:56.241Z",
  "beat": {
    "hostname": "4201halwsd00001",
    "name": "4201halwsd00001",
    "version": "6.0.0-alpha1"
  },
  "metricset": {
    "host": "localhost",
    "module": "windows",
    "name": "perfmon",
    "rtt": 0
  },
  "type": "metricsets",
  "windows": {
    "perfmon": {
      "disk": {
        "bytes_written": 163831.444077
      },
      "processor": {
        "processor_performance": 36.804798,
        "processor_time": 100.000000
      }
    }
  }
}

martinscholz83 · 2017-03-20T08:42:42Z

Todo:

Adding tests

Most counters require two sample values in order to compute a displayable value. This means to display the first result we need two values

ruflin · 2017-03-25T20:46:23Z

+1 on having this in the windows module.

ruflin · 2017-03-28T07:05:54Z

@maddin2016 Could you update the CHANGELOG.asciidoc ? About the config options structure not sure what we should do. Would like to get the input from @andrewkroh too. I tend to the in the direction of the "flatter" one as it makes the code simpler and has less indentations to get wrong, but we loose the grouping concept. The nice thing about having it in beat, we still have the possibility to change it later in case we see people use it very differently.

martinscholz83 · 2017-03-28T07:09:16Z

I update the asciidoc. What about the windows.go file in the module folder. Is this this necessary?

- windows
    - perfmon
    windows_windows.go????

ruflin · 2017-03-28T07:16:32Z

You need a doc.go in the module folder and one in the metricset folder. What would windows_windows.go be for?

martinscholz83 · 2017-03-28T07:18:43Z

The doc.go is present. I thought there have to be a windows.go file like system.go in system to initialize the module.

ruflin · 2017-03-28T07:22:26Z

As long as there is no special implementation on the module level, the file is not needed and a "default module" will be initialised.

martinscholz83 · 2017-03-28T07:46:23Z

What about the failing tests. Seems it tries to build for linux and mac. Something is have missed?

ruflin · 2017-03-28T07:48:41Z

The first part which is failing is because of the linting. Make sure to run make fmt and make update before the next push. Looking into the other ones now.

ruflin · 2017-03-28T07:50:04Z

The crosscompile error is because you are missing the doc.go in the perfmon directory. Adding it should also solve the other build error.

martinscholz83 · 2017-03-28T08:54:36Z

@ruflin, can you rebuild travis again. I don't see any output whats going wrong.

ruflin · 2017-03-28T08:56:00Z

@maddin2016 I think travis is a bit grumpy today. Have the same issue on other PRs. Probably we just have to wait :-(

martinscholz83 · 2017-03-28T08:57:23Z

Ok 👍

martinscholz83 · 2017-03-28T10:26:20Z

How i can pdh_windows_integration_test.go only test on windows?

ruflin · 2017-03-28T10:35:08Z

You can either call it pdh_integration_windows_test.go or add a build flag on top of the file to specify only windows.

martinscholz83 · 2017-03-28T10:39:48Z

On top i had specified //+build integration windows. But that doesn't seem to work. I will rename the file and see what happens.

martinscholz83 · 2017-03-28T13:42:38Z

🎉

andrewkroh · 2017-03-28T14:11:00Z

@maddin2016 All green, great! So we are going to merge this is and would like to iterate on it with some additional (smaller) PRs. We're starting a meta issue to track some changes to the module. Hopefully you can help us with these changes. Thanks for the contribution!!

martinscholz83 · 2017-03-28T15:01:34Z

Of course!!

sy43165 · 2017-05-23T19:22:14Z

I'm experimenting with the metricbeat-6.0.0-alpha1 and I have an issue/concern. Forgive the noob question but do I report that here?

andrewkroh · 2017-05-23T20:59:53Z

@sy43165 Please use https://discuss.elastic.co/c/beats.

Add module windows

43ee390

martinscholz83 added 2 commits March 16, 2017 09:24

import module

569e7f4

Format fields.yml

a02bdc9

fix _meta

331a56f

martinscholz83 force-pushed the module-perfmonbeat branch from c610ab3 to 331a56f Compare March 16, 2017 09:13

andrewkroh added enhancement Metricbeat Metricbeat module review :Windows labels Mar 16, 2017

andrewkroh assigned ruflin and andrewkroh and unassigned ruflin and andrewkroh Mar 16, 2017

andrewkroh requested review from ruflin and andrewkroh March 16, 2017 21:38

andrewkroh requested changes Mar 16, 2017

View reviewed changes

andrewkroh added the needs tests label Mar 16, 2017

ruflin reviewed Mar 17, 2017

View reviewed changes

[WIP]

30a1966

ruflin reviewed Mar 17, 2017

View reviewed changes

add hints from @ruflin and @andrewkroh

d287fa9

martinscholz83 added 2 commits March 20, 2017 10:33

mark first fetch

7e6b88c

Most counters require two sample values in order to compute a displayable value. This means to display the first result we need two values

add tests

9047b36

martinscholz83 added 2 commits March 28, 2017 09:13

update changelog.asciidoc

f670912

Merge branch 'master' into module-perfmonbeat

267bf17

martinscholz83 added 2 commits March 28, 2017 09:57

make fmt && make update

88b0a79

add perfmon doc.go

6cadedf

rename integration test file

dd2b539

ruflin mentioned this pull request Mar 28, 2017

Enhance windows/perfmon metricset #3828

Closed

10 tasks

andrewkroh merged commit 3d2b0fa into elastic:master Mar 28, 2017

andrewkroh added the v6.0.0-alpha1 label Mar 28, 2017

monicasarbu removed the needs tests label Apr 3, 2017

monicasarbu mentioned this pull request Apr 6, 2017

Remove windows module from the short config file #3936

Merged

New module windows for metricbeat #3758

New module windows for metricbeat #3758

Conversation

martinscholz83 commented Mar 16, 2017

elasticmachine commented Mar 16, 2017

elasticmachine commented Mar 16, 2017

martinscholz83 commented Mar 16, 2017

andrewkroh left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ruflin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

martinscholz83 commented Mar 19, 2017

martinscholz83 commented Mar 20, 2017

martinscholz83 commented Mar 20, 2017 • edited Loading

ruflin commented Mar 25, 2017

ruflin commented Mar 28, 2017

martinscholz83 commented Mar 28, 2017

ruflin commented Mar 28, 2017

martinscholz83 commented Mar 28, 2017 • edited Loading

ruflin commented Mar 28, 2017

martinscholz83 commented Mar 28, 2017

ruflin commented Mar 28, 2017

ruflin commented Mar 28, 2017 • edited Loading

martinscholz83 commented Mar 28, 2017

ruflin commented Mar 28, 2017

martinscholz83 commented Mar 28, 2017

martinscholz83 commented Mar 28, 2017

ruflin commented Mar 28, 2017

martinscholz83 commented Mar 28, 2017

martinscholz83 commented Mar 28, 2017

andrewkroh commented Mar 28, 2017

martinscholz83 commented Mar 28, 2017

sy43165 commented May 23, 2017

andrewkroh commented May 23, 2017 • edited Loading

andrewkroh left a comment •

edited

Loading

martinscholz83 commented Mar 20, 2017 •

edited

Loading

martinscholz83 commented Mar 28, 2017 •

edited

Loading

ruflin commented Mar 28, 2017 •

edited

Loading

andrewkroh commented May 23, 2017 •

edited

Loading