Metricbeat Prototype #611

ruflin · 2016-01-04T09:36:00Z

This is a working prototype of Metricbeat. This PR is intended to discuss the prototype and get it to a mergeable state. See #619

Currently it has the following metrics as examples:

MySQL Module with the following metrics
- Status
Redis module with the following metrics
- Stats

Todo

The following things are not implemented yet and have to be done before merging (or a separate issue has to be opened):

CHANGELOG.md
Improve logging
Do not load modules / metrics which are not in the configuration
Improve panic handling if metric crashes
Make index configurable in libbeat to be passed by each metric
Add integration test framework

ruflin · 2016-01-07T09:30:37Z

Community Metricbeat Modules

Over time the discussion will come up which metrics we maintain from the elastic side and which ones should be maintained by the community. To solve this issue I propose the following solution.

There are two versions of metricbeat, the version supported by elastic and the community version. The community version fully depends on the supported version except for the modules and metrics. It lives in it's on repository for example under elastic/metricbeat-community. The only requirement is that there are no name conflicts between the modules of the two metricbeats.

The modules in the community versions could also be maintained by community contributors. In addition we could have an "incubation" phase to also migrate some of the modules over time from the community beat into our own beat.

Here is a working prototype branch of the metricbeat-community:

https://github.com/ruflin/beats/tree/metricbeat-community/metricbeat-community

Note: This branch is just as an example and is not intended to be merged

The same concept could also be applied to protocols in packetbeat.

tsg · 2016-01-10T12:08:59Z

metricbeat/beat/list.go

+	_ "github.com/elastic/beats/metricbeat/module/elasticsearch/cluster-stats"
+	_ "github.com/elastic/beats/metricbeat/module/elasticsearch/info"
+	_ "github.com/elastic/beats/metricbeat/module/elasticsearch/stats"
+	_ "github.com/elastic/beats/metricbeat/module/redis/info"


Another option here would be to just import the module and let the module import the metrics. Not sure which way is more maintainable, just figured I mention the option for discussion.

This would definitively be nicer as this list here would have to be updated less frequently and it imports would be more local. I tried this first but it doesn't work because of cyclic imports. As I designed metricbeat so far that a MetricSet registers with the module and the module doesn't know in advance about all the MetricSets, the MetricSet imports its module. Not sure if there is a way around the cyclic imports to make it mor maintainable. An other option could be to automatically generate this list.

tsg · 2016-01-10T13:00:22Z

Another thing that we'll need is a way for each module to generate JSON for the Elasticsearch template.

tsg · 2016-01-10T13:03:23Z

metricbeat/helper/metric.go

+
+			// Set type from event if set
+			if _, ok := event["type"]; ok {
+				typeName = event["type"].(string)


Would be worth checking the type assertion (typeName, ok = event["type"].(string)) to avoid a run time panic.

tsg · 2016-01-10T13:21:41Z

I think we will need a way to add custom fields not only by module and metric, but also on a per "host" basis. For example, if one wants to monitor multiple Redis servers, they would need to add some fields to the events to know which events refer to each hosts.

Perhaps it would be worth making the host a first class concept? Otherwise the logic to handle multiple hosts will get duplicated in every module.

tsg · 2016-01-10T13:22:34Z

metricbeat/module/redis/redis.go

+}
+
+func Connect() redis.Conn {
+	conn, err := redis.Dial("tcp", "127.0.0.1:6379")


Obviously a bit simplistic here :-).

Prototype ;-)

ruflin · 2016-01-10T18:39:05Z

@urso @tsg Thanks a lot for the review. I will integrate the inputs and will let you know as soon as a next version for review is available.

monicasarbu · 2016-01-28T18:49:12Z

metricbeat/beat/config.go

+type RawMetricsConfig struct {
+	Metricbeat struct {
+		Modules map[string]struct {
+			MetricSets map[string]interface{} `yaml:"metricsets"`


this is common.MapStr right?

Good point, it is :-)

mrkschan · 2016-01-29T05:54:20Z

May I ask if the community version is compiled as one binary called "metricbeat" or multiple binaries?

monicasarbu · 2016-01-29T08:53:12Z

@mrkschan That is a good question. It is one binary as you can add support for another service in Metricbeat by adding a new module.

mrkschan · 2016-01-29T14:46:31Z

So, would that be bloated (eventually) and would it be possible that dependency of one service is having conflict with another?

ruflin · 2016-01-29T19:17:57Z

The goal is to keep this as lightweight as possible and also to keep the module as independent as possible. This means, at any time it should be possible to only build on module and for example in the case of redis, only have a redisbeat. Of course, this doesn't mean there won't be any conflicts with dependency. I'm tempted to say we solve it when we get there. Do you have any specific library in mind?

mrkschan · 2016-01-31T14:14:55Z

@ruflin, i don't get what do you mean by building one module. Do you mean we will have some scripts to generate a main() for building a particular module (e.g. generating a main() for the redis module to get redisbeat)?

Re: conflicts. I don't have any idea which library would introduce conflict with another so far, though, I can foresee backward incompatibility of a particular library in the future.

ruflin · 2016-01-31T19:46:18Z

@mrkschan Here is an example I built some weeks ago. It is not up-to-date anymore but should give you and understand of what I mean: https://github.com/ruflin/beats/tree/metricbeat-community/metricbeat-community. Just assume it would be called redisbeat and the module would be redis. Please be aware that most of these points are still under discussion.

Which particular library are you referring to?

tsg · 2016-02-05T15:16:51Z

metricbeat/module/mysql/status/status_test.go

+func TestFetch(t *testing.T) {
+
+	if testing.Short() {
+		t.Skip("Skipping in short mode, because it requires Redis")


s/Redis/Mysql

(fixed, multiple times ...)

tsg · 2016-02-05T15:21:06Z

I'm good with merging at this point. Thanks @ruflin for driving it up to here!

This is the first basic implementation of MetricBeat. For more details on what it contains, check below. = MetricSets The following basic MetricSet are part of this first version. * Apache module with Apache status * Redis module with Redis info * MySQL module with status Be aware that only very few metrics are collected per MetricSet, but more could be added easily = Data Model The current data model of each metricset can be found in the file beater/metricbeat.go in the header docs. = Error Handling It is assumed, that a module or metricset can panic. This panic is catched to not kill all other modules. This error is reported in the log files. The failing module / metricset will not be restarted = Configuration Each module and metricSet can have its own configuration. The following fields are generic configurations per module and are always available: * Period * Hosts More details can be found in etc/beat.yml = Testing * Golang test framework for unit tests * Golang test framework combined with docker-compose for integration tests * Python libbeat system test framework combined with docker-compose for system tests All tests are intended to be run inside the docker environment. = Vendoring The prevent dependency conflicts between modules, each module can have its own vendor environment. This is implemented as an example in the MySQL module.

Metricbeat Prototype

ruflin mentioned this pull request Jan 4, 2016

Metricbeat #619

Closed

18 tasks

monicasarbu added the discuss Issue needs further discussion. label Jan 4, 2016

ruflin changed the title ~~Metricbeat Prototypte~~ Metricbeat Prototype Jan 7, 2016

tsg reviewed Jan 10, 2016
View reviewed changes

ruflin force-pushed the metricbeat branch from bd108ae to efeffe2 Compare January 21, 2016 12:39

ruflin added in progress Pull request is currently in progress. and removed discuss Issue needs further discussion. labels Jan 25, 2016

ruflin force-pushed the metricbeat branch from efeffe2 to 63780c4 Compare January 26, 2016 22:24

monicasarbu reviewed Jan 28, 2016
View reviewed changes

ruflin force-pushed the metricbeat branch 5 times, most recently from 0304766 to d7aa431 Compare February 4, 2016 13:15

ruflin force-pushed the metricbeat branch 3 times, most recently from 16e3618 to f4800cd Compare February 5, 2016 13:35

ruflin added review and removed in progress Pull request is currently in progress. labels Feb 5, 2016

tsg reviewed Feb 5, 2016
View reviewed changes

ruflin force-pushed the metricbeat branch 3 times, most recently from 899b301 to 330a6ff Compare February 5, 2016 16:36

ruflin force-pushed the metricbeat branch from 181de75 to 168e63f Compare February 8, 2016 08:26

tsg added a commit that referenced this pull request Feb 8, 2016

Merge pull request #611 from ruflin/metricbeat

4e27fc3

Metricbeat Prototype

tsg merged commit 4e27fc3 into elastic:master Feb 8, 2016

ruflin deleted the metricbeat branch February 9, 2016 08:54

andrewkroh mentioned this pull request Jan 14, 2020

[meta] Update to ECS 1.2 to 1.4 #13940

Closed

51 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Metricbeat Prototype #611

Metricbeat Prototype #611

ruflin commented Jan 4, 2016

ruflin commented Jan 7, 2016

tsg Jan 10, 2016

ruflin Jan 10, 2016

tsg commented Jan 10, 2016

tsg Jan 10, 2016

ruflin Jan 10, 2016

tsg commented Jan 10, 2016

tsg Jan 10, 2016

ruflin Jan 10, 2016

ruflin commented Jan 10, 2016

monicasarbu Jan 28, 2016

ruflin Jan 29, 2016

mrkschan commented Jan 29, 2016

monicasarbu commented Jan 29, 2016

mrkschan commented Jan 29, 2016

ruflin commented Jan 29, 2016

mrkschan commented Jan 31, 2016

ruflin commented Jan 31, 2016

tsg Feb 5, 2016

ruflin Feb 5, 2016

tsg commented Feb 5, 2016

Metricbeat Prototype #611

Metricbeat Prototype #611

Conversation

ruflin commented Jan 4, 2016

Todo

ruflin commented Jan 7, 2016

Community Metricbeat Modules

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tsg commented Jan 10, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tsg commented Jan 10, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ruflin commented Jan 10, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mrkschan commented Jan 29, 2016

monicasarbu commented Jan 29, 2016

mrkschan commented Jan 29, 2016

ruflin commented Jan 29, 2016

mrkschan commented Jan 31, 2016

ruflin commented Jan 31, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tsg commented Feb 5, 2016