MongoDB module #6283

kvch · 2018-02-05T17:29:44Z

I added a new Filebeat module for MongoDB what I created during testing the FB module generator tooling.

This is the dashboard I created:

It does not look good, as I hasn't been able to get real error messages from my instance.

ruflin · 2018-02-05T23:41:31Z

filebeat/module/mongodb/log/_meta/fields.yml

+  description: >
+      Contains fields from MongoDB logs.
+  fields:
+  - name: timestamp


Should we use @timestamp for this?

What do you mean?

Defining timestamp here means it will be mongodb.log.timestamp in the event. But normally the processed timestamp from the log line when the event happened ends up in @timestamp. Is this a different timestamp here?

I think a separate timestamp field is required, because it stores the actual time the log was generated. @timestamp is the time it was processed. These two times are not necessarily the same.

We have similar behaviour in other modules, for example Kafka. What we do there is that we have the timestamp of the actual log line in @timestamp and the one when the log line was actually read in read_timestamp. Name of read_timestamp is not great but we don't have a better one yet.

If you look at the ingest pipeline of the Kafka module you will see the the timestamps are moved around to make sure @timestamp is the one after processing.

ruflin · 2018-02-05T23:41:43Z

filebeat/module/mongodb/log/_meta/fields.yml

+        Context of message
+    example: initandlisten
+    type: keyword
+  - name: message


We should use the common message field here.

I dropped type: text.

I was thinking this should go into the top level message field. But looking at other modules I see we don't do this yet.

The behaviour I would expect is that the extracted message part of the log line ends up in the top level message field and the raw log line for example under log.message or log.raw. I'm good with keeping it as is as it's not different in other modules yet.

If we have it here, it probably makes sense to keep it as text and not keyword.

I readded text.

ruflin · 2018-02-12T00:20:59Z

filebeat/module/mongodb/log/_meta/fields.yml

+    description: >
+        Severity level of message
+    example: I
+    type: keyword


This is interesting as I so far assume serverity is a sortable integer in ECS. It seems what is used here as severity is more log.level. @MikePaquette FYI

I came up with field names based on MongoDB logging guide: https://docs.mongodb.com/manual/reference/log-messages/ It might provide more context on the subject.
However, I agree that it is a bit weird in MongoDB. If you look at the section "Severity", you can see the taxonomy of log levels. So I am not sure why it's named severity.
Does it cause problem in ECS?

I think we can keep it as is as it's under the mongodb namespace. In case we map this to ECS later it would end up in log.level from my perspective. No need to solve this now.

kvch · 2018-02-15T16:18:32Z

@ruflin I think it's ready for a new round of review. Travis fails, because of XPack installation problems.

ruflin

PR looks good to me. I would suggest we release this first as beta, get some feedback and then push it to GA in the next minor?

ruflin · 2018-02-19T04:04:33Z

filebeat/module/mongodb/_meta/config.yml

@@ -0,0 +1,8 @@
+- module: mongodb
+  # All logs


What does all logs mean here? Does mongodb only have 1 type of logs?

Yes, that's all I found in their docs: https://docs.mongodb.com/manual/reference/log-messages/

ruflin · 2018-02-19T04:05:43Z

filebeat/module/mongodb/log/_meta/fields.yml

+    description: >
+        Severity level of message
+    example: I
+    type: keyword


I think we can keep it as is as it's under the mongodb namespace. In case we map this to ECS later it would end up in log.level from my perspective. No need to solve this now.

ruflin · 2018-02-19T04:06:37Z

filebeat/module/mongodb/log/ingest/pipeline.json

@@ -0,0 +1,41 @@
+{
+  "description": "Pipeline for parsing MongoDB log logs",


Mongodb log logs -> I think I start to understand why you call the filset log. What exactly is in these logs?

kvch · 2018-04-10T05:23:36Z

Everything is green.

kvch added module review Filebeat Filebeat in progress Pull request is currently in progress. labels Feb 5, 2018

ruflin reviewed Feb 5, 2018

View reviewed changes

kvch force-pushed the feature/filebeat/mongodb-module branch from 6a4b180 to 06c3a57 Compare February 6, 2018 15:43

kvch removed the in progress Pull request is currently in progress. label Feb 9, 2018

ruflin reviewed Feb 12, 2018

View reviewed changes

kvch force-pushed the feature/filebeat/mongodb-module branch 2 times, most recently from b7eb3bb to 9f51a27 Compare February 15, 2018 14:32

ruflin reviewed Feb 19, 2018

View reviewed changes

kvch force-pushed the feature/filebeat/mongodb-module branch from 9f51a27 to bd64df9 Compare March 27, 2018 10:08

kvch added 8 commits March 28, 2018 19:53

filebeat: add MongoDB module

3857fad

minor fixes

9fdae35

add missing docs update

ad55c5c

fix timestamp

c6aa260

fix grok

2b3ad3f

fix timestamp format

00befff

remove unnecessary words from description

69d5e5a

add changelog entry

96f80d0

kvch force-pushed the feature/filebeat/mongodb-module branch from 78eedde to 96f80d0 Compare March 28, 2018 17:54

ruflin merged commit 6f99a96 into elastic:master Apr 10, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MongoDB module #6283

MongoDB module #6283

kvch commented Feb 5, 2018 •

edited

Loading

ruflin Feb 5, 2018

kvch Feb 6, 2018

ruflin Feb 9, 2018

kvch Feb 9, 2018

ruflin Feb 12, 2018

ruflin Feb 5, 2018

kvch Feb 9, 2018

ruflin Feb 12, 2018

kvch Mar 28, 2018

ruflin Feb 12, 2018

kvch Feb 15, 2018

ruflin Feb 19, 2018

kvch commented Feb 15, 2018

ruflin left a comment

ruflin Feb 19, 2018

kvch Mar 28, 2018

ruflin Feb 19, 2018

ruflin Feb 19, 2018

kvch commented Apr 10, 2018

		@@ -0,0 +1,41 @@
		{
		"description": "Pipeline for parsing MongoDB log logs",

MongoDB module #6283

MongoDB module #6283

Conversation

kvch commented Feb 5, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kvch commented Feb 15, 2018

ruflin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kvch commented Apr 10, 2018

kvch commented Feb 5, 2018 •

edited

Loading