"match regular expression" stream rule type fails to properly validate message starting with a blank space #1936

StCyr · 2016-03-16T13:04:39Z

Problem description

Whe you want to match the beginning of a syslog message that starts with a blank space against a "match regular expression" stream rule, you must not include the leading blank space in your regular expression if you want the UI to tell you the message will match. Though, you must include it if you want the StreamMatcherFilter to effectively route your message to your stream.

Steps to reproduce the problem

Have a system sending your graylog server a syslog message starting with an empty space (eg: " Login incorrect: [root] (from client BLABLABLA port 0 cli a.b.c.d)\n";
Set up the following stream rule to match the beginning of the message: "^Login incorrect.*";
Confirm that the UI tells you the message from step 1 will match;
Confirm that no messages get routed to the stream;
Set up the following stream rule to match the beginning of the message: "^.Login incorrect.*";
Confirm that the UI tells you the message from step 1 wont't match;
Confirm that messages get routed to the stream.

Sample data

Here are sample data to reproduce the problem:

Message

{
  "message": {
    "message": " Login incorrect: [root] (from client BLABLABLA port 0 cli a.b.c.d)\n",
    "timestamp": "2016-03-16T09:27:44.249Z",
    "process_id": "22562",
    "level": 5,
    "application_name": "freeradius",
    "source": "louise",
    "facility": "local5",
    "_id": "50b39030-eb59-11e5-a255-08002703c127",
    "gl2_source_input": "56e13cd1e4b0deefe9f54cf8",
    "gl2_source_node": "3aee6e01-9fdb-43d4-83b7-324a58e265d6",
    "streams": [],
    "gl2_remote_port": 41948,
    "gl2_remote_ip": "w.x.y.z"
  },
  "index": "graylog2_0"
}

Rules

 "stream_rules": [
    {
      "id": "56e91f0de4b0fcf9693e796a",
      "field": "application_name",
      "value": "freeradius",
      "stream_id": "56e91ebae4b0fcf9693e790f",
      "type": 1,
      "inverted": false
    },
    {
      "id": "56e91f2fe4b0fcf9693e798e",
      "field": "message",
      "value": "^Login incorrect.*",
      "stream_id": "56e91ebae4b0fcf9693e790f",
      "type": 2,
      "inverted": false
    }
  ]

Environment

Graylog Version: 1.3
Elasticsearch Version: 1.7.5
MongoDB Version: 3.2.4
Operating System: Ubuntu 14.04
Browser version: Firefox 44.0.2

IRC Logs

10:46:19 - StCyr: mmm, created a new stream, restarted the whole server, still not having my messages routed to the stream though the "Rules Editor UI" say they match the rules?!?
10:46:45 - StCyr: Here's a debug log after pausing-starting the stream => http://pastebin.com/40uCCexf
10:47:12 - StCyr: Message 50b39030-eb59-11e5-a255-08002703c127 matches the rule in the "Edit Rules" UI
10:54:37 - joschi: StCyr: could you post some example messages and the stream rules?
10:54:51 - joschi: StCyr: you don't need to restart Graylog when creating new streams or changing stream rules
10:54:58 - StCyr: pastebin?
10:58:35 - StCyr: message: http://pastebin.com/NsEwkzJG
10:59:01 - StCyr: rules: http://pastebin.com/UC0mGNg7
11:00:35 - StCyr: there's always this blank space at the start of all my messages, but "Edit Rules" UI doesn't mind
11:02:17 - StCyr: joschi: maybe I should try to add this blank space in my regular expression though "Edit Rules" UI doesn't mind?
11:05:53 - StCyr: the thing is: if I add this blank space in my regex the "Edit Rules" UI say message doesn't match O.o
12:09:27 - joschi: StCyr: the leading blank is definitely a problem. please create an issue for that problem (the web interface saying the rule matched while it didn't) at https://github.com/Graylog2/graylog2-server/issues

The text was updated successfully, but these errors were encountered:

dennisoelkers · 2016-03-24T10:11:53Z

This is related to message fields being trimmed in the Message class, therefore the message which is retrieved by the web interface and passed to the server for stream rule matching is already trimmed of all leading whitespace. Will need to investigate if there is a good reason for trimming fields.

Seems to be introduced in 152e901 due to #186, unfortunately without any given context/reason.

lennartkoopmann · 2016-03-24T14:38:38Z

This was introduced because people struggled with "invisible" whitespaces at the beginning or more importantly end of the string. We decided to trim them off and let people use the \s regex notation instead.

The UI should be consistent with this of course or as a workaround be able to alert that there are whitespaces.

I'd vote for not changing the existing backend behaviour because this can lead to hard to debug side effects when upgrading and stream rules are suddenly no longer matching.

dennisoelkers · 2016-03-29T10:47:39Z

If we do not want to change the backend behavior, we have no way to fix this issue, as the frontend simply does not know that certain fields are trimmed.

We could look into moving the trimming to the stream rule matchers (or to be safe: match a stream rule both against the trimmed and non-trimmed field if it matches against the start of the string), although I think correcting a bug which causes an inconsistency between predicted and actual matching results justifies a breaking change, as long as it is documented properly.

Before this change, one of the public constructors of `Message` was using `addField()` to add fields in the `Message` object, while the other was adding the fields by hand to the internal `Map` used to store message fields. That created an inconsistency, as the constructor using `addField()` trims all `String` field values by default, unlike the other constructor. You can read more about it in: #1936. This commit ensures both constructors trim `String` values, while still ensuring `Message(final String message, final String source, final DateTime timestamp)` will create a `Message` object containing the required fields given as arguments. Making this change will break stream rules checking for one or more leading or trailing whitespaces in message fields. Fixes #1936

Before this change, one of the public constructors of `Message` was using `addField()` to add fields in the `Message` object, while the other was adding the fields by hand to the internal `Map` used to store message fields. That created an inconsistency, as the constructor using `addField()` trims all `String` field values by default, unlike the other constructor. You can read more about it in: #1936. This commit ensures both constructors trim `String` values, while still ensuring `Message(final String message, final String source, final DateTime timestamp)` will create a `Message` object containing the required fields given as arguments. Making this change will break stream rules, extractors, and drool rules checking for one or more leading or trailing whitespaces in message fields. Fixes #1936

Before this change, one of the public constructors of `Message` was using `addField()` to add fields in the `Message` object, while the other was adding the fields by hand to the internal `Map` used to store message fields. That created an inconsistency, as the constructor using `addField()` trims all `String` field values by default, unlike the other constructor. You can read more about it in: #1936. This commit ensures both constructors trim `String` values, while still ensuring `Message(final String message, final String source, final DateTime timestamp)` will create a `Message` object containing the required fields given as arguments. Making this change will break stream rules, extractors, and drool rules checking for one or more leading or trailing whitespace in message fields. Fixes #1936

kroepke added bug P2 S1 labels Mar 18, 2016

kroepke added this to the 2.0.0 milestone Mar 18, 2016

dennisoelkers self-assigned this Mar 22, 2016

kroepke modified the milestones: 2.1.0, 2.0.0 May 2, 2016

joschi mentioned this issue Jul 18, 2016

Stream simulator says a message matches all rules but stream never matches #2494

Closed

edmundoa self-assigned this Jul 18, 2016

edmundoa mentioned this issue Jul 20, 2016

Always trim message field values on Message class #2510

Merged

joschi closed this as completed in #2510 Jul 28, 2016

kroepke added the triaged label Sep 21, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

"match regular expression" stream rule type fails to properly validate message starting with a blank space #1936

"match regular expression" stream rule type fails to properly validate message starting with a blank space #1936

StCyr commented Mar 16, 2016

dennisoelkers commented Mar 24, 2016 •

edited

Loading

lennartkoopmann commented Mar 24, 2016

dennisoelkers commented Mar 29, 2016

"match regular expression" stream rule type fails to properly validate message starting with a blank space #1936

"match regular expression" stream rule type fails to properly validate message starting with a blank space #1936

Comments

StCyr commented Mar 16, 2016

Problem description

Steps to reproduce the problem

Sample data

Message

Rules

Environment

IRC Logs

dennisoelkers commented Mar 24, 2016 • edited Loading

lennartkoopmann commented Mar 24, 2016

dennisoelkers commented Mar 29, 2016

dennisoelkers commented Mar 24, 2016 •

edited

Loading