Incorrect prospector template

[mazilu88]

Hello and thanks for your great module!

I tried to use "include_lines" without succes because the prospector template defines it as

  include_lines:
    - []
    - []

When I switched the syntax to the one from the official documentation it filtered the log accordingly.
https://www.elastic.co/guide/en/beats/filebeat/current/configuration-filebeat-options.html

The following example configures Filebeat to export any lines that start with "ERR" or "WARN":

include_lines: ["^ERR", "^WARN"]

Ca you please check it out.

Thanks!

[pcfens]

Is Filebeat not starting correctly or not parsing the files correctly? Both formats are valid YAML arrays, but not all YAML parsers are the same. It's not hard to fix, but I want to make sure that there isn't some other error going on with the config file.

Any chance you can include the relevant part of your manifest that's creating the config file example you included?

[pcfens]

There's a chance that this is caused because the lines weren't wrapped in quotes. Could you try the include_exclude_fix branch to see if that works as expected too?

[mazilu88]

Filebeat is not parsing the file correctly in the sense that it includes all of the entries in syslog, not just the ones in include_lines.
First try was with

  include_lines:
     - ["balbla\\["]
     - ["randomword"]

Second try did job

 include_lines: ["balbla\\[", "randomword"]

I did not had the time to test the other branch.

[pcfens]

I should have noticed the issue earlier - do you have any idea where the square brackets around "randomword" are coming from? They're not in the template, so I'm not sure where they're coming from.

Do you know what version of filebeat you're using, and a copy of the test manifest might help too since I'm not able to re-create the issue.

[mazilu88]

The square brackets came from my declaration and it seems that it was the problem

include_lines  => [
      '["balbla\\["]',
      '["randomword"]',
]

But in that case it seemed to me that it should not match at all and not match all lines. Thus I thought it was a problem in the yaml that needed fixing. Sorry for wasting your time.

[pcfens]

I think that the square brackets made the YAML parser think that each line was an array. Wrapping the lines in quotes should help with that, so I'll merge in the branch as extra protection against problems like this in the future.

Thanks for helping find a weak point.