Fields changed for new Beats input in Graylog 3

[Nklya]

Hello.

I've noticed that source field now contain filename instead of host for new Beats input in Graylog 3. I haven't found anything in the changelog.

Expected Behavior

Field source in Beats input contains a hostname

Current Behavior

Field source in Beats input contains a filename

Possible Solution

The previous behavior can be reached by using Beats (deprecated) input.
I think at least it should be documented and added to changelog.

Steps to Reproduce (for bugs)

1. Install Graylog 3.
2. Setup filebeat to stream any log to Graylog
3. Add new input with type Beats
4. You will receive messages with field source in Beats input contains a filename instead of hostname.

Context

We have many inputs and everywhere field source contains hostname. Now this change will lead to misunderstandings and changes dashboards and alerts.

Your Environment

• Graylog Version: 3.0.1-2 (also tested 3.0.0)
• Elasticsearch Version: 6.6.0
• MongoDB Version: 4.0.2
• Operating System: Ubuntu 18.04
• Browser version: Chrome 74.0.3729.108

[mpfz0r]

How did you configure the "add beats prefix" option on the Input?

Refs #5537

[Nklya]

no_beats_prefix: true in the input

[mpfz0r]

That's why I changed this default to false
filebeat uses the source field for the filename since a while now: https://www.elastic.co/guide/en/beats/filebeat/current/migration-changed-fields.html#_the_file_field_was_renamed_to_source

with the prefix, it would would be filebeat_source and not collide with graylogs source field.

[Nklya]

Thank you very much 🤗 !

I changed it to false and now everything works as before.

This checkbox in input settings:

[Nklya]

In case someone hit the same.

You can avoid this problem by adding this to filebeat.yml without disabling no_beats_prefix

processors:
- rename:
    fields:
     - from: "source"
       to: "file"

[nix-power]

Meaning this checkbox should not be checked ?

[mpfz0r]

Meaning this checkbox should not be checked ?

That's up to you. By default it's not checked, which adds a prefix like winlobeat_ or filebeat_ to ever field.
That prevents the mentioned name collision for the source field.
Alternatively you can rename this field with aforementioned beats config snippet.