Logs should be prefixed with pipeline name when using multiple pipelines.

[robcowart]

I have two pipelines loaded at the same time, both of which are using the netflow codec. A log messages are received such as...

[2017-09-17T09:35:00,928][WARN ][logstash.codecs.netflow ] Can't (yet) decode flowset id 1024 from source id 0, because no template to decode it with has been received. This message will usually go away after 1 minute.

The question is... which of the two pipelines logged this message? This question applies regardless of which plugin is logging the message.

Logstash should prefix all log messages with the pipeline name of the pipeline from which the message was generated.

[jordansissel]

@robcowart focusing on your specific example for a moment, if we can improve this log message to be actionable, what action would a user take? Nothing?

My sense is that adding the pipeline to the log message (in this specific case) will do nothing to making this log message actionable. In fact, the log message tells you to "just wait" as the action ("this message will usually go away ...").

I'm not against adding the pipeline to log messages, but in this case I am having difficulty finding how this would improve the usefuless of the specific flowset warning.

Specific to the missing flowset problem:

1. Maybe we should move this warning to be a debug message instead of a warning, since there is no action the user can take to improve the problem.
2. Maybe we add a metric for the netflow codec that counts how many payloads were lost because a template is missing

And separately, we can work on adding the pipeline name to the logs messages

[robcowart]

This was only an example. Another would be the DNS filter... if I misconfigure the nameserver attribute (wrong IP address) in one of the pipelines I will get a lot of lookup timeout messages. Without knowing which pipeline is throwing the error I have to check each one individually until I find it.

Specific to this message from the Netflow codec, that message will only go away if the codec can understand the template. That isn't always the case (lookup the issue where MPLS values weren't properly decided) and this message will continue forever. In such cases the device may be able to reconfigured to send a less problematic flow record. If I have different devices sending events to different pipelines, the number of devices I would have to verify would be reduced.

Again, these are just examples.