Clarify mapping related to logging

[bmagistro]

Hello,

For logging from some of our applications we produce a log line that is ISO8601 Date (microsecond + tz), log level, thread id/name, file, function, line number, message. This provides information about where in our code the log message originated, any thoughts on where/how to map these items into ecs?

From this line it seems like the mapping would be as follows optionally logging the original to log.original.
date/time to @timstamp
log level (think syslog facility) to log.level
thread id/name to ??
file to ??
function to ??
line number to ??
message to message

If process.tid and process.thread (or equivalent) are added, I can see those being used for thread id/name.

Thanks!

[webmat]

Hey @bmagistro thanks for opening this issue.

Your mapping so far makes sense. Here's suggestions on the fields you were wondering about:

• log.level:
  • we do have log.level already for the raw value of your logging level (think DBG, DEBUG, INFO, etc.)
  • We have been discussing creating a second field for a normalized value for log level in this issue: proposal for Event fields #129
  • Right now we don't have anything to track syslog facility per se, but if you want to record that, you're more than welcome to add a keyword field at log.facility in your index *. It's been discussed before here syslog PRI #36 but we haven't had time to flesh that out yet.
• thread id: it's not yet officially in ECS, but you can look at this discussion here Missing thread, process + thread id naming DEC+HEX #82 (comment). This is likely how it will look, once we add it. So you can record this at process.thread.tid.
• We have the file and error fields that could be used for this. Although right now file has been thought through mostly in the security context (e.g. FIM, virus detection), and error doesn't yet have fields that would be useful to track the source of error yet. I've opened Using ECS to track the source of errors or exceptions #154 for this, because I agree we should support this common use case.
  • The one field we already have is error.message, so I'd suggest using this one over message.

* ECS is not like most other schemas. When a field is missing from ECS, you're welcome to add it in your events anyway. We're working on documentation that will explain this better (e.g. how to do so safely vs future versions of ECS), but that documentation isn't ready yet. Still being worked on here #130.

[bmagistro]

Thanks for the suggestions. Will see where things go, the bigger concern I had with adding/changing the template for our use was compatibility with future changes. Ideally we can map the items we want into existing/planned fields to minimize those impacts.

I did mean severity and not facility...too many things going on.

For these messages, they may not be errors (entries in application log file as opposed to stack trace/thrown exceptions). If this was just a log file coming up from beats, I'm assuming the "message" component would map to "message"?

[webmat]

Yes, the work on #130 is about documenting how to avoid these clashes as best as possible. It's a tricky nut to crack, without getting very deep into details :-)

Note that if you map your event information to fields that end up not being the official field in ECS, you should be able to gradually move to ECS by using Elasticsearch' alias field type or copyTo. The bigger concern would be if you end up using the same field name with a conflicting field type. This would be the more painful situation.

Typically most messages go to message, you are correct. And in that vein, I've actually raised the question (internally) whether we should even have error.message on an event. The main message should always be in message. Other error details could be broken out in the error field set, whether it's exceptions or simpler error events, though.

[bmagistro]

Thanks for the input. I think I have gotten our common logs (auditd, secure, messages, nginx, apache, and some of our apps) mapped in at a very basic level.
With in the log section, we added path that would map to filebeat "source". For the app log described above, leveraging error.function and adding a "line" to file seemed to make the most sense. I would like to get a little more run time but would be happy to share the logstash filter file for this if anyone is interested. I am very certain that we can be doing MUCH more parsing than we are in some logs.

[webmat]

Hey @bmagistro, I'll close this for the time being, as it looks like everything has been answered.

Please open other issues if you have further questions, or if you've faced problems and would like to contribute ideas to ECS.

Thanks for getting in touch!