New sandbox decoders for Microsft IIS and ULS log formats

[dkolli]

We have written new sandbox decoders in Lua for Microsoft Internet Information Server (IIS) and Sharepoint ULS logs for our needs. Would you be interested in reviewing these and including in the core product if useful.

[rafrombrc]

We'll get less GC churn if we define msg outside the process_message function, like so:

local msg = {
    Timestamp = nil,
    Payload = nil,
    Hostname = nil,
    Type = "iis",
    Fields = nil,
}

[rafrombrc]

First, I'm terribly sorry for taking to long to review this code. Thank you for your contribution! This looks very useful, and overall things look pretty good. In addition to the inline comments I made, it would be great if you could add an entry to the CHANGES.txt changelog file, and extra great if you could add the little pieces needed for the decoders to show up in the documentation.

To add them to the docs, you just add a stub .rst file for each decoder (this one for the nginx access log decoder is a good example) and then add them to the index.rst and index_noref.rst index pages.

Thanks again for your work!

[dkolli]

Thank you Rob for taking time to review these decoders and provide your feedback. I'll make all suggested code and documentation changes asap.

[dkolli]

I just committed all changes to the decoders and the docs. I hope i didn't miss any that we discussed above.

[rafrombrc]

Everything looks good with the changes you made, thanks! Sorry to move the goalposts, but in addition to Trink's point above about making the extract_quote function local in your decoders, there's one more comment I forgot to make the first time around.

The other HTTP server log decoders (nginx, apache are all using a similar message schema to represent each HTTP request handled by the server. This is helpful because then those messages are in the format expected by the HTTP Status Graph filter. It would be great if you could match that message schema, then the Status Graph filter would also work for graphing and anomaly detection based on IIS server logs.

[dkolli]

There are some fields like substatus are unique to the IIS logs. Looks like status field is a match. There are other fields like remote ip etc. aren't matching on names at this point. I'll take a look at field by field and try to map the field names with nginx format. I'm assuming this what you are expecting by matching this with nginx/apache, correct?

[rafrombrc]

@dkolli Yes, that's what I meant. I've actually done most of this work, as well as added some tests for the decoders. You can see what I've done in the iis_uls_decoders branch that I just pushed up to our repo. I'm just trying to track down some actual IIS and ULS log files so I can run them through to make sure I didn't break anything in the process of making my changes.

Or maybe you could grab my branch and try them out on your log files?