-
Notifications
You must be signed in to change notification settings - Fork 1.4k
How to use structured logging
NLog 4.5 introduces structured logging - also called semantic logging. This document describes how to use structural logging. The implementation in NLog supports the message templates syntax, so it is recommended to check: https://messagetemplates.org/
Structured logging makes it easier to store and query log-events, as the logevent message-template and message-parameters are preserved, instead of just transforming them into a formatted message.
The normal .NET string.Format(...)
will only accept input strings like this:
logger.Info("Logon by user:{0} from ip_address:{1}", "Kenny", "127.0.0.1");
logger.Debug("Shopitem:{0} added to basket by user:{1}", "Jacket", "Kenny");
When storing these log-events in a database (or somewhere else), then it can be difficult to query all actions performed by a certain user. Also it could be hard to group similar events.
The workaround would then be to perform RegEx-queries to recognize logevent messages and convert them into a more structured format, and register which parameter index is the user. The RegEx might even have to extract the needed details from the formatted message, making it even more fragile. Maintaining these RegEx-queries can become rather cumbersome.
Further history: messagetemplates.org
NLog has always supported log-event metadata called event-properties, but it requires a little effort to generate a log-event with properties. See also the Fluent-Logger-API
NLog 4.5 makes it possible to capture log-event-properties from the log-message-template, so they can be easily processed by the NLog destination target:
logger.Info("Logon by {user} from {ip_address}", "Kenny", "127.0.0.1"); // Logon by "Kenny" from "127.0.0.1"
logger.Debug("{shopitem} added to basket by {user}", new { Id=6, Name = "Jacket", Color = "Orange" }, "Kenny");
Any NLog destination target that is able to handle log-event-properties will automatically experience the benefit of doing structured logging.
The formatting of message depends on the datatype of the parameters.
Example:
logger.Info("Order {orderid} created for {user}", 42, "Kenny");
Will be formatted as:
Order 42 created for "Kenny"
The formatting is controlled by the datatype of the parameter:
- string: surrounded with double quotes, e.g
"hello"
- number: no quotes
- null: printed as
NULL
- list/ienumerable/array:
"item1", "item2"
- dictionary:
"key1"="value1", "key2"="value2"
- objects:
ToString()
It's possible to control formatting by preceding @
or $
:
-
@
will format the object as JSON -
$
forcesToString()
Object o = null;
logger.Info("Test {value1}", o); // null case. Result: Test NULL
logger.Info("Test {value1}", new DateTime(2018,03, 25)); // datetime case. Result: Test 25-3-2018 00:00:00 (locale TString)
logger.Info("Test {value1}", new List<string> { "a", "b" }); // list of strings. Result: Test "a", "b"
logger.Info("Test {value1}", new[] { "a", "b" }); // array. Result: Test "a", "b"
logger.Info("Test {value1}", new Dictionary<string, int> { { "key1", 1 }, { "key2", 2 } }); // dict. Result: Test "key1"=1, "key2"=2
var order = new Order
{
OrderId = 2,
Status = OrderStatus.Processing
};
logger.Info("Test {value1}", order); // object Result: Test MyProgram.Program+Order
logger.Info("Test {@value1}", order); // object Result: Test {"OrderId":2, "Status":"Processing"}
logger.Info("Test {value1}", new { OrderId = 2, Status = "Processing"}); // anonymous object. Result: Test { OrderId = 2, Status = Processing }
logger.Info("Test {@value1}", new { OrderId = 2, Status = "Processing"}); // anonymous object. Result: Test {"OrderId":2, "Status":"Processing"}
When formatting the log message, then it will automatically serialize the parameters (see Formatting of the message). It is also possible to configure NLog Layout to serialize the individual properties. This can be done by using ${event-properties} or ${all-event-properties}, for example:
<target xsi:type="File" name="jsonFile" fileName="c:\temp\nlog-json-${shortdate}.log">
<layout>${longdate}|${level}|${logger}|${message}|${all-event-properties}{exception:format=tostring}</layout>
</target>
NLog Layouts with option includeEventProperties="true"
(Before NLog 5.0 it was includeAllProperties="true"
) will output captured properties from structured logging:
It can be configured like this:
<target xsi:type="File" name="jsonFile" fileName="c:\temp\nlog-json-${shortdate}.log">
<layout xsi:type="JsonLayout" includeEventProperties="true">
<attribute name="time" layout="${longdate}" />
<attribute name="level" layout="${level:upperCase=true}"/>
<attribute name="message" layout="${message}" />
</layout>
</target>
NLog 4.5 adds support for rendering the raw message-template, instead of just the formatted message:
NLog 4.5 extends the following NLog LayoutRenderers with support for serializing with property reflection into JSON when format="@"
- ${event-properties}
- ${exception}
- ${gdc} - Global Context
- ${scopeproperty} - Thread Context (Before NLog 5.0 it was ${mdc} + ${mdlc})
In code you can use LogEventInfo.Properties
. If having special interest in format-options for the captured parameters then one can use LogEventInfo.MessageTemplateParameters
Any NLog destination target that handles event-properties will be able to take advantage of structured logging. Many targets already has support for the option includeEventProperties="true"
, that comes automatically when inheriting from TargetWithContext.
NLog Targets with supports for NLog Layout as payload will automatically support structured logging:
- NLogViewer Target makes use of Log4JXml Layout.
- WebService Target can perform JSON-post with Json Layout.
- Network Target can publish both Json Layout and Xml Layout to any TCP/UDP-endpoint.
- Database Target can write to an XML column using Xml Layout
There are already many custom NLog targets that provide the ability to store log-events in central storage, and allow a Web-Application to query the log-events along with their event-properties. These targets will automatically benefit from structured logging, and allow the Web-Application to perform effective queries.
NLog 4.6.3 allows you to navigate into a complex object and extract a single property. Ex:
logger.Debug("{shopitem} added to basket by {user}", new { Id=6, Name = "Jacket", Color = "Orange" }, "Kenny");
Then one can extract specific details from shopItem
-property:
<layout xsi:type="JsonLayout">
<attribute name="shopitemId" layout="${event-properties:shopitem:objectPath=Id}" />
</layout>
To avoid object reflection for a custom type, then just implement the IFormattable-interface, and NLog will call ToString() instead of doing reflection of properties.
NLog 4.7 allows you to transform large (or dangerous) objects into a more reduced/safe object:
LogManager.Setup().SetupSerialization(s =>
s.RegisterObjectTransformation<System.Net.WebException>(ex => new {
Type = ex.GetType().ToString(),
Message = ex.Message,
StackTrace = ex.StackTrace,
Source = ex.Source,
InnerException = ex.InnerException,
Status = ex.Status,
Response = ex.Response.ToString(), // Call your custom method to render stream as string
})
);
This can also be used to output object-fields, as the NLog Object Reflection only sees object-properties. By transforming into an Anonymous Type, and assigning the wanted object-fields to named properties.
If more dynamic reflection is required, then one can also just return a Dictionary<string, object>
instead of returning an Anonymous Type (The performance hit is a little higher).
Note when making an override for an object-type, then the returned object-type must be of a different object-type. Because NLog caches the reflection of the return object-type, and afterwards NLog skips checking for transformation for already cached object-types.
Create a new class that NLog.IValueFormatter
and set NLog.Config.ConfigurationItemFactory.Default.ValueFormatter
You can override the builtin NLog JSON converter with a different implementation.
You need a custom NLog.IJsonConverter
and set NLog.Config.ConfigurationItemFactory.Default.JsonConverter
Example of using JSON.NET that supports JSON.NET object annotations like [JsonIgnore]
:
internal class JsonNetSerializer : NLog.IJsonConverter
{
readonly JsonSerializerSettings _settings;
public JsonNetSerializer()
{
_settings = new JsonSerializerSettings
{
Formatting = Formatting.Indented,
ReferenceLoopHandling = ReferenceLoopHandling.Ignore
};
}
/// <summary>Serialization of an object into JSON format.</summary>
/// <param name="value">The object to serialize to JSON.</param>
/// <param name="builder">Output destination.</param>
/// <returns>Serialize succeeded (true/false)</returns>
public bool SerializeObject(object value, StringBuilder builder)
{
try
{
var jsonSerializer = JsonSerializer.CreateDefault(_settings);
var sw = new System.IO.StringWriter(builder, System.Globalization.CultureInfo.InvariantCulture);
using (var jsonWriter = new JsonTextWriter(sw))
{
jsonWriter.Formatting = jsonSerializer.Formatting;
jsonSerializer.Serialize(jsonWriter, value, null);
}
}
catch (Exception e)
{
NLog.Common.InternalLogger.Error(e, "Error when custom JSON serialization");
return false;
}
return true;
}
}
e.g. logging "Hello {0} with {Message}"
For backward compatibility (and performance), then NLog will by default skip full parsing if it detects the first parameter as being positional index. This will ensure repeated positional-placeholders supported by string.Format
will continue to work:
string.Format("{1}{0}{2}{0}", Environment.NewLine, "Hello", "World");
When parseMessageTemplates='true'
then NLog will always parse all parameters, and if one is non-numeric, then all the parameters will be treated as structured parameters. Mixing and numeric and structured parameters is not recommended, as it hurts performance (backtrack) and numeric parameters are most of the time not so descriptive.
The Names of the parameters should be unique
e.g. Order {orderId}
Use ${message:raw=true}
NLog will by default attempt to parse log-events as structured log-events. This gives a minor overhead, that will not be noticable by most.
It is possible to disable this behavior, telling NLog it should not attempt to parse log-events as structured log-events, with this xml-setting:
<nlog parseMessageTemplates="false">
...
</nlog>
- Troubleshooting Guide - See available NLog Targets and Layouts: https://nlog-project.org/config
- Getting started
- How to use structured logging
- Troubleshooting
- FAQ
- Articles about NLog
-
All targets, layouts and layout renderers
Popular: - Using NLog with NLog.config
- Using NLog with appsettings.json