Skip to content

Commit

Permalink
Merge pull request #123 from MV10/ms_config_v2
Browse files Browse the repository at this point in the history
Support MS Config v2, fixes #108, #111, #99
  • Loading branch information
MV10 authored Jul 9, 2018
2 parents 331fe7d + c05b658 commit b27c618
Show file tree
Hide file tree
Showing 14 changed files with 696 additions and 106 deletions.
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -183,3 +183,4 @@ UpgradeLog*.htm
FakesAssemblies/
/.vs/serilog-sinks-mssqlserver/v15/sqlite3
/.vs/serilog-sinks-mssqlserver/v15/Server/sqlite3
/.vs
152 changes: 132 additions & 20 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,20 +3,21 @@
A Serilog sink that writes events to Microsoft SQL Server. While a NoSql store allows for more flexibility to store the different kinds of properties, it sometimes is easier to use an already existing MS SQL server. This sink will write the logevent data to a table and can optionally also store the properties inside an Xml column so they can be queried.

**Package** - [Serilog.Sinks.MSSqlServer](http://nuget.org/packages/serilog.sinks.mssqlserver)
| **Platforms** - .NET 4.5 and .NET Standard 2.0

From version 5.2 and up, this sink also support the Audit capabilities.
| **Platforms** - .NET Framework 4.5 and .NET Standard 2.0

## Configuration

At minimum a connection string and table name are required.
At minimum a connection string and table name are required.

To use a connection string from the `connectionStrings` section of your application config, specify its name as the value of the connection string.


To use a connection string from the `<connectionStrings>` element of your application config file, specify its name as the value of the connection string.
#### Code (.NET Framework)

#### Code
Older .NET Framework applications can use the `ConfigurationManager` API shown below. Newer .NET Framework applications (using a Framework version that is .NET Standard compliant) should use the _Microsoft.Extensions.Configuration_ version in the next section.

```csharp
var connectionString = @"Server=..."; // or the name of a connection string in your .config file
var connectionString = @"Server=..."; // or the name of a connection string in the app config
var tableName = "Logs";
var columnOptions = new ColumnOptions(); // optional
Expand All @@ -25,9 +26,30 @@ var log = new LoggerConfiguration()
.CreateLogger();
```

#### XML

If you are configuring Serilog with the `ReadFrom.AppSettings()` XML configuration support, you can use:
#### Code (.NET Standard / .NET Core)

The application configuration parameter is optional for .NET Standard libraries or .NET Core applications.

```csharp
var appSettings = new ConfigurationBuilder()
.SetBasePath(Directory.GetCurrentDirectory())
.AddJsonFile("appsettings.json")
.Build(); // more likely you will inject an IConfiguration reference
var connectionString = @"Server=..."; // or the name of a connection string in the app config
var tableName = "Logs";
var columnOptions = new ColumnOptions(); // optional
var log = new LoggerConfiguration()
.WriteTo.MSSqlServer(connectionString, tableName, appConfiguration: appSettings, columnOptions: columnOptions)
.CreateLogger();
```


#### Serilog AppSettings package (.NET Framework)

.NET Framework libraries or applications can call `ReadFrom.AppSettings()` to configure Serilog using the [_Serilog.Settings.AppSettings_](https://github.com/serilog/serilog-settings-appsettings) package. This will apply configuration parameters from the `app.config` or `web.config` file:

```xml
<add key="serilog:using:MSSqlServer" value="Serilog.Sinks.MSSqlServer" />
Expand All @@ -36,6 +58,30 @@ If you are configuring Serilog with the `ReadFrom.AppSettings()` XML configurati
<add key="serilog:write-to:MSSqlServer.autoCreateSqlTable" value="true"/>
```


#### Serilog Configuration package (.NET Standard / .NET Core)

.NET Standard libraries and .NET Core applications can call `ReadFrom.Configuration(IConfiguration)` to configure Serilog using the [_Serilog.Settings.Configuration_](https://github.com/serilog/serilog-settings-configuration) package (version [**3.0.0-dev-00111**](https://www.nuget.org/packages/Serilog.Settings.Configuration/3.0.0-dev-00111) or newer). This will apply configuration parameters from the application configuration (not only `appsettings.json` as shown here, but any other valid `IConfiguration` source):


```json
{
"Serilog": {
"Using": ["Serilog.Sinks.MSSqlServer"],
"MinimumLevel": "Debug",
"WriteTo": [
{ "Name": "MSSqlServer",
"Args": {
"connectionString": "Server...",
"tableName": "Logs"
}
}
]
}
}
```


## Table definition

You'll need to create a table like this in your database:
Expand Down Expand Up @@ -75,15 +121,15 @@ If you set the `autoCreateSqlTable` option to `true`, the sink will create a tab

## Standard columns

The "standard columns" used by this sink (apart from obvious required columns like Id) are described by the StandardColumn enumeration and controlled by `columnOptions.Store`.
The "standard columns" used by this sink (apart from obvious required columns like Id) are described by the StandardColumn enumeration and controlled through code by the `columnOptions.Store` collection.

By default (and consistent with the SQL command to create a table, above) these columns are included:
- StandardColumn.Message
- StandardColumn.MessageTemplate
- StandardColumn.Level
- StandardColumn.TimeStamp
- StandardColumn.Exception
- StandardColumn.Properties
- `StandardColumn.Message`
- `StandardColumn.MessageTemplate`
- `StandardColumn.Level`
- `StandardColumn.TimeStamp`
- `StandardColumn.Exception`
- `StandardColumn.Properties`

You can change this list, as long as the table definition is consistent:

Expand Down Expand Up @@ -122,14 +168,14 @@ The log event properties `User` and `Other` will now be placed in the correspond

#### Excluding redundant items from the Properties column

By default, additional properties will still be included in the XML data saved to the Properties column (assuming that is not disabled via the `columnOptions.Store` parameter). This is consistent with the idea behind structured logging, and makes it easier to convert the log data to another (e.g. NoSQL) storage platform later if desired.
By default, additional properties will still be included in the data saved to the XML Properties or JSON LogEvent column (assuming one or both are enabled via the `columnOptions.Store` parameter). This is consistent with the idea behind structured logging, and makes it easier to convert the log data to another (e.g. NoSQL) storage platform later if desired.

However, if necessary, then the properties being saved in their own columns can be excluded from the XML. Use the `columnOptions.Properties.ExcludeAdditionalProperties` parameter in the sink configuration to exclude the redundant properties from the XML.
However, if necessary, then the properties being saved in their own columns can be excluded from the data. Use the `columnOptions.Properties.ExcludeAdditionalProperties` parameter in the sink configuration to exclude the redundant properties from the XML.


### XML configuration for columns
### Columns defined by AppSettings (.NET Framework)

Columns can be defined with the name and data type of the column in SQL Server. Columns specified must match database table exactly. DataType is case sensitive, based on SQL type (excluding precision/length).
Custom columns can be defined with the name and data type of the column in SQL Server. Columns specified must match database table exactly. DataType is case sensitive, based on SQL type (excluding precision/length). This section will be processed automatically if it exists in the application's `web.config` or `app.config` file.

```xml
<configSections>
Expand All @@ -144,6 +190,72 @@ Columns can be defined with the name and data type of the column in SQL Server.
</MSSqlServerSettingsSection>
```

### ColumnOptions defined by Configuration (.NET Standard / .NET Core)

For projects using the Serilog Configuration package, most properties of the `ColumnOptions` object are configurable. (The only property not currently supported is the filter-predicate `columnOptions.Properties.PropertyFilter`).

The equivalent of adding custom columns as shown in the .NET Framework example above looks like this:

```json
{
"Serilog": {
"Using": ["Serilog.Sinks.MSSqlServer"],
"MinimumLevel": "Debug",
"WriteTo": [
{ "Name": "MSSqlServer",
"Args": {
"connectionString": "Server...",
"tableName": "Logs",
"columnOptionsSection": {
"customColumns": [
{ "ColumnName": "EventType", "DataType": "int", "AllowNull": false },
{ "ColumnName": "Release", "DataType": "varchar", "DataLength": 32 }
]
}
}
}
]
}
}
```

As the name suggests, `columnOptionSection` is an entire configuration section in its own right. All possible entries and some sample values are shown below. All properties and subsections are optional.

```json
"columnOptionsSection": {
"addStandardColumns": [ "LogEvent" ],
"removeStandardColumns": [ "MessageTemplate", "Properties" ],
"customColumns": [
{ "ColumnName": "EventType", "DataType": "int", "AllowNull": false },
{ "ColumnName": "Release", "DataType": "varchar", "DataLength": 32 }
],
"disableTriggers": true,
"id": { "columnName": "Id" },
"level": { "columnName": "Level", "storeAsEnum": false },
"properties": {
"columnName": "Properties",
"excludeAdditionalProperties": true,
"dictionaryElementName": "dict",
"itemElementName": "item",
"omitDictionaryContainerElement": false,
"omitSequenceContainerElement": false,
"omitStructureContainerElement": false,
"omitElementIfEmpty": true,
"propertyElementName": "prop",
"rootElementName": "root",
"sequenceElementName": "seq",
"structureElementName": "struct",
"usePropertyKeyAsElementName": false
},
"timeStamp": { "columnName": "Timestamp", "convertToUtc": true },
"logEvent": { "columnName": "LogEvent", "excludeAdditionalProperties": true }
"message": { "columnName": "Message" },
"exception": { "columnName": "Exception" },
"messageTemplate": { "columnName": "MessageTemplate" },
}
```


### Options for serialization of the log event data

#### JSON (LogEvent column)
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,82 @@
using System;
using System.Data;

namespace Serilog.Configuration
{
/// <summary>
/// Converts the SQL data types of custom columns specified
/// in app configuration to the equivalent .NET types
/// </summary>
public static class ConvertSqlDataType
{
/// <summary>
/// Converts the SQL data types of custom columns specified
/// in app configuration to the equivalent .NET types
/// </summary>
/// <param name="sqlDataType">The SQL data type of custom columns from app configuration</param>
/// <param name="length">A length applied to types that accept a length</param>
/// <returns></returns>
public static DataColumn GetEquivalentType(string sqlDataType, int length = 0)
{
DataColumn c = new DataColumn();
switch(sqlDataType)
{
case "bigint":
c.DataType = typeof(long);
break;
case "varbinary":
case "binary":
if(length == 0) throw new ArgumentException($"SQL {sqlDataType} column requires a non-zero length argument.");
c.DataType = Type.GetType("System.Byte[]");
c.ExtendedProperties["DataLength"] = length;
break;
case "bit":
c.DataType = typeof(bool);
break;
case "char":
case "nchar":
case "ntext":
case "nvarchar":
case "text":
case "varchar":
if(length == 0) throw new ArgumentException($"SQL {sqlDataType} column requires a non-zero length argument.");
c.DataType = Type.GetType("System.String");
c.MaxLength = length;
break;
case "date":
case "datetime":
case "datetime2":
case "smalldatetime":
c.DataType = typeof(DateTime);
break;
case "decimal":
case "money":
case "numeric":
case "smallmoney":
c.DataType = typeof(Decimal);
break;
case "float":
c.DataType = typeof(double);
break;
case "int":
c.DataType = typeof(int);
break;
case "real":
c.DataType = typeof(float);
break;
case "smallint":
c.DataType = typeof(short);
break;
case "time":
c.DataType = typeof(TimeSpan);
break;
case "uniqueidentifier":
c.DataType = typeof(Guid);
break;
default:
throw new ArgumentException($"SQL {sqlDataType} is not a recognized or supported column data-type.");
}
return c;
}
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
// Copyright 2015 Serilog Contributors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

using System.Text.RegularExpressions;

namespace Serilog.Configuration
{
/// <summary>
/// Details of an individual column described in the app config.
/// </summary>
public class Column
{
private string _dataType = "varchar";
private const string ValidSqlDataTypes = "(bigint)|(bit)|(binary)|(varbinary)|(char)|(date)|(datetime)|(datetime2)|(decimal)|(float)|(int)|(money)|(nchar)|(ntext)|(numeric)|(nvarchar)|(real)|(smalldatetime)|(smallint)|(smallmoney)|(text)|(time)|(uniqueidentifier)|(varchar)";

/// <summary>
/// The name of a custom SQL Server column.
/// </summary>
public string ColumnName { get; set; }

/// <summary>
/// The data type of a custom SQL Server column.
/// </summary>
public string DataType
{
get => _dataType;
set
{
if(Regex.Match(value, ValidSqlDataTypes).Success)
{
_dataType = value;
}
else
{
// If validation fails, ignore the value, which matches the behavior
// of the .NET Framework [RegexStringValidator] attribute used by the
// original version of the SQL sink project.
}
}
}

/// <summary>
/// The size of certain SQL Server column types such as varchar;
/// </summary>
public int DataLength { get; set; } = 0;

/// <summary>
/// Controls whether the column is nullable
/// </summary>
public bool AllowNull { get; set; } = true;
}
}
Loading

0 comments on commit b27c618

Please sign in to comment.