Release new version of ECS.NET 8.6.0

[Mpdreamz]

• sanity check the process field set (@gregkalapos)
• Extend the code generator to include richer meta information in xml documentation. (@Mpdreamz )
• Merge Further Elastic.Ingest Improvements #110 as it has a lot of integration tests using the new types. (@gregkalapos)
• We will aim for an 8.4.0-alpha first to validate our build tooling. (@Mpdreamz)
• Include direct support for ECS fields in message templates (@Mpdreamz) Set ECS fields from message templates #229
• Utilize labels for value types and metadata for objects. (@Mpdreamz) Set ECS fields from message templates #229
• Do not assume we can safely serialize to metadata [BUG] MetaDataDictionaryConverter - NullReferenceException #219 (@Mpdreamz)
• Bump to 8.5/8.6 if that bump is not complicated, prioritize releasing (@Mpdreamz)
• Breaking changes documentation. Include an paragraph describing what's expected to break with this new major version and why. (@Mpdreamz)

[bonyaroslav]

@Mpdreamz @gregkalapos Any news when we can use it? We're waiting for many months already

[odin568]

Same for me. Eager to use updated version to get rid of some workarounds

[lewinskimaciej]

Is this package still being maintained? We desperately need #167 but it seems that there hasn't been a release in 1,5 years. Is there something we can do to help push this out?

[Mpdreamz]

Still maintained folks! Thanks for the nudges for an update.

I have been on leave for an extended period of time but am back working on getting an update out the door. Any feedback on the upcoming pre-release would be most welcomed!

Will comment on this issue when it goes up on nuget.

[Mpdreamz]

Heads up here that we just shipped version 8.4.0-alpha1 to nuget.

I intend to wait two weeks to collect feedback on this new version before shipping it as a GA release (8.4.0).

A LOT of changes have gone in since we last release 1.6.0-alpha1, you can see the full list of changes here

Here are some highlights:

New integrations

• Elasticsearch.Extensions.Logging an integration with Microsoft.Extensions.Logging that writes logs directly to Elasticsearch or Elastic Cloud (thank you @sgryphon for all the help)
• Elastic.CommonSchema.Serilog.Sink a Serilog sink that writes ecs logs directly to Elasticsearch or Elastic Cloud
• Elastic.CommonSchema.Log4Net Log4Net integration that logs to files in ecs-logging format. (thank you @andreycha!)
• Elastic.Ingest.Elasticsearch.CommonSchema a specialization of Elastic.Ingest.Elasticsearch to push data to datastreams indices that is fully ECS aware and can bootstrap the target datastreams/indices confirming ECS mappings.

Big Updates:

• Completely rewrote the code generator from scratch to accomodate the ECS spec format changes since 1.6 to 8.x.
• Spun off the ingest components into their own seperate libraries: elastic/elastic-ingest-dotnet
• Elastic.CommonSchema was completely rearchitected as we rewritten the code generator from scratch to align with the ECS specification changes.
  • EcsDocument now has an AssignField(path, value) method that can dynamically assign ECS fields based on path. This allows all the structured logging integrations to directly assign ECS fields too.
  • Through CreateNewWithDefaults all logging integrations share a common base default for how ecs documents get created and enriched.
• NLog integration has seen continued improvements from @snakefoot (Can't thank you enough for all your efforts!)
• Moved CI back onto Github Actions (thank you @v1v).

Breaking Changes.

• The root object for ECS documents in Elastic.CommonSchema was renamed from Base to EcsDocument.

[HubDevUser]

Can you please include .netstandard2.0 in the official 8.4.0 release. We need to consume this in fullframework, the alpha1 release only contains .netstandard2.1 assemblies @Mpdreamz

[reydelleon-skuvault]

@Mpdreamz Are there plans to update documentation to show how to put all these together? For example, how to enhance the EcsDocument with custom fields.

I noticed that you're putting forward an alternative to Serilog.Sinks.Elasticsearch. it would be helpful to see some examples of how you would configure your sink to achieve a similar experience (knowing that they don't have the same features) than that of the contrib sink. Many of us have been using that other sink for a long time and would like to make any possible migration as painless as possible.

[Mpdreamz]

@reydelleon-skuvault absolutely, docs are a big prerequisite for the 8.4.0 release.

I tried to capture some of the main differences between the sinks here for now: https://github.com/elastic/ecs-dotnet/tree/main/src/Elastic.CommonSchema.Serilog.Sink#comparison-with-serilogsinkselasticsearch

Any feedback or additional questions you may have are more than welcomed!

[Mpdreamz]

A quick extremely raw example of custom EcsDocument implementation is here: https://github.com/elastic/ecs-dotnet/blob/main/src/Elastic.CommonSchema.BenchmarkDotNetExporter/Domain/BenchmarkDocument.cs#L16

[HubDevUser]

How can i set a custom index, it always defaults to ecs-dotnet-logs
?

[Mpdreamz]

How can i set a custom index, it always defaults to ecs-dotnet-logs ?

Using what integration?

[HubDevUser]

Using aspnet core similar to the example you added earlier examples/aspnetcore-with-extensions-logging

[Mpdreamz]

Using aspnet core similar to the example you added earlier examples/aspnetcore-with-extensions-logging

If you are using Elasticsearch.Extensions.Logging you can configure it in appsettings.json

{
  "Logging": {
    "Elasticsearch": {
      "ShipTo": {
        "NodePoolType": "Static",
        "NodeUris": [ "http://localhost:9200" ]
      },
      "DataStream": {
        "DataSet": "my.application"
      }
    },
    "LogLevel" : {
      "Default": "Trace",
      "Microsoft": "Warning"
    }
  }
}

This will log to logs-my.application-default.

You can also configure this in the application builder.

builder.Host.ConfigureLogging((_, loggingBuilder) =>
{
	loggingBuilder.AddElasticsearch(opts =>
	{
		opts.DataStream = new DataStreamNameOptions
		{
			Type = "logs", DataSet = "dotnet", Namespace = "default"
		}
	}),

[Mpdreamz]

Can you please include .netstandard2.0 in the official 8.4.0 release. We need to consume this in fullframework, the alpha1 release only contains .netstandard2.1 assemblies @Mpdreamz

I've just pushed 8.4.0-alpha2 with netstandard2.0 support.

[HubDevUser]

It doesn't contain a .netstandard2.0 under libs, can you take a look please ?

[reydelleon-skuvault]

@Mpdreamz I'm trying to use the new sink but I'm not seeing a way to specify auth credentials when configuring it. There is nothing about this in the README and looking through the code I wasn't able to identify a way to do this.

Can you provide some guidance on this and perhaps add something in the examples in the README?

[lewinskimaciej]

@Mpdreamz I too can't seem to find a way of setting auth on new Serilog Elastic sink. I think this would be achievable by exposing setter for ElasticsearchSinkOptions.Transport or some other, maybe nicer way. Currently it's always DefaultHttpTransport with default TransportConfiguration.

[HubDevUser]

@reydelleon-skuvault @lewinskimaciej see #267 (comment) for example on setting up the auth options

[HubDevUser]

@Mpdreamz how can we configure for .netfullframework where ILogger is not an option?

[LiorBanai]

@HubDevUser you can use ILogger in net framework using the nuget:

		<PackageReference Include="Microsoft.Extensions.Logging.Abstractions" Version="7.0.0" />

[reydelleon-skuvault]

@reydelleon-skuvault @lewinskimaciej see #267 (comment) for example on setting up the auth options

@HubDevUser The example you linked is for the logging provider, not the sink. @Mpdreamz We need an example that works for the sink. Do we have one?

[reydelleon-skuvault]

Found the way to do it, but it is rather convoluted if all I want to change is the authentication.

var transportConfig = new TransportConfiguration(new Uri("http://localhost:9200/"));
transportConfig.Authentication(new BasicAuthentication("elastic", "some-password"));
var transport = new DefaultHttpTransport(transportConfig);
var sinkOptions = new ElasticsearchSinkOptions(transport)
{
	...
};
loggerConfiguration
	.WriteTo.Elasticsearch(sinkOptions);

@Mpdreamz Is there a better way to do this?

[Mpdreamz]

@reydelleon-skuvault thanks for bringing this to my attention I've opened #286 to address this usability issue.

With that you should be able to use:

.WriteTo.Elasticsearch(nodes, opts =>
{
	opts.BootstrapMethod = BootstrapMethod.Failure;
	opts.DataStream = new DataStreamName("logs", "console-example");

}, transport =>
{
	transport.Authentication();
})

[Mpdreamz]

I just pushed 8.4.0-alpha3 to nuget that includes netstandard2.0 for all projects and includes the fix to make it easier to configure the transport options when using the serilog sink.

Thanks everyone for kicking the tires!

[Mpdreamz]

I just pushed ECS.NET 8.4.0-alpha4 that includes everyones feedback and PR's much obliged to everyone kicking the tires!

I opened #291 to bump us to 8.6.0 and this includes some non breaking changes to how we bootstrap index templates. This PR will allow new version to upgrade template indices.

[reydelleon-skuvault]

@Mpdreamz Is there a good way to see logs about the sink activity? I have SelfLog enabled for Serilog and the sink configured but thought I'm not getting any data in Elasticsearch, there is no indication as to what is the cause for this. Nothing written to the self log file (is it supposed to?).

I have disabled pinging (a shot in the dark) but that didn't resolve it.

I do have logs in the Console and local file, so there is content. It is just not making it to Elasticsearch.

[Mpdreamz]

We are logging all export errors to Serilog's Self log:

ecs-dotnet/src/Elastic.CommonSchema.Serilog.Sink/ElasticsearchSink.cs

Line 77 in af53e3c

ExportResponseCallback = (response, _) =>

The following diagnostics method is still ugly and not intended to make it to 1.0 but we ship with something called a ChannelListener<> if you do the following.

	.WriteTo.Elasticsearch(....
    {
		ConfigureChannel = channelOpts =>  {
			channelOpts.BufferOptions = new BufferOptions { ExportMaxConcurrency = 10 };

			SomeStaticPlace = new ChannelListener<EcsDocument, BulkResponse>().Register(channelOpts);
		}
	})

What does SomeStaticPlace.ToString() report?

[reydelleon-skuvault]

@Mpdreamz This is what I get from the code you posted:

Failed publish over channel: ChannelListener.
Exported Buffers: 0
Exported Items: 0
Export Responses: 0
Export Retries: 0
Export Exhausts: 0
Inbound Buffer Read Loop Started: False
Inbound Buffer Publishes: 0
Inbound Buffer Publish Failures: 0
Outbound Buffer Read Loop Started: False
Outbound Buffer Read Loop Exited: False
Outbound Buffer Publishes: 0
Outbound Buffer Publish Failures: 0

Exception: 

That's it. No more details than that. Here is my setup (without the diagnostic code), in case I'm doing something wrong here:

loggerConfiguration
			.WriteTo.Elasticsearch<EnhancedEcsDocument>(
				new[] {new Uri("http://localhost:9200")},
				options =>
				{
					options.DataStream = new DataStreamName("logs", "myservice", environmentName.ToLowerInvariant());
					options.BootstrapMethod = BootstrapMethod.None ;
					};
				},
				transportConfig =>
					transportConfig
						.Authentication(new BasicAuthentication( "someUSer", "somePassword"))
						.DisablePing()
			);

[reydelleon-skuvault]

I ended up (successfully) shipping the logs to a different Elasticsearch server, which means two things: 1. the problem was the server I was trying to ship to and 2. The sink works as expected.

That said, I still don't know what the problem with the original server was because I didn't get anything in the Self log.

[Mpdreamz]

@reydelleon-skuvault Thanks for reporting back another instance works!

I would still love to improve the experience here and understand why the other instance does not work.

Can you run the following in a new console application? Of course using

ChannelListener<EnhancedEcsDocument, BulkResponse>? listener = null;
var waitHandle = new CountdownEvent(1);

// -- Setup Serilog --
var nodes = new[] { new Uri("http://localhost:9200") };
Log.Logger = new LoggerConfiguration()
	.MinimumLevel.Debug()
	.MinimumLevel.Override("Microsoft", LogEventLevel.Information)
	.Enrich.FromLogContext()
	.WriteTo.Elasticsearch<EnhancedEcsDocument>(nodes, opts =>
	{
		opts.BootstrapMethod = BootstrapMethod.None;
		opts.DataStream = new DataStreamName("logs", "console-example");
		opts.ConfigureChannel = channelOpts => {
			channelOpts.BufferOptions = new BufferOptions
			{
				ExportMaxConcurrency = 1,
				OutboundBufferMaxSize = 2,
				WaitHandle = waitHandle
			};
			listener = new ChannelListener<EnhancedEcsDocument, BulkResponse>().Register(channelOpts);
		};
	}, 
	transportConfig =>
		transportConfig
			.Authentication(new BasicAuthentication( "someUSer", "somePassword"))
			.DisablePing()
    )
	.CreateLogger();

// -- Log 2 items and wait for flush --
Log.Logger.Information("Writing event 1");
Log.Logger.Information("Writing event 2");

if (!waitHandle.WaitHandle.WaitOne(TimeSpan.FromSeconds(10)))
	throw new Exception($"No flush occurred in 10 seconds: {listener}", listener?.ObservedException);
else
{
	Console.WriteLine("Successfully indexed data to Elasticsearch");
	Console.WriteLine(listener);
}

Thanks in advance!

[reydelleon-skuvault]

@Mpdreamz This is what I get:

Successfully indexed data to Elasticsearch
Failed publish over channel: ChannelListener.
Exported Buffers: 1
Exported Items: 2
Export Responses: 1
Export Retries: 0
Export Exhausts: 0
Inbound Buffer Read Loop Started: True
Inbound Buffer Publishes: 2
Inbound Buffer Publish Failures: 0
Outbound Buffer Read Loop Started: True
Outbound Buffer Read Loop Exited: False
Outbound Buffer Publishes: 1
Outbound Buffer Publish Failures: 0

Exception: System.Exception: Unsuccessful () low level call on POST: /_bulk?filter_path=error%2C%20items.%2A.status%2Citems.%2A.error
 ---> System.Net.Http.HttpRequestException: An error occurred while sending the request.
 ---> System.IO.IOException: The response ended prematurely.
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendi
ngRequestsCts, CancellationToken originalCancellationToken)
   at Elastic.Transport.HttpTransportClient.RequestAsync[TResponse](RequestData requestData, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---


Process finished with exit code 0.

Once more, I think the issue has to do with my server setup in this case. There seems to be more info here than in my previous attempt. Note that although there are no publishing failures reported in the metrics, the logs in this case never made it to Elasticsearch. For comparison, here is the output using a different server, where the credentials are wrong.

Exception: System.Exception: Unsuccessful (401) low level call on POST: /_bulk?filter_path=error%2C%20items.%2A.status%2Citems.%2A.error
 ---> Elastic.Transport.TransportException: Could not authenticate with the specified node. Try verifying your credentials or check your Shield configuration. Call: Status code 401 from: POST /_bulk?fil
ter_path=error%2C%20items.%2A.status%2Citems.%2A.error
 ---> Elastic.Transport.PipelineException: Could not authenticate with the specified node. Try verifying your credentials or check your Shield configuration.
   at Elastic.Transport.DefaultRequestPipeline`1.ThrowBadAuthPipelineExceptionWhenNeeded(ApiCallDetails details, TransportResponse response)
   at Elastic.Transport.DefaultRequestPipeline`1.CallProductEndpointAsync[TResponse](RequestData requestData, CancellationToken cancellationToken)
   at Elastic.Transport.DefaultHttpTransport`1.RequestAsync[TResponse](HttpMethod method, String path, PostData data, RequestParameters requestParameters, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   --- End of inner exception stack trace ---

[UPDATE]
This is what I found when digging on the server side logs:

2023-04-03T16:46:23.723704200Z {"@timestamp":"2023-04-03T16:46:23.723Z", "log.level": "WARN", "message":"received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=redacted, remoteAddress=redacted}", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][transport_worker][T#10]","log.logger":"org.elasticsearch.xpack.security.transport.netty4.SecurityNetty4HttpServerTransport","elasticsearch.cluster.uuid":"2cKEp0AMR1GWWNSxsG4mmQ","elasticsearch.node.id":"fDnS7aKgRmWtqYHMLs6TZA","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"elastic-cluster"}

This seems to explain the exception outputted by the listener. There is still the fact that I'm not seeing any of this in the self-log, but that might also be an issue with my own setup.

[reydelleon-skuvault]

@Mpdreamz Is there documentation around SSL certificate configuration. What would be the best course of action if we're using a test Elasticsearch server (in Docker container, for example) that has a self signed certificate? Can I disable certificate validation on the sink for testing?

[reydelleon-skuvault]

@Mpdreamz I got another question about customizing the EcsDocument.

The examples I have found peppered in READMEs and tests all show contrived examples of how to subclass EcsDocument to introduce custom properties. They seem to suggest that you can do nesting (think the Contoso class with CompanyName as a nested property). In the TryRead, ReceiveProperty and WriteAdditionalProperties methods the examples show us handling the root property. Take this test for example: https://github.com/elastic/ecs-dotnet/blob/main/tests/Elastic.CommonSchema.Serilog.Tests/Repro/GithubIssue167.cs

Here is the Tryread() method for visibility:

protected override bool TryRead(string propertyName, out Type type)
{
	type = propertyName switch
	{
		"contoso" => typeof(Contoso),
		_ => null
	};
	return type != null;
}

The problem that I see with this is that I don't necessarily set the whole object in a LogEvent property. I might enrich the LogEvent with the contoso.company_name and set it to a single string value, but I will not set a contoso property in the LogEvent and set the value to the serialized contents of the object.

I'm confused as to how all this should work (adding custom properties to EcsDocument). The benchmark example doesn't use Serilog. Is it that this only works when using the Elastic agent directly, without Serilog? Does it work with Serilog at all? Maybe I have to use the MapCustom function when working with Serilog and do the mapping myself?

Perhaps what I'm missing is how things should look like in the Serilog LogEvent for this to work.

Any guidance on this would be appreciated, since I need to add properties (not in metadata/labels).

[Mpdreamz]

@Mpdreamz Is there documentation around SSL certificate configuration. What would be the best course of action if we're using a test Elasticsearch server (in Docker container, for example) that has a self signed certificate? Can I disable certificate validation on the sink for testing?

.WriteTo.Elasticsearch(nodes, opts =>
{
	//
}, transport =>
{
	transport.ServerCertificateValidationCallback(CertificateValidations.AllowAll);
})

[Mpdreamz]

Just a heads up that today we (finally) released a GA version of the ECS .NET libraries

New documentation: https://www.elastic.co/guide/en/ecs-logging/dotnet/current/intro.html

You should be able to upgrade to 8.6.0 today!

Thanks a ton for everyone involved, all the contributions, feedback and questions truly helped make this a mammoth of a release. It took a considerable time and effort to move us from 1.6.0-alpha1 to 8.6.0 with the main issue being specification format changes requiring us to retool our code generation from the ground up.

• Elastic.CommonSchema
  Strongly typed representation of the ECS specification. Includes serialization routines.

Formatters:

• Elastic.CommonSchema.Serilog
• Elastic.CommonSchema.NLog
  big shoutout to @snakefoot for the bulk of the contribution and acting as NLog expert on so many occasions
• Elastic.CommonSchema.Log4Net
  massive thank you to @andreycha for this contribution.

Datashippers:

• Elastic.Serilog.Sinks
• Elastic.Extensions.Logging
  Thank you @sgryphon for the initial contribution!
• Elastic.CommonSchema.BenchmarkDotNetExporter

Enrichers:

While all ECS logging libraries are Activity aware for distributed tracing, installing the following enrichers will use even richer tracing information from the Elastic APM Agent https://github.com/elastic/apm-agent-dotnet

• Elastic.Apm.SerilogEnricher
• Elastic.Apm.NLog

New architecture:

We laid out a completely new architecture for our datashippers in this repository now taking advantage of https://github.com/elastic/elastic-ingest-dotnet System.Threading.Channel backed abstractions to push buffered data to any datasource.

• Elastic.Ingest.Elasticsearch.CommonSchema

Is a dedicated ECS event data shipper that can bootstrap target datastreams/indices with the appropriate ECS component templates and settings.