Change default Etw mode to EtwSelfDescribingEventFormat from EtwManifestEventFormat

[benaadams]

Description

There is a startup cost from generating manifest data for Etw #49362

However is trying to optimizing it a wasted effort when it is an older format?

e.g. Switching to EtwSelfDescribingEventFormat #49409 by default removes these allocations entirely and it probably the format where the effort to optimize should be focused?

From #27621 (comment)

ManifestBuilder functionality is largely a back-compat artifact for correct interoperation with legacy win7 and win8 era ETW readers. For any new usage he recommends customers create EventSource with the Self describing events.

and #27621 (comment)

If you wish to use complex types (that is something besides primitives, strings and DateTime), then you need to use the EtwSelfDescribingFormat flag when creating the EventSouce. This is very easy. In your TestEventSource class simpy add

private TestEventSource() : base(EventSourceSettings.EtwSelfDescribingEventFormat) { }

Things should work from there. We recommend people do this in general. We would have made this the default, except is is a breaking change for people using EventSource in manifest based scenarios.

The docs do also suggest Self Describing is preferred? https://docs.microsoft.com/en-us/windows/win32/tracelogging/trace-logging-about

TraceLogging is the new Windows 10 event tracing for user-mode applications and kernel-mode drivers. TraceLogging is a format for self-describing Event Tracing for Windows (ETW). TraceLogging builds on Event Tracing for Windows (ETW) and provides a simplified way to instrument code.

TraceLogging offers the simplicity of Windows software trace preprocessor (WPP) with the added benefit of being able to include structured data with events, the capability of correlating events, and all without requiring a separate instrumentation manifest XML file. Events are self-describing which means that a binary containing the instrumentation manifest does not need to be registered on the system in order to use the Trace Data Helper (TDH) APIs to decode and show events.
...
TraceLogging builds upon ETW but introduces several important improvements:

• Tracing an event is as simple as an API call.
• Events are self-describing and do not require any additional binaries containing the compiled event manifest to interpret the event. All the metadata about the event is recorded in the event.
• Activities inside a single process can be easily expressed through start and stop events.
• TraceLogging is compatible with existing instrumentation support. The new ETW APIs continue to support the old providers. You can invest in the new ETW APIs for strategic scenarios while leaving your old instrumentation points as they are.
• TraceLogging offers the same event tracing API for Windows 10, Xbox One, and Windows 10 Mobile.
• TraceLogging APIs are accessible from C, C++, .NET, and Windows Runtime.

[ghost]

Tagging subscribers to this area: @tarekgh, @tommcdon, @pjanotti
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

There is a startup cost from generating manifest data for Etw #49362

However is trying to optimizing it a wasted effort when it is an older format?

e.g. Switching to EtwSelfDescribingEventFormat #49409 by default removes these allocations entirely and it probably the format where the effort to optimize should be focused?

From #27621 (comment)

ManifestBuilder functionality is largely a back-compat artifact for correct interoperation with legacy win7 and win8 era ETW readers. For any new usage he recommends customers create EventSource with the Self describing events.

and #27621 (comment)

If you wish to use complex types (that is something besides primitives, strings and DateTime), then you need to use the EtwSelfDescribingFormat flag when creating the EventSouce. This is very easy. In your TestEventSource class simpy add

private TestEventSource() : base(EventSourceSettings.EtwSelfDescribingEventFormat) { }

Things should work from there. We recommend people do this in general. We would have made this the default, except is is a breaking change for people using EventSource in manifest based scenarios.

The docs do also suggest Self Describing is preferred? https://docs.microsoft.com/en-us/windows/win32/tracelogging/trace-logging-about

TraceLogging is the new Windows 10 event tracing for user-mode applications and kernel-mode drivers. TraceLogging is a format for self-describing Event Tracing for Windows (ETW). TraceLogging builds on Event Tracing for Windows (ETW) and provides a simplified way to instrument code.

TraceLogging offers the simplicity of Windows software trace preprocessor (WPP) with the added benefit of being able to include structured data with events, the capability of correlating events, and all without requiring a separate instrumentation manifest XML file. Events are self-describing which means that a binary containing the instrumentation manifest does not need to be registered on the system in order to use the Trace Data Helper (TDH) APIs to decode and show events.
...
TraceLogging builds upon ETW but introduces several important improvements:

• Tracing an event is as simple as an API call.
• Events are self-describing and do not require any additional binaries containing the compiled event manifest to interpret the event. All the metadata about the event is recorded in the event.
• Activities inside a single process can be easily expressed through start and stop events.
• TraceLogging is compatible with existing instrumentation support. The new ETW APIs continue to support the old providers. You can invest in the new ETW APIs for strategic scenarios while leaving your old instrumentation points as they are.
• TraceLogging offers the same event tracing API for Windows 10, Xbox One, and Windows 10 Mobile.
• TraceLogging APIs are accessible from C, C++, .NET, and Windows Runtime.

Author:	benaadams
Assignees:	-
Labels:	area-System.Diagnostics.Tracing, tenet-performance, untriaged
Milestone:	-

[KalleOlaviNiemitalo]

"breaking change for people using EventSource in manifest based scenarios" would mean software that writes events to Windows Event Log, as in #27002.

[benaadams]

would mean software that writes events to Windows Event Log

For Windows 7 and Server 2008 R2 or earlier; however for them mainstream support ended on January 13, 2015 and Extended support ended January 14, 2020. Does .NET 6.0; not released for another 8 months, need to maintain compat with them?

[KalleOlaviNiemitalo]

Doesn't Windows Event Log in Windows 10 still require the manifest-based format? IIRC, the self-describing format always uses channel 11 (so cannot distinguish between Admin and Operational channels) and assigns event IDs at run time (so you cannot define reliable triggers for the events).

runtime/src/libraries/System.Private.CoreLib/src/System/Diagnostics/Tracing/TraceLogging/Statics.cs

Line 24 in 60dc59b

public const byte TraceLoggingChannel = 0xb;

[benaadams]

IIRC, the self-describing format always uses channel 11

Is that a pre-Windows 10 era artefact? https://docs.microsoft.com/en-us/windows/win32/api/traceloggingprovider/nf-traceloggingprovider-traceloggingchannel

Though suggests some extra work is needed to update TraceLogging in dotnet/runtime rather than just flipping the enum

Remarks

Channels are used in advanced Event Tracing for Windows (ETW) scenarios such as writing to system-defined event consumers such as the event log. When using channels, a manifest must be registered with the system to define the provider and its channels. A manifest for a TraceLogging provider should define the provider and the channels but should omit the event definitions, since they are managed by TraceLogging.

The eventChannel parameter must be a compile-time constant 0 to 255. If no TraceLoggingChannel(n) arg is provided, the default channel level is 11 (WINEVENT_CHANNEL_TRACELOGGING), indicating that this is a normal TraceLogging event. If multiple TraceLoggingChannel(n) args are provided, the value from the last TraceLoggingChannel(n) is used.

Warning If your provider will run on Windows earlier than Windows 10, do not use TraceLoggingChannel. For an event to be recognized as TraceLogging-compatible, it must either have the channel set to 11 or it must have been marked by the ETW runtime during EventWrite. The Windows 10 ETW runtime will recognize and mark TraceLogging events regardless of channel, but earlier versions of Windows require a system update in order to support marking TraceLogging events. For events captured on systems without an update, the channel is the only way to recognize a TraceLogging event, so events with other channels may be harder to decode.

[KalleOlaviNiemitalo]

EventSource reads EventAttribute.Message and copies the text to the instrumentation manifest. The Message Compiler then generates the resources that can be linked into a DLL, registered with Windows Event Log, and displayed by Event Viewer. AFAIK, the self-describing event format does not include these messages.

Hypothetically, it might be possible to generate and register the instrumentation manifest with the message strings but write the events themselves in the self-describing format. However, if the self-describing format uses event names rather than stable IDs, I doubt that Windows Event Log would be able to locate the correct event definitions in the manifest.

Related links, although these are rather old and would not know if Windows Event Log has been improved since then:

• The EventSource NuGet package and support for the Windows Event Log (Channel Support) (2013-08-10)
• Announcing the EventSource NuGet Package – Write to the Windows Event Log (2013-08-09)

[noahfalk]

@benaadams - I see this has gone quiet for a few months, were you still pursuing this? I got the impression you and @josalem were trying to pursue source generators instead so maybe this was moot?

[benaadams]

@benaadams - I see this has gone quiet for a few months, were you still pursuing this? I got the impression you and @josalem were trying to pursue source generators instead so maybe this was moot?

Think it could possibility be breaking and source generators would reduce the cost; as would VS not listening to verbose events machine wide, which I think was a different issue