Unhandled NullReference exception at PassThroughProcessor.Process crashes application

[rikbosch]

Hi,

We've had an application crash, with a stackstrace coming from application insights PassThroughProcessor:

(source: windows event-log)

Application: Ao.TnT.Grains.Host.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: System.NullReferenceException
   at Microsoft.ApplicationInsights.Shared.Extensibility.Implementation.PassThroughProcessor.Process(Microsoft.ApplicationInsights.Channel.ITelemetry)
   at Microsoft.ApplicationInsights.WindowsServer.TelemetryChannel.SamplingTelemetryProcessor.Process(Microsoft.ApplicationInsights.Channel.ITelemetry)
   at Microsoft.ApplicationInsights.Extensibility.PerfCounterCollector.QuickPulse.QuickPulseTelemetryProcessor.Process(Microsoft.ApplicationInsights.Channel.ITelemetry)
   at Ao.Shared.Grains.Host.ApplicationInsights.Filters.SqlDependencyFilter.Process(Microsoft.ApplicationInsights.Channel.ITelemetry)
   at Microsoft.ApplicationInsights.TelemetryClient.Track(Microsoft.ApplicationInsights.Channel.ITelemetry)
   at Ao.Shared.Grains.Host.ApplicationInsights.AITelemetryConsumer.TrackTrace(System.String, Orleans.Runtime.Severity, System.Collections.Generic.IDictionary`2<System.String,System.String>)
   at Orleans.Runtime.LoggerImpl.TrackTrace(System.String, Orleans.Runtime.Severity)
   at Orleans.Runtime.LoggerImpl.WriteLogMessage(Int32, Orleans.Runtime.Severity, System.String, System.Object[], System.Exception)
   at Orleans.Runtime.LoggerImpl.Log(Int32, Orleans.Runtime.Severity, System.String, System.Object[], System.Exception)
   at Orleans.Runtime.AsynchAgent.AgentThreadProc(System.Object)
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Threading.ThreadHelper.ThreadStart(System.Object)

We have 2 applications reporting with the same insights key.

We're running several instances of these applications in a Service Fabric cluster. The weird thing is that all of the instances crashed, reporting the same stacktrace at the same time in our cluster.

Any idea's what could be causing this?

Version Info

SDK Version : Microsoft.ApplicationInsights.WindowsServer 2.5.0
.NET Version : 4.6.2
How Application was onboarded with SDK(VisualStudio/StatusMonitor/Azure Extension) :
OS : Windows Server 2016
Hosting Info (IIS/Azure WebApps/ etc) : ServiceFabric

[lmolkova]

Duplicate: microsoft/ApplicationInsights-dotnet#707

@rikbosch

This likely happens when TelemetryConfiguration got disposed. It happens when TelemetryConfiguration/TelemetryClient and other AI primitives are added as transient in DI frameworks - they should always be singletons.
It may also happen if TelemetryConfiguration.Active is disposed.

[Update] When DI host is disposed, it causes TelemteryConfiguration to be disposed as well. If some telemetry is reported after that (and it could happen because of async nature of tracked operations), calling Track would result in the similar exception.
Note: some operations are tracked automatically and may throw.

Several questions:

• do you use ApplicationInsights-ServiceFabric SDK by chance?
• do you have ApplicationInsights.config?
• otherwise, how do you configure AI?

https://github.com/Microsoft/ApplicationInsights-dotnet/blob/develop/src/Microsoft.ApplicationInsights/Extensibility/Implementation/PassThroughProcessor.cs#L26

        public void Process(ITelemetry item)
        {
            this.sink.Process(item);
        }

calls https://github.com/Microsoft/ApplicationInsights-dotnet/blob/develop/src/Microsoft.ApplicationInsights/Extensibility/TelemetrySink.cs#L206

        public void Process(ITelemetry item)
        {
            this.TelemetryProcessorChain.Process(item);
        }

And TelemetryProcessorChain is null when TelemetryConfiguration is disposed.

We will add the check to prevent NRE, but it would just ignore the call, without reporting telemetry.

I noticed orleans in stack traces, but I don't see if it is ever disposed configuration: https://github.com/dotnet/orleans/blob/0de4ec1d3e74d2e6baddfc7936eb0a0c0a050d47/src/TelemetryConsumers/Orleans.TelemetryConsumers.AI/AITelemetryConsumer.cs