OpenTelemetry - NullReferenceException

[ealsur]

Using SDK 3.35.1-preview, we have 2 reports of NullReferenceException:

System.NullReferenceException
  HResult=0x80004003
  Message=Object reference not set to an instance of an object.
  Source=Microsoft.Azure.Cosmos.Client
  StackTrace:
   at Microsoft.Azure.Cosmos.Tracing.TraceWriter.WriteTraceDatum(IJsonWriter writer, Object value)
   at Microsoft.Azure.Cosmos.Tracing.TraceWriter.TraceJsonWriter.WriteTrace(IJsonWriter writer, ITrace trace, Boolean isRootTrace)
   at Microsoft.Azure.Cosmos.Tracing.TraceWriter.WriteTrace(IJsonWriter writer, ITrace trace)
   at Microsoft.Azure.Cosmos.Diagnostics.CosmosTraceDiagnostics.WriteTraceToJsonWriter(JsonSerializationFormat jsonSerializationFormat)
   at Microsoft.Azure.Cosmos.Diagnostics.CosmosTraceDiagnostics.ToJsonString()
   at Microsoft.Azure.Cosmos.Diagnostics.CosmosTraceDiagnostics.ToString()
   at Microsoft.Azure.Cosmos.CosmosException.GetMessageHelper(HttpStatusCode statusCode, Headers headers, String responseBody, CosmosDiagnostics diagnostics)raceWriterTrac
   at Microsoft.Azure.Cosmos.CosmosException.<>c__DisplayClass2_0.<.ctor>b__0()
   at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
   at System.Lazy`1.ExecutionAndPublication(LazyHelper executionAndPublication, Boolean useDefaultConstructor)
   at System.Lazy`1.CreateValue()
   at Microsoft.Azure.Cosmos.CosmosException.get_Message()
   at Microsoft.Azure.Cosmos.CosmosException.RecordOtelAttributes(CosmosException exception, DiagnosticScope scope)
   at Microsoft.Azure.Cosmos.Telemetry.OpenTelemetryCoreRecorder.<>c.<.cctor>b__22_3(Exception exception, DiagnosticScope scope)
   at Microsoft.Azure.Cosmos.Telemetry.OpenTelemetryCoreRecorder.IsExceptionRegistered(Exception exception, DiagnosticScope scope)
   at Microsoft.Azure.Cosmos.Telemetry.OpenTelemetryCoreRecorder.MarkFailed(Exception exception)
   at Microsoft.Azure.Cosmos.ClientContextCore.<RunWithDiagnosticsHelperAsync>d__40`1.MoveNext()
   at Microsoft.Azure.Cosmos.ClientContextCore.<OperationHelperWithRootTraceAsync>d__30`1.MoveNext()
   at .....

 

  This exception was originally thrown at this call stack:
    Microsoft.Azure.Cosmos.Tracing.TraceWriter.WriteTraceDatum(Microsoft.Azure.Cosmos.Json.IJsonWriter, object)
    Microsoft.Azure.Cosmos.Tracing.TraceWriter.TraceJsonWriter.WriteTrace(Microsoft.Azure.Cosmos.Json.IJsonWriter, Microsoft.Azure.Cosmos.Tracing.ITrace, bool)
    Microsoft.Azure.Cosmos.Tracing.TraceWriter.WriteTrace(Microsoft.Azure.Cosmos.Json.IJsonWriter, Microsoft.Azure.Cosmos.Tracing.ITrace)
    Microsoft.Azure.Cosmos.Diagnostics.CosmosTraceDiagnostics.WriteTraceToJsonWriter(Microsoft.Azure.Cosmos.Json.JsonSerializationFormat)
    Microsoft.Azure.Cosmos.Diagnostics.CosmosTraceDiagnostics.ToJsonString()
    Microsoft.Azure.Cosmos.Diagnostics.CosmosTraceDiagnostics.ToString()
    Microsoft.Azure.Cosmos.CosmosException.GetMessageHelper(System.Net.HttpStatusCode, Microsoft.Azure.Cosmos.Headers, string, Microsoft.Azure.Cosmos.CosmosDiagnostics)

Another example:

at Microsoft.Azure.Cosmos.Tracing.TraceWriter.WriteTraceDatum(IJsonWriter writer, Object value)
   at Microsoft.Azure.Cosmos.Tracing.TraceWriter.TraceJsonWriter.WriteTrace(IJsonWriter writer, ITrace trace, Boolean isRootTrace)
   at Microsoft.Azure.Cosmos.Tracing.TraceWriter.WriteTrace(IJsonWriter writer, ITrace trace)
   at Microsoft.Azure.Cosmos.Diagnostics.CosmosTraceDiagnostics.WriteTraceToJsonWriter(JsonSerializationFormat jsonSerializationFormat)
   at Microsoft.Azure.Cosmos.Diagnostics.CosmosTraceDiagnostics.ToJsonString()
   at Microsoft.Azure.Cosmos.Diagnostics.CosmosTraceDiagnostics.ToString()
   at Microsoft.Azure.Cosmos.CosmosException.GetMessageHelper(HttpStatusCode statusCode, Headers headers, String responseBody, CosmosDiagnostics diagnostics)
   at Microsoft.Azure.Cosmos.CosmosException.<>c__DisplayClass2_0.<.ctor>b__0()
   at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
   at System.Lazy`1.ExecutionAndPublication(LazyHelper executionAndPublication, Boolean useDefaultConstructor)
   at System.Lazy`1.CreateValue()
   at Microsoft.Azure.Cosmos.CosmosException.get_Message()
   at Microsoft.Azure.Cosmos.CosmosException.RecordOtelAttributes(CosmosException exception, DiagnosticScope scope)
   at Microsoft.Azure.Cosmos.Telemetry.OpenTelemetryCoreRecorder.<>c.<.cctor>b__22_3(Exception exception, DiagnosticScope scope)
   at Microsoft.Azure.Cosmos.Telemetry.OpenTelemetryCoreRecorder.IsExceptionRegistered(Exception exception, DiagnosticScope scope)
   at Microsoft.Azure.Cosmos.Telemetry.OpenTelemetryCoreRecorder.MarkFailed(Exception exception)
   at Microsoft.Azure.Cosmos.ClientContextCore.<RunWithDiagnosticsHelperAsync>d__40`1.MoveNext()
   at Microsoft.Azure.Cosmos.ClientContextCore.<OperationHelperWithRootTraceAsync>d__30`1.MoveNext()
   at CFPullModelAllVersionsAndDeletesMode.Program.<ReadChangeFeed>d__16.MoveNext() in C:\Users\rodimiroc\OneDrive - Microsoft\Documents\Downloads\FFCF-solution2\cosmosdb-changefeed\Program.cs:line 211

The problem happens consistently. The way to repro it is generate any error condition, for example, attempt to Create a document that contains a ttl property with null value.

public class ReproItem
{
    public string id {get;set;} = "id";

    public int? ttl {get;set}
}

Which generates a CosmosException, catch the CosmosException and try to print the Diagnostics.ToString()

Workaround currently is to disable Open Telemetry integration: CosmosClientOptions.IsDistributedTracingEnabled = false;

[alexaloni]

We are using System.Text.Json serializer with cosmos client , if we comment JsonIgnore attribute on TTL property (example of attribute below), we get 100 % repro of null ref exception with default IsDistributedTracingEnabled being set to true.
[JsonPropertyName( "ttl")]
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
public int? TimeToLive { get; set; }

[sourabh1007]

@alexaloni Thanks for reporting this. but I am not able replicate it locally. Can you please provide the code snippet to repro this error.

[aavasthy]

@alexaloni I am able to replicate the issue and currently working on the fix. Meanwhile, the workaround is to either disable the IsDistributedTracingEnabled flag or if the flag is on then either subscribe to an operation level activity or network level activity.