Intermittent "System.InvalidOperationException: Collection was modified; enumeration operation may not execute." When Reading A Document With a Dictionary Property (SDK 3.17.1)

[colton-manville]

Describe the bug
Intermittent InvalidOperationException when reading a document that has a Dictionary property.

Exception:
System.InvalidOperationException: Collection was modified; enumeration operation may not execute. at System.ThrowHelper.ThrowInvalidOperationException(ExceptionResource resource) at System.Collections.Generic.List.Enumerator.MoveNextRare() at System.String.Join(String separator, IEnumerable values) at Microsoft.Azure.Documents.Collections.DictionaryNameValueCollection.CompositeValue.get_Value() at Microsoft.Azure.Documents.Collections.DictionaryNameValueCollection.Get(String key) at Microsoft.Azure.Documents.StoreClient.UpdateResponseHeader(DocumentServiceRequest request, INameValueCollection headers) at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()

To Reproduce
We are reading a document that has a Dictionary property. This happens intermittently.

Expected behavior
Exception should not be thrown when reading a document.

Actual behavior
Intermittent InvalidOperationException when reading a document that has a Dictionary property.

Environment summary
SDK Version: 3.17.1
OS Version: Windows

Additional context
System.InvalidOperationException: Collection was modified; enumeration operation may not execute. at System.ThrowHelper.ThrowInvalidOperationException(ExceptionResource resource) at System.Collections.Generic.List.Enumerator.MoveNextRare() at System.String.Join(String separator, IEnumerable values) at Microsoft.Azure.Documents.Collections.DictionaryNameValueCollection.CompositeValue.get_Value() at Microsoft.Azure.Documents.Collections.DictionaryNameValueCollection.Get(String key) at Microsoft.Azure.Cosmos.Common.SessionContainer.SetSessionToken(SessionContainerState self, DocumentServiceRequest request, INameValueCollection responseHeaders) at Microsoft.Azure.Documents.StoreClient.CaptureSessionToken(Nullable statusCode, SubStatusCodes subStatusCode, DocumentServiceRequest request, INameValueCollection headers) at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Azure.Cosmos.RequestHandler.<SendAsync>d__6.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Azure.Cosmos.Handlers.AbstractRetryHandler.<ExecuteHttpRequestAsync>d__2.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Azure.Cosmos.Handlers.AbstractRetryHandler.<SendAsync>d__1.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Azure.Cosmos.RequestHandler.<SendAsync>d__6.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Azure.Cosmos.RequestHandler.<SendAsync>d__6.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler.<SendAsync>d__6.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler.<SendAsync>d__8.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Azure.Cosmos.ContainerCore.<ProcessItemStreamAsync>d__81.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Azure.Cosmos.ContainerCore.<ReadItemAsync>d__56.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Azure.Cosmos.ClientContextCore.<RunWithDiagnosticsHelperAsync>d__38.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Azure.Cosmos.ClientContextCore.<OperationHelperWithRootTraceAsync>d__29.MoveNext()

[j82w]

Hi @colton-manville,

What consistency are you using, and which operations is this being hit on?

[colton-manville]

Strong consistency.

We are doing a ReadItemAsync() call. I thought I copied the part of the call stack with the cosmos API we were calling. I will edit the bug report with the full call stack starting at the CosmosDB SDK API we called.

--- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Azure.Cosmos.ContainerCore.<ReadItemAsync>d__56.MoveNext()

[j82w]

I believe I have the issue fixed. There was a background refresh that was reusing the same request object as the original task. The request object is not thread safe. The background refresh was modifying the headers at the same time the original task was leading to this issue. The fix should be included in the next release.

[colton-manville]

Great, thanks so much.

[j82w]

@xinlian12 please upgrade to the latest 3.18.0 release.