[QUERY] Getting the following error from eventhub : Operation is not valid due to the current state of the object

[digeler]

Hi Team , we started to get the following exception when try to read from event hub :

2021-05-12 16:24:42.4493 Info EventHubClientFactory LogConnectionString got connection string for namespace [namespace=testdgeuse1inputeh]
2021-05-12 16:24:42.4503 Info ReadEventHubDataCmdlet RunAsync starting To Read
2021-05-12 16:24:42.9097 Error ReadEventHubDataCmdlet ProcessRecord System.InvalidOperationException: Operation is not valid due to the current state of the object.
at void Microsoft.Azure.Amqp.Transport.TransportStream.Flush()
at Task System.IO.Stream.FlushAsync(CancellationToken cancellationToken)+(object state) => { }
at void System.Threading.Tasks.Task.s_ecCallback(object obj)
at void System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, object state)
at void System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, object state)
at async Task System.Net.Security.SslStream.ForceAuthenticationAsync(TIOAdapter adapter, bool receiveFirst, byte[] reAuthenticationData, bool isApm)
at void System.Threading.Tasks.TaskToApm.End(IAsyncResult asyncResult)
at void System.Net.Security.SslStream.EndAuthenticateAsClient(IAsyncResult asyncResult)
at void Microsoft.Azure.Amqp.Transport.TlsTransport.HandleOpenComplete(IAsyncResult result, bool syncComplete)
at void Microsoft.Azure.Amqp.ExceptionDispatcher.Throw(Exception exception)
at TAsyncResult Microsoft.Azure.Amqp.AsyncResult.End(IAsyncResult result)
at void Microsoft.Azure.Amqp.AmqpObject+OpenAsyncResult.End(IAsyncResult result)
at void Microsoft.Azure.Amqp.AmqpObject.EndOpen(IAsyncResult result)
at void Microsoft.Azure.Amqp.Transport.TlsTransportInitiator.HandleTransportOpened(IAsyncResult result)
at void Microsoft.Azure.Amqp.Transport.TlsTransportInitiator.OnTransportOpened(IAsyncResult result)
at async Task Azure.Messaging.EventHubs.Amqp.AmqpConnectionScope.CreateAndOpenConnectionAsync(Version amqpVersion, Uri serviceEndpoint, EventHubsTransportType transportType, IWebProxy proxy, string scopeIdentifier, TimeSpan timeout)
at async Task Microsoft.Azure.Amqp.FaultTolerantAmqpObject.OnCreateAsync(TimeSpan timeout)
at async Task Microsoft.Azure.Amqp.Singleton.GetOrCreateAsync(TimeSpan timeout)
at async Task Microsoft.Azure.Amqp.Singleton.GetOrCreateAsync(TimeSpan timeout)
at async Task Azure.Messaging.EventHubs.Amqp.AmqpConnectionScope.OpenManagementLinkAsync(TimeSpan timeout, CancellationToken cancellationToken)
at async Task Microsoft.Azure.Amqp.FaultTolerantAmqpObject.OnCreateAsync(TimeSpan timeout)
at async Task Microsoft.Azure.Amqp.Singleton.GetOrCreateAsync(TimeSpan timeout)
at async Task Microsoft.Azure.Amqp.Singleton.GetOrCreateAsync(TimeSpan timeout)
at async Task Azure.Messaging.EventHubs.Amqp.AmqpClient.GetPropertiesAsync(EventHubsRetryPolicy retryPolicy, CancellationToken cancellationToken)
at async Task Azure.Messaging.EventHubs.Amqp.AmqpClient.GetPropertiesAsync(EventHubsRetryPolicy retryPolicy, CancellationToken cancellationToken)
at async Task Azure.Messaging.EventHubs.EventHubConnection.GetPropertiesAsync(EventHubsRetryPolicy retryPolicy, CancellationToken cancellationToken)
at async Task<string[]> Azure.Messaging.EventHubs.EventHubConnection.GetPartitionIdsAsync(EventHubsRetryPolicy retryPolicy, CancellationToken cancellationToken)
at async Task<string[]> Azure.Messaging.EventHubs.Consumer.EventHubConsumerClient.GetPartitionIdsAsync(CancellationToken cancellationToken)
at void Microsoft.test.Tools.EventHubConsumerClientExtension+<>c__DisplayClass1_0+<b__1>d.MoveNext() in C:/repos/oom-alerts-prom/test-testgon/src/Tools/Extensions/EventHubConsumerClientExtension.cs:line 48
at async Task<IReadOnlyCollection> Microsoft.test.Infrastructure.EnumerableExtension.SelectManyAsync<TSourceItem, TDestinationResult>(IEnumerable enumerable, Func<TSourceItem, Task<IEnumerable>> selectorAsync) in C:/repos/oom-alerts-prom/test-testgon/src/Infrastructure/Extensions/EnumerableExtension.cs:line 274
at async IAsyncEnumerable Microsoft.test.Tools.EventHubConsumerClientExtension.ReadEventsUntilEmptyAsync(IReadOnlyCollection eventHubClients, EventPosition eventPosition, ReadEventOptions readEventOptions, CancellationToken cancellationToken) in C:/repos/oom-alerts-prom/test-testgon/src/Tools/Extensions/EventHubConsumerClientExtension.cs:line 44
at async IAsyncEnumerable Microsoft.test.Tools.EventHubConsumerClientExtension.ReadEventsUntilEmptyAsync(IReadOnlyCollection eventHubClients, EventPosition eventPosition, ReadEventOptions readEventOptions, CancellationToken cancellationToken)
at TResult System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable+ConfiguredValueTaskAwaiter.GetResult()
at async ValueTask System.Linq.AsyncEnumerable+SelectEnumerableAsyncIterator<TSource, TResult>.MoveNextCore() in //Ix.NET/Source/System.Linq.Async/System/Linq/Operators/Select.cs:line 199
at TResult System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable+ConfiguredValueTaskAwaiter.GetResult()
at async ValueTask System.Linq.AsyncIteratorBase.MoveNextAsync() in //Ix.NET/Source/System.Linq.Async/System/Linq/AsyncIterator.cs:line 70
at async ValueTask System.Linq.AsyncIteratorBase.MoveNextAsync() in /_/Ix.NET/Source/System.Linq.Async/System/Linq/AsyncIterator.cs:line 75
at TResult System.Runtime.CompilerServices.ValueTaskAwaiter.GetResult()
at async Task Microsoft.test.Tools.ReadEventHubDataCmdlet.RunAsync() in C:/repos/oom-alerts-prom/test-testgon/src/Tools/Cmdlets/ClusterCmdlets/WorkerClusterCmdlets/ReadEventHubDataCmdlet.cs:line 145
at async Task Microsoft.test.Tools.ReadEventHubDataCmdlet.RunAsync() in C:/repos/oom-alerts-prom/test-testgon/src/Tools/Cmdlets/ClusterCmdlets/WorkerClusterCmdlets/ReadEventHubDataCmdlet.cs:line 145
at void Microsoft.test.Infrastructure.TaskExtension.Await(Task task) in C:/repos/oom-alerts-prom/test-testgon/src/Infrastructure/Extensions/TaskExtension.cs:line 20
at void Microsoft.test.Tools.ExtendedCmdlet.ProcessRecord() in C:/repos/oom-alerts-prom/test-testgon/src/Tools/Cmdlets/ExtendedCmdlet.cs:line 86
Read-testEventHubData: Operation is not valid due to the current state of the object.

We are using .netcore 3.1 I have seen a lot of issues in Gh regarding .net 5 but its not our case.

This is where the execption is thrown :

thanks

[jsquire]

This was resolved via internal mail thread; the underlying cause was most likely a transient network issue that caused the AMQP library to encounter an unexpected state. The v5.4.1 release of the Event Hubs SDK was enhanced with additional handling for special cases of InvalidOperationException encountered when AMQP connections/links are created.

@digeler was able to update and confirmed that moving to the newer SDK version resolved the issue. I'm going to close this out, but please feel free to reopen if you believe further discussion is needed.