Event Hubs lease lock exception showing as warnings / exceptions

[jeffhollan]

We should change Microsoft.Azure.EventHubs.RecieverDisconnectedException to info

Very similar to this issue #1784 - maybe it was already covered. Have some customers who are concerned that they see "New receiver with higher epoch...." exceptions in app insights. I assumed this would have been covered in #1784 but just got a new report last night from an MVP

/cc @paulbatum who was on the thread with customer

[jeffhollan]

Looks like it's also related to this one #1947. I'm not sure if we need to make a change for the v2 libraries and right now only special handling the v1 assembly type? Either way bumping this up to P0 as this has come up in support stand-ups twice this week.

I think the right way forward is to show these as "info" logs

[jeffhollan]

Just had a new ticket pop up. This one is the following stack trace and is a v2 app

Stack trace

Microsoft.ServiceBus.Messaging.LeaseLostException: Exception of type 'Microsoft.ServiceBus.Messaging.LeaseLostException' was thrown. ---> Microsoft.WindowsAzure.Storage.StorageException: The remote server returned an error: (409) Conflict. ---> System.Net.WebException: The remote server returned an error: (409) Conflict. at Microsoft.WindowsAzure.Storage.Shared.Protocol.HttpResponseParsers.ProcessExpectedStatusCodeNoException[T](HttpStatusCode expectedStatusCode, HttpStatusCode actualStatusCode, T retVal, StorageCommandBase`1 cmd, Exception ex) in c:\\Program Files (x86)\\Jenkins\\workspace\\release_dotnet_master\\Lib\\Common\\Shared\\Protocol\\HttpResponseParsers.Common.cs:line 55 at Microsoft.WindowsAzure.Storage.Blob.CloudBlob.<>c__DisplayClass33.b__32(RESTCommand`1 cmd, HttpWebResponse resp, Exception ex, OperationContext ctx) in c:\\Program Files (x86)\\Jenkins\\workspace\\release_dotnet_master\\Lib\\ClassLibraryCommon\\Blob\\CloudBlob.cs:line 3183 at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.EndGetResponse[T](IAsyncResult getResponseResult) in c:\\Program Files (x86)\\Jenkins\\workspace\\release_dotnet_master\\Lib\\ClassLibraryCommon\\Core\\Executor\\Executor.cs:line 299 --- End of inner exception stack trace --- at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.EndExecuteAsync[T](IAsyncResult result) in c:\\Program Files (x86)\\Jenkins\\workspace\\release_dotnet_master\\Lib\\ClassLibraryCommon\\Core\\Executor\\Executor.cs:line 50 at Microsoft.WindowsAzure.Storage.Blob.CloudBlob.EndRenewLease(IAsyncResult asyncResult) in c:\\Program Files (x86)\\Jenkins\\workspace\\release_dotnet_master\\Lib\\ClassLibraryCommon\\Blob\\CloudBlob.cs:line 1979 at Microsoft.WindowsAzure.Storage.Core.Util.AsyncExtensions.<>c__DisplayClass4.b__3(IAsyncResult ar) in c:\\Program Files (x86)\\Jenkins\\workspace\\release_dotnet_master\\Lib\\ClassLibraryCommon\\Core\\Util\\AsyncExtensions.cs:line 114 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.ServiceBus.Messaging.BlobLeaseManager.d__24.MoveNext() --- End of inner exception stack trace --- at Microsoft.ServiceBus.Messaging.BlobLeaseManager.d__24.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.ServiceBus.Messaging.BlobLeaseManager.d__25.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Azure.WebJobs.ServiceBus.EventHubListener.Checkpointer.d__0.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Azure.WebJobs.ServiceBus.EventHubListener.EventProcessor.d__11.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Azure.WebJobs.ServiceBus.EventHubListener.EventProcessor.d__10.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.ServiceBus.Common.TaskHelpers.EndAsyncResult(IAsyncResult asyncResult) at Microsoft.ServiceBus.Messaging.IteratorAsyncResult`1.<>c.b__24_1(TIteratorAsyncResult thisPtr, IAsyncResult r) at Microsoft.ServiceBus.Messaging.IteratorAsyncResult`1.StepCallback(IAsyncResult result)

[brettsam]

Looking back through the history here, it looks like we addressed this in WebJobs v2.x with the code here: https://github.com/Azure/azure-webjobs-sdk/blob/v2.x/src/Microsoft.Azure.WebJobs.ServiceBus/MessagingExceptionHandler.cs#L120-L127.

But with the EH/SB APIs changing as we moved to WebJobs v3.x, that detail was lost. I'm going to do something similar for this here.

FYI -- read this explanation for why these exceptions are (generally) normal: https://stackoverflow.com/a/41867611/3516125.