EventProcessorHost Exception: A receiver with a higher epoch

[charlesw23]

SUMMARY:
When an EventProcessorHost is holding open a connection and the Receiver with a higher epoch and another instance of the webjob tries to connect. An exception gets thrown internally that does not bubble up. The EventProcessorHost does not receive any events due to the error and then the [EventHubTrigger] does not fire

REPRO:

1. In azure Create the following resources: EventHub, StorageAccount1, StorageAccount2, ApplicationServicePlan with your Webjob

2. Create a consumer group in your EventHub

3. Create an application that implements EventProcessorHost using StorageAccount1.

Program.cs

public static void Setup()
        {
_eventProcessorHost = new EventProcessorHost(eventProcessorHostName, EventHubName, ConsumerGroupName, EvnetHubConnectionString, eventProcessorStorageString);

JobHostConfig.DashboardConnectionString = storageConnectionStirng; 
JobHostConfig.StorageConnectionString = storageConnectionStirng;

 EventHubConfig.AddEventProcessorHost("EventHubListener", _eventProcessorHost);
 JobHostConfig.UseEventHub(EventHubConfig);
 JobHostConfig.Tracing.ConsoleLevel = TraceLevel.Verbose;  //want to see all the things
 var host = new JobHost(JobHostConfig);
 host.RunAndBlock();
}

Functions.cs

 public static async Task Trigger([EventHubTrigger("EventHubListener")] EventData[] inEventDatas)
{
var SetBreakPointHere = 1;
}

3. Deploy Application to Azure and Run Against your EventHub

4. In your local Dev environment change the application to use StorageAccount2 for all of the things.

5. Run the application in your local dev environment and create a breakpoint at the following line in functions.cs
   var SetBreakPointHere = 1;

6. Send Events to the EventHub

note: you can get the Exceptions that bubble up out of the event hub libs by attaching to the event hub with ServiceBusExplorer.exe found

ServiceBusExplorer

Expected behavior

Application should bubble up the following error for the partition listeners

<14:00:15> Exception: A receiver with a higher epoch '14084' already exists. A new receiver with epoch 0 cannot be created. Make sure you are creating receiver with increasing epoch value to ensure connectivity, or ensure all old epoch receivers are closed or disconnected. TrackingId:ddd20f17-3daf-4d6b-a146-9d9b49e65196_B21, SystemTracker:dec2017:eventhub:dec2017sub18431|consumergroup, Timestamp:2/15/2018 7:00:14 PM Reference:3547ee92-3e03-4419-8361-712f96e50f48, TrackingId:f6d2380f-9496-400d-942a-89707c7dcff3_B21, SystemTracker:dec2017:eventhub:dec2017sub18431|consumergroup, Timestamp:2/15/2018 7:00:14 PM TrackingId:937226b8a5c24c80a3c546862c303f68_G17, SystemTracker:gateway2, Timestamp:2/15/2018 7:00:14 PM

Actual behavior

Nothing, no error break point does not fire

Related information

Provide any related information

Feel free to contact me if you need help with reproduction of this bug.

[brettsam]

@pragnagopa, I know you have a PR out for bubbling up exceptions in EventHubs. Will this be handled there?

[charlesw23]

It may be the EventProcessorHost itself that is swallowing the exception. I was talking with someone familiar with high throughput eventhubs. He recommended trying the new microsoft.azure.eventhubs.processor namespace. Said "you should be switching to the new namespace ASAP it handles exceptions better and surfaces them", thought I would pass that along.

[pragnagopa]

Yes. This should will be addressed in V2 as we already moved to using microsoft.azure.eventhubs.processor.

[pragnagopa]

Fix is checked in. Issue #1604 tracks testing the logging works if there are errors in evenprocessorhost

[mathewc]

Issue and associated fix for logging was here: #1705