[BUG] EventHub Error occurred while refreshing token that is not retriable. Not scheduling refresh task

[vimorra]

Describe the bug
Seems that there are issues in the refersh of token into the class com.azure.core.amqp.implementation.ActiveClientTokenManager. This happen after 1hour of run of a stress test session, this cause the failure of the pods and reduce the consumer rate from the EH.
The code is based on:

com.azure
azure-messaging-eventhubs
5.3.1

Environment Context:
• Dedicated EH with 1CU
• AKS with Node-pool of 12 Nodes(F16S_v2) with max 22 pods per node
• Each Pod establish a connection with EH, consuming from dedicated partitions

Exception or Stack Trace

[2020-11-25 08:24:15,990][parallel-1][ERROR][com.azure.core.amqp.implementation.ActiveClientTokenManager]: Error occurred while refreshing token that is not retriable. Not scheduling refresh task. Use ActiveClientTokenManager.authorize() to schedule task again. audience[amqp://stresstest1125-eastus2-ehns.servicebus.windows.net/activity-stream/ConsumerGroups/sdk-etl-filter/Partitions/85] scopes[amqp://stresstest1125-eastus2-ehns.servicebus.windows.net/activity-stream/ConsumerGroups/sdk-etl-filter/Partitions/85]
ReactorDispatcher instance is closed. (ClientLogger.java:347)
[2020-11-25 08:24:15,990][parallel-1][ERROR][com.azure.core.amqp.implementation.ActiveClientTokenManager]: Error occurred while refreshing token that is not retriable. Not scheduling refresh task. Use ActiveClientTokenManager.authorize() to schedule task again. audience[amqp://stresstest1125-eastus2-ehns.servicebus.windows.net/activity-stream/ConsumerGroups/sdk-etl-filter/Partitions/85] scopes[amqp://stresstest1125-eastus2-ehns.servicebus.windows.net/activity-stream/ConsumerGroups/sdk-etl-filter/Partitions/85]
ReactorDispatcher instance is closed. (ClientLogger.java:347)
[2020-11-25 08:24:46,158][parallel-1][ERROR][com.azure.core.amqp.implementation.ReactorDispatcher]: ReactorDispatcher instance is closed. (ClientLogger.java:347)
[2020-11-25 08:24:46,158][parallel-1][ERROR][com.azure.core.amqp.implementation.ReactorDispatcher]: ReactorDispatcher instance is closed. (ClientLogger.java:347)
[2020-11-25 08:24:46,159][parallel-1][ERROR][com.azure.core.amqp.implementation.ActiveClientTokenManager]: Error occurred while refreshing token that is not retriable. Not scheduling refresh task. Use ActiveClientTokenManager.authorize() to schedule task again. audience[amqp://stresstest1125-eastus2-ehns.servicebus.windows.net/activity-stream/ConsumerGroups/sdk-etl-filter/Partitions/122] scopes[amqp://stresstest1125-eastus2-ehns.servicebus.windows.net/activity-stream/ConsumerGroups/sdk-etl-filter/Partitions/122]
ReactorDispatcher instance is closed. (ClientLogger.java:347)
[2020-11-25 08:24:46,159][parallel-1][ERROR][com.azure.core.amqp.implementation.ActiveClientTokenManager]: Error occurred while refreshing token that is not retriable. Not scheduling refresh task. Use ActiveClientTokenManager.authorize() to schedule task again. audience[amqp://stresstest1125-eastus2-ehns.servicebus.windows.net/activity-stream/ConsumerGroups/sdk-etl-filter/Partitions/122] scopes[amqp://stresstest1125-eastus2-ehns.servicebus.windows.net/activity-stream/ConsumerGroups/sdk-etl-filter/Partitions/122]
ReactorDispatcher instance is closed. (ClientLogger.java:347)

To Reproduce
Steps to reproduce the behavior:

Code Snippet

Below the implementation of the connection with EH

EventProcessorClientBuilder eventProcessorClientBuilder = new EventProcessorClientBuilder()
.connectionString(ACTIVITY_EVENTHUB_CONNECTION_STR)
.consumerGroup(CONSUMER_GROUP_ID)
.processEventBatch(PARTITION_BATCH_PROCESS, 15, Duration.ofSeconds(30))
.processError(ERROR_HANDLER)
.partitionOwnershipExpirationInterval(Duration.ofSeconds(150))
.checkpointStore(new BlobCheckpointStore(blobContainerAsyncClient)); 

EventProcessorClient eventProcessorClient = eventProcessorClientBuilder
.buildEventProcessorClient(); 

eventProcessorClient.start(); while(eventProcessorClient.isRunning()) {​​​​​     Thread.sleep(1000); }

Expected behavior
The expected behavior is that the token is refreshed and than is possible to use the same connection without open a new one.

[YijunXieMS]

This doesn't happen anymore after AKS pod config change.

[YijunXieMS]

@vimorra Can we close this issue since it doesn't reproduce any more after AKS pod config change?
"ReactorDispatcher instance is closed" causes the refreshing token error. In theory the ReactorDispatcher isn't closed until the client is closed or the thread that runs the dispatching is interrupted.

[vimorramsk]

Yes I confirm, we can close this issue at the moment. The test environment was not configured properly in term for resources allocation and K8S rise OOM to close the POD.

[YijunXieMS]

Thanks @vimorra