LeaseLostException & WebException on Scale-Out

[m1nkeh]

Investigative information

Please provide the following:

• Timestamp: 30-07-2018 23:00:00 UTC
• Function App version (1.0 or 2.0-beta): 1.0
• Function App name: wbch-dev-core
• Function name(s) (as appropriate): Testmatcher / TestCleaner
• Invocation ID: N/A
• Region: North Europe

Repro steps

1. Create an empty Azure v1 Function w/ EventHub Trigger
2. Deploy to Consumption Plan
3. Provision EventHub with 32 partitions.
4. Load EventHub w/ 50,000 events
5. Watch scale-out raise exceptions for LeaseLostException, and WEbException repeatedly

Expected behavior

According to this post: https://stackoverflow.com/questions/51602965/azure-eventhub-function-leaselostexception#51603421

These should be tracked as warnings in the latest host deployment?

Actual behavior

Tracked as exceptions

Known workarounds

None

[m1nkeh]

7 days, any acknowledgement? :(

[tohling]

@m1nkeh we will be fixing this in a future release. In the meantime, kindly treat these errors as informational messages.

[lukasvosyka]

+1 for fixing

[m1nkeh]

guys, is there an ETA on this please... is it in a backlog somewhere? or perhaps already resolved in V2??

cc: @tohling

[garygsscott]

I am experiencing this too. Any ETA on a fix for this?

cc: @tohling

[m1nkeh]

[fabiocav]

Sorry about the delayed response here. To confirm the impact, this is primarily generating noise in the logs, correct? Those exceptions are normal, and handled by the runtime, but currently still tracked/logged. Some improvements were made in 2.0, so it would be interesting to understand if the behavior there is satisfactory.

@tohling were there any other open work items related to this?

[fabiocav]

Actually, the issue linked to the SO question referenced in the original issue was resolved with PRs linked to Azure/azure-webjobs-sdk#1760

@m1nkeh can you confirm?

@garygsscott if you're seeing similar exceptions, can you please open a different issue and share details/stack?

Thanks!

[m1nkeh]

Hi @fabiocav, sorry i keep meaning to reply to this.. no i do not believe anything is resolved - simply follow the repro steps, and you will see! 🙂

I am currently working on a new project, with all the latest up to date SDK levels, Azure Functions 2.0 .Net Core etc.. etc.. and still seeing this behaviour. Well the exceptions are slightly different now, but they are related to the same stuff.

Host level exceptions (all of them) are annoying noise in Application Insights.. this particular one, relating to the lease lost stuff is especially frustrating because under heavy load / scaling there are just constant exceptions and everyone on my team just 'knows' they are fine, and OK to ignore.

Why not just change these to warnings? The number of times i have to explain why these are OK to ignore with a diagram, and explanation of event hubs, and leases is just a massive time sink.

Aside: Would you like me to update the ticket to be functions v2.0 specific ?

[garygsscott]

Hi @fabiocav, @m1nkeh - I managed to suppress 90 % of these warning messages in my function by creating a dedicated consumer unit within my event hub and then ensuring that my event hub triggered function only connected to that one particular consumer unit. These messages still appear but with much less frequency after taking this approach. I actually work for the same organisation as @m1nkeh so do not believe a new case is justified and I agree with @m1nkeh that if these are not blocking exceptions and just warnings then shouldn't they be suppressed (to reduce the noise in troubleshooting more than anything else).

[charlesYong076]

+1 for fixing

[brettsam]

This has been fixed with Azure/azure-webjobs-sdk#2176. The next version of the EventHubs package will contain the fix.