PendingOrchestrationMessages count does not update when a lease is lost and reclaimed rapidly.

[ConnorMcMahon]

Starting in this PR, we now abandon prefetched orchestration messages when we lose our lease. However, we fail to decrease pendingOrchestrationMessages when we do this, meaning that all of those abandoned messages still count against our limit, and we will not reclaim the "space" those lost messages are taking up until the application restarts.

[cgillum]

The abandon message code path should be decrementing the pending message count here. Any idea why that's not happening?

[ConnorMcMahon]

Ah, so we do handle it in that case. My case was subtly different, and I tried to make the issue describe the more general case, but as you pointed out, it should work there.

The case I was looking at it looked like we managed to reclaim the partition lease very quickly, and never actually abandoned the prefetched messages. However, when we reacquire the lease, it looks like we actually create a new ControlQueue. Since when we actually process the prefetched messages, when it comes time to delete the message, it is no longer in the pendingMessageIds field, as this is an instance variable, and the new ControlQueue starts with a fresh HashSet. Since we don't find a record of the message id in pendingMessageIds, we don't decrement.

[cgillum]

Oh wow, that's very subtle. Great find!

[ConnorMcMahon]

Fixed when updated DurableTask.AzureStorage to 1.7.5 in #1294