Fix queue count in rate limiters

[BrennanConroy]

Queue count can be updated by a cancellation request or inline when removing items from the queue (dispose a lease, replenish tokens, newest first kicking older queued items).

Since we just changed cancellation to not wait for callbacks to complete inline, that introduces a race where we don't update the queue count and loop until we removed enough items from the queue to add a new item.

do
{
    var item = queue.DequeueHead();
    item.Property; // <-- null ref
    // ...
}
while (_options.QueueLimit - _queueCount < permitCount);

Since the queue count might not be updated in the racy case, the DequeueHead() call will end up dequeuing an empty item (null) and null ref when access properties on it.

The end effect is that an incoming request will fail with a NRE instead of getting added to the queue successfully.

This PR changes the queue count to be updated by whoever has the lock when we want to update the queue count which avoids these issues.

[ghost]

Tagging subscribers to this area: @mangod9
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Queue count can be updated by a cancellation request or inline when removing items from the queue (dispose a lease, replenish tokens, newest first kicking older queued items).

Since we just changed cancellation to not wait for callbacks to complete inline, that introduces a race where we don't update the queue count and loop until we removed enough items from the queue to add a new item.

do
{
    var item = queue.DequeueHead();
    item.Property; // <-- null ref
    // ...
}
while (_options.QueueLimit - _queueCount < permitCount);

Since the queue count might not be updated in the racy case, the DequeueHead() call will end up dequeuing an empty item (null) and null ref when access properties on it.

The end effect is that an incoming request will fail with a NRE instead of getting added to the queue successfully.

This PR changes the queue count to be updated by whoever has the lock when we want to update the queue count which avoids these issues.

Author:	BrennanConroy
Assignees:	-
Labels:	area-System.Threading
Milestone:	-

[BrennanConroy]

/backport to release/8.0

[github-actions]

Started backporting to release/8.0: https://github.com/dotnet/runtime/actions/runs/5929263533