fix(*): prevent queues from growing without bounds

[hanshuebner]

Summary

This commit implements an upper limit on the number of batches that may be waiting on a queue for processing. Once the limit has been reached, the oldest batch is dropped from the queue and an error message is logged. The maximum number of batches that can be waiting on a queue is configured through the max_queued_batches parameter of the queue, which defaults to 100 and can be globally overriden with the max_queued_batches parameter in kong.conf

A more complete rewrite of the queuing system is in the works but will not make it into 3.2.

KAG-303

This reverts commit 218cc0a and fffb4e8

Checklist

• The Pull Request has tests
• There's an entry in the CHANGELOG
• There is a user-facing docs PR against https://github.com/Kong/docs.konghq.com - PUT DOCS PR HERE

[mashapedeployment]

The backport to release/2.8.x failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-release/2.8.x release/2.8.x
# Navigate to the new working tree
cd .worktrees/backport-release/2.8.x
# Create a new branch
git switch --create backport-10247-to-release/2.8.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 94a31c6471f5a52e22968e88f522135213233324
# Push it to GitHub
git push --set-upstream origin backport-10247-to-release/2.8.x
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-release/2.8.x

Then, create a pull request where the base branch is release/2.8.x and the compare/head branch is backport-10247-to-release/2.8.x.

[mashapedeployment]

The backport to release/3.0.x failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-release/3.0.x release/3.0.x
# Navigate to the new working tree
cd .worktrees/backport-release/3.0.x
# Create a new branch
git switch --create backport-10247-to-release/3.0.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 94a31c6471f5a52e22968e88f522135213233324
# Push it to GitHub
git push --set-upstream origin backport-10247-to-release/3.0.x
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-release/3.0.x

Then, create a pull request where the base branch is release/3.0.x and the compare/head branch is backport-10247-to-release/3.0.x.

[mashapedeployment]

The backport to release/3.1.x failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-release/3.1.x release/3.1.x
# Navigate to the new working tree
cd .worktrees/backport-release/3.1.x
# Create a new branch
git switch --create backport-10247-to-release/3.1.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 94a31c6471f5a52e22968e88f522135213233324
# Push it to GitHub
git push --set-upstream origin backport-10247-to-release/3.1.x
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-release/3.1.x

Then, create a pull request where the base branch is release/3.1.x and the compare/head branch is backport-10247-to-release/3.1.x.

[dndx]

@hanshuebner I might be missing some context but what is driving the inclusion of this feature again? What's the implication of this change for batch queue rewrite later? Can we remove this new config option safely in the future?

[hanshuebner]

@dndx We've decided to include this band-aid fix because an important customer demands an immediate solution (FTI-4756). We will be able to deprecate the global parameter and convert it to the plugin specific parameter - if set - when we move on to the new implementation.