-
Notifications
You must be signed in to change notification settings - Fork 131
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Feat[MQB]: Enhance queue consumption monitor alarm log with additional details #420
base: main
Are you sure you want to change the base?
Feat[MQB]: Enhance queue consumption monitor alarm log with additional details #420
Conversation
Signed-off-by: Aleksandr Ivanov <aivanov71@bloomberg.net>
Signed-off-by: Aleksandr Ivanov <aivanov71@bloomberg.net>
Signed-off-by: Aleksandr Ivanov <aivanov71@bloomberg.net>
Signed-off-by: Aleksandr Ivanov <aivanov71@bloomberg.net>
Signed-off-by: Aleksandr Ivanov <aivanov71@bloomberg.net>
Signed-off-by: Aleksandr Ivanov <aivanov71@bloomberg.net>
Signed-off-by: Aleksandr Ivanov <aivanov71@bloomberg.net>
Signed-off-by: Aleksandr Ivanov <aivanov71@bloomberg.net>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good. Few questions.
-
Maybe, we can print
Storage::numMessages
andStorage::numBytes
as well? -
Can we get rid of
QueueEngineUtil_AppState::head()
andQueueConsumptionMonitor::SubStreamInfo::d_headCb
now? -
Is
QueueConsumptionMonitor::onTransitionToIdle
"level triggered" (vs "edge triggered")? It can be noisy, how often we want that log?
@chrisbeard please take a look at the output
Regarding
Aren't they the same as Storage::numMessages() and Storage::numBytes? In my test they printed the same values. Are there any possible scenarios when they will differ? |
Regarding
Is Regarding |
Signed-off-by: Aleksandr Ivanov <aivanov71@bloomberg.net>
Looking at the It may make sense to print the stats for the App (vs the entire queue) |
Thanks, make sense, so I will add |
Signed-off-by: Aleksandr Ivanov <aivanov71@bloomberg.net>
|
Ok. This can flap, we often see a lot of subsequent logs. We are increasing the size of what's logged, so we may want to throttle the logging. |
When a queue starts to fill up, it is valuable to see information about which AppIds are impacted, and information about the messages in the queue.
Especially in the case of subscriptions (which we are enabling for everyone now), messages that match no subscription expression will build up in the put aside list.
To help make this situation clearer to operators and users (what apps are impacted, why are messages building up, how old is the head of the queue for each app, etc), we can log more information when the watermark alarm is triggered:
storage()->capacityMeter()->printShortSummary()
);This is to help debug why a message doesn't match a subscription.
Alarm log looks like this:
Implementation details:
QueueConsumptionMonitor
intoRootQueueEngine
class, where more data is available;QueueConsumptionMonitor
and called in case of alarm to log alarm data;RootQueueEngine::logAlarmCb
;