-
Notifications
You must be signed in to change notification settings - Fork 141
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Feat[MQB]: track queue depth per appId #320
Conversation
01396d3
to
9222a9a
Compare
9222a9a
to
90a7c9e
Compare
, d_opsPretty(bdljsn::WriteOptions() | ||
.setSpacesPerLevel(4) | ||
.setStyle(bdljsn::WriteStyle::e_PRETTY) | ||
.setSortMembers(true)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
sort
is necessary to have less diff when we update sample json data in ITs
On the other hand, I can do this in ITs
4f55f63
to
8ae6b9c
Compare
8ae6b9c
to
102331e
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Build 187 of commit 102331e has completed with FAILURE
8252871
to
9644ed4
Compare
Signed-off-by: Evgeny Malygin <emalygin@bloomberg.net>
9644ed4
to
0ee07e5
Compare
|
||
return it->value()->confirm(&data->second); | ||
return rc; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@dorjesinpo I checked your proposal to do this update in RootQueueEngine
. The problem is that we don't expose message size (data->second.d_size
) from this level (VirtualStorage
/VirtualStorageCatalog
) to the upper level of RootQueueEngine
. Passing this up the exec stack will be harmful both for interfaces and performance.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Build 289 of commit 0ee07e5 has completed with FAILURE
if (!d_subContextsHolder.empty()) { | ||
bsl::list<StatSubContextMp>::iterator it = | ||
d_subContextsHolder.begin(); | ||
while (it != d_subContextsHolder.end()) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We got rid of the loop in the put
method and here it is back again.
At the moment, I do not see a good way to utilize VirtualStorageCatalog::numMessages
so let's keep the loop.
Provided we won't maintain statistics for all Dynamic Apps.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Check for empty d_subContextsHolder
should be fast if we don't have subcontexts for dynamic apps.
We might utilize numMessages
on the time when we do snapshot, but this change is more complex, since we don't have a good way to ensure that each and every virtual storage is alive, and this change is intrusive.
I would rather merge this PR for now, since it helps to test and compare queue metrics. We can think of this in the future, when we have prof
outputs for the scenario in question.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actually, I would also like to move metrics update code to the header and mark it inline
, with hope that it will have less impact on performance. We have inline functions defined for StatContext
, but we break the inline chain here when we define in .cpp
. I would also like to do this change after we merge metrics update.
Signed-off-by: Evgeny Malygin <emalygin@bloomberg.net>
Signed-off-by: Evgeny Malygin <emalygin@bloomberg.net>
Signed-off-by: Christopher Beard <cbeard9@bloomberg.net> fixing Solaris build (bloomberg#434) Signed-off-by: dorjesinpo <129227380+dorjesinpo@users.noreply.github.com> Remove `-DBMQ_ENABLE_MSG_GROUPID` from the build system We do not ever want to build with this flag when releasing, and users often manage to enable this flag accidentally. Because message group IDs are not fully implemented, we remove this temporary definition. It can be added in later if we ever come back to this feature. Signed-off-by: Patrick M. Niedzielski <patrick@pniedzielski.net> Make unit tests pass without `BMQ_ENABLE_MSG_GROUPID` The unit tests currently assume that message group IDs are enabled, and since have updated our build system to no longer enable this feature, the unit tests now fail in CI. This patch guards the message group ID tests with preprocessor conditionals, disabling the parts of tests that try to set and check message group IDs. When `BMQ_ENABLE_MSG_GROUPID` is set, these parts of the unit tests run again. Signed-off-by: Patrick M. Niedzielski <patrick@pniedzielski.net> Fix mqbstat doc formatting (bloomberg#438) Signed-off-by: Christopher Beard <cbeard9@bloomberg.net> Fix[bmqeval]: limit expression length to avoid stack overflow (bloomberg#441) Signed-off-by: Evgeny Malygin <emalygin@bloomberg.net> Fix Solaris unit tests (bloomberg#440) Signed-off-by: Anton Pryakhin <apryakhin1@bloomberg.net> Docs[BMQ]: Use `.dox` files rather than `.md` files Package group documentation in `libbmq` was converted to Markdown files named `README.md`, and which was tied to the directory containing the code for the package group using Doxygen `@dir` commands. However, when generating the documentation, this left several empty pages in the documentation named `README`, which we were not able to remove. The solution for this that this patch uses is to switch from `.md` files to `.dox` files, which contain a single Doxygen-style C++ comment containing the `@dir` command. Unlike `.md` files, these do not automatically create pages, so there is no empty `README` page created for each package group. The cost of this is that `.dox` files cannot be simple Markdown files, but instead need to be wrapped in a C++ comment. Signed-off-by: Patrick M. Niedzielski <patrick@pniedzielski.net> Docs[BMQ] bde -> doxygen conversion fixes (bloomberg#443) * Doc[BMQT] minor bde -> doxygen docs * Doc[BMQA] minor bde -> doxygen docs * Doc[BMQA] re-wrap data member comments * Doc[BMQT] re-wrap data member comments * Apply suggestions from code review --------- Signed-off-by: Christopher Beard <cbeard9@bloomberg.net> Signed-off-by: Chris Beard <chrisbeard@users.noreply.github.com> Co-authored-by: Evgeny Malygin <678098@protonmail.com> Feat: track queue depth per appId (bloomberg#320) Signed-off-by: Evgeny Malygin <emalygin@bloomberg.net> configurator, bmqit: mode protos (bloomberg#447) Signed-off-by: Jean-Louis Leroy <jleroy9@bloomberg.net> Revert "configurator, bmqit: mode protos (bloomberg#447)" (bloomberg#449) This reverts commit a4b20db. Fix[mqbs_virtualstoragecatalog.cpp]: fix Solaris build (bloomberg#450) Signed-off-by: Evgeny Malygin <emalygin@bloomberg.net> fix: configurator: apply app ids (bloomberg#452) Signed-off-by: Jean-Louis Leroy <jleroy9@bloomberg.net> Fix [MQB]: mqbc::StorageMgr: Transition to available only when all primary active (bloomberg#416) * mqbc::StorageMgr: Ban 'processPrimaryStatusAdvisory' in non-FSM mode Signed-off-by: Yuan Jing Vincent Yan <yyan82@bloomberg.net> * mqbc::StorageMgr: Transition to available only when all primary active Signed-off-by: Yuan Jing Vincent Yan <yyan82@bloomberg.net> * mqbc::StorageMgr: clang-format Signed-off-by: Yuan Jing Vincent Yan <yyan82@bloomberg.net> * mqbc::StorageMgr: Healing replica buffers primary status advisories Signed-off-by: Yuan Jing Vincent Yan <yyan82@bloomberg.net> * mqbs::FileStore: Rename setPrimary -> setActivePrimary Signed-off-by: Yuan Jing Vincent Yan <yyan82@bloomberg.net> * mqbc::StorageMgr: Comment about check if all partitions available Signed-off-by: Yuan Jing Vincent Yan <yyan82@bloomberg.net> --------- Signed-off-by: Yuan Jing Vincent Yan <yyan82@bloomberg.net> Fix some compiler warnings in mqb (bloomberg#455) * -Wunused-parameter * -Wshadow * -Wswitch-enum Signed-off-by: Christopher Beard <cbeard9@bloomberg.net> It: Include full path for admin stat it test failures (bloomberg#453) * It: Include full path for admin stat it test failures This patch makes it a little easier to debug the metric & operation that causes an integration test for stats to fail. Signed-off-by: Christopher Beard <cbeard9@bloomberg.net> * Update src/integration-tests/test_admin_client.py Co-authored-by: Evgeny Malygin <678098@protonmail.com> Signed-off-by: Chris Beard <chrisbeard@users.noreply.github.com> --------- Signed-off-by: Christopher Beard <cbeard9@bloomberg.net> Signed-off-by: Chris Beard <chrisbeard@users.noreply.github.com> Co-authored-by: Evgeny Malygin <678098@protonmail.com> Feat: Add queue history size metric (bloomberg#436) * [WIP] Feat: Add queue history size metric This adds a new queue metric that counts the number of GUIDs in that queue's history. This is useful for identifying excessive memory utilization from history and potential history garbage collection issues (where history is filled up faster than it's cleaned up). Signed-off-by: Christopher Beard <cbeard9@bloomberg.net> * It: Extend admin it for history size stat Signed-off-by: Christopher Beard <cbeard9@bloomberg.net> --------- Signed-off-by: Christopher Beard <cbeard9@bloomberg.net> Feat[plugins]: report queue depth per appId to prometheus (bloomberg#446) Signed-off-by: Evgeny Malygin <emalygin@bloomberg.net> [Fix] m_bmqstoragetool::FileManagerImpl: Asserts not have side effects (bloomberg#461) Signed-off-by: Yuan Jing Vincent Yan <yyan82@bloomberg.net> Feat[MQB]: Enhance queue consumption monitor alarm log with additional details (bloomberg#420) Enhance filebackedstorage alarm log Signed-off-by: Aleksandr Ivanov <aivanov71@bloomberg.net> Cleanup Signed-off-by: Aleksandr Ivanov <aivanov71@bloomberg.net> Add test to mqbu_capacitymeter.t Signed-off-by: Aleksandr Ivanov <aivanov71@bloomberg.net> mqbc::StorageUtil, mqbi::StorageMgr: updateQueue -> updateQueuePrimary (bloomberg#466) Signed-off-by: Yuan Jing Vincent Yan <yyan82@bloomberg.net> Fix[MQB]: misc warnings (bloomberg#464) Allow dots in subscription property names Message properties allow arbitrary strings for property names, but our subscription expression language is more limited, requiring an initial alphabetic character followed by any number of alphanumeric characters and underscores. Some producers have begun using hierarchical message property names, separated by dots (“.”), and are unable to use subscriptions to filter or route according to these message properties. This patch extends the expression language grammar to enable matching on subscription property names with dots in them. This change is a pure extension: the language recognized by the subscription expression grammar after this patch is a strict superset of the language recognized by the subscription expression grammar before this patch. This patch also extends the unit test for the lexer to ensure this is a strict superset. Signed-off-by: Patrick M. Niedzielski <patrick@pniedzielski.net> fix: clean app subscriptions on reconfigure Signed-off-by: dorjesinpo <129227380+dorjesinpo@users.noreply.github.com> Fix[mqbstat_domainstats.cpp]: empty tier StringRef (bloomberg#431) Signed-off-by: Evgeny Malygin <emalygin@bloomberg.net> Fix Solaris build, it does not support ctor delegation Signed-off-by: Aleksandr Ivanov <aivanov71@bloomberg.net> Doc: Document app subscriptions (bloomberg#463) * Docs upgrade jekyll -> 4.3.3 Signed-off-by: Christopher Beard <cbeard9@bloomberg.net> * Docs: Document app subscriptions Signed-off-by: Christopher Beard <cbeard9@bloomberg.net> * Expand on difference in subscriptions Signed-off-by: Christopher Beard <cbeard9@bloomberg.net> * Minor subscription doc clarifications Signed-off-by: Christopher Beard <cbeard9@bloomberg.net> * Elaborate on subscription details Signed-off-by: Christopher Beard <cbeard9@bloomberg.net> * Clarify consumer subscription on broker Signed-off-by: Christopher Beard <cbeard9@bloomberg.net> --------- Signed-off-by: Christopher Beard <cbeard9@bloomberg.net> fix: enhanced detection of duplciate PUSHes (bloomberg#472) Signed-off-by: dorjesinpo <129227380+dorjesinpo@users.noreply.github.com> Fix ntf-core version in build_darwin.sh Signed-off-by: Aleksandr Ivanov <aivanov71@bloomberg.net> Add logAppsSubscriptionInfoCb into InMemoryStorage Signed-off-by: Aleksandr Ivanov <aivanov71@bloomberg.net> Add IT for capacity meter enhanced log Signed-off-by: Aleksandr Ivanov <aivanov71@bloomberg.net> Fix comments Signed-off-by: Aleksandr Ivanov <aivanov71@bloomberg.net> Fix [CI] ntf-core version for macosx build (bloomberg#473) Merge mwc into bmq MWC or "MiddleWare Core" was a package group developed to support a myriad of applications at Bloomberg. It's been useful to share common middleware components between similar technologies, but doesn't make much sense to support as its own open source library. Moving forward we are merging it into the BMQ package group to better maintain it for the BlazingMQ project. Signed-off-by: Taylor Foxhall <tfoxhall@bloomberg.net> Fix conflict Signed-off-by: Aleksandr Ivanov <aivanov71@bloomberg.net> Fix conflict Signed-off-by: Aleksandr Ivanov <aivanov71@bloomberg.net> Fix mwctst Signed-off-by: Aleksandr Ivanov <aivanov71@bloomberg.net>
VirtualStorageCatalog
on Primary and update per-appId queue metrics on these calls, if necessarysort
json stats on admin command