-
Notifications
You must be signed in to change notification settings - Fork 2.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Allow plugin to create their own django-rq queues #6651
Comments
I already added this feature in my own netbox fork along with some new unittest functions to ensure the implementation is working as expected If we agree on this I can move on and update the netbox documentation to add some notes regarding dedicated plugin queues. |
Thanks for this!
Creating multiple queues is just half the battle though, right? They also need to be ranked according to priority so that
Without such a mechanism in place, plugins would be relying on administrators to manually reconfigure their NetBox also currently has a |
You are absolutely right, I forgot to mention half of the proposed solution...sorry for this 😞
From my point of view, queue administration should be tackled at the sysadmin level. What I had in mind in the first place is to have:
The second point implies that the plugin maintainer inform potential user that they must create a dedicated rqworker instance listening to the plugin queues (container / systemd unit). No matter the underlying implementation to spawn new rqworker it is really simple nowadays to add an additional container/pod (cloud) or systemd unit (on-premise) to handle each plugin background workload. Even if creating new rqworker instance is a manual operation done by sysadmin, I see the following benefits:
|
Maybe there's a middle ground, where NetBox provides its own high- and low-priority queues (in addition to the default queue) for consumption by plugins. NetBox's rqworker would service all three of these accordingly by default. Then I'd feel more comfortable putting the burden on the user to service any additional queues needed by plugins. |
I will try to sum up our two ideas to define a proposal target design. What Netbox would provides out of the boxNetbox would define 3 queues : Netbox would provides a single rq-worker (systemd/container) which listen to the 3 queues above. As stated by the RQ documentation, the queue order is important here and should stay as What Netbox would allow for plugin developersPlugin developers have the ability to define their own dedicated queue(s) in the plugin's config. This implies a new rq-worker instance must be created and configured to listen to the plugin dedicated queue(s). In such case, end-user of the plugin must ensure this rq-worker is installed and configured properly according to any plugin install documentation. |
I like that! Just one tweak:
I would arrange them high > default > low. NetBox would continue to queue webhooks, reports, etc. in the default queue. |
I pinged the Docker folks for their take on this as well. Per @tobiasge in the #netbox-docker channel on the NetDev Slack:
|
I should also note that the PR for this should be based on the |
Thanks for the update @jeremystretch I'd also wanted to confirm where should I put some words in the documentation regarding this new feature (default queue + plugin related) ? I was thinking about a new dedicated section inside the plugin page right after the caching configuration. |
@maximumG Yep, that sounds perfect! |
FYI with #6713 now closed, NetBox has only a single |
Fixes #6651: Add plugin's queueing system
NetBox version
v2.11.7
Feature type
New functionality
Proposed functionality
I'd like to propose the ability for netbox plugin to define some dedicated django-rq queue(s). This new feature has already been discussed in #4908 but was missing a owner back in time.
The idea would be to store queue names in a new attribute inside the
PluginConfig
class. This attributes would be alist
in order to defined several queue to create by django-rq.In order to avoid queue name collision, the actual queue names will be prepended by the plugin name.
As an exemple the following config will lead to 3 queues being created using django-rq:
The creation of the queues will need to override django settings (RQ_QUEUES) and can be done in the
ready
method of thePluginConfig
class.We can also override django RQ_QUEUES settings directly in the
settings.py
, during init of all plugins.If you are interested in such feature I can take care of a PR.
Use case
All netbox plugins have the possibility to use the default queue to do some background/housekeeping tasks. If these tasks are long-running tasks they will block all other netbox-native background jobs, including netbox-reports & netbox-script.
Database changes
No response
External dependencies
No response
The text was updated successfully, but these errors were encountered: