Update celery routing for celery 4+ #25567
Conversation
iamsobanjaved
force-pushed
the
iamsobanajved/celery-routing-update
branch
from
8d6bf21 to
820f2a9
Compare
iamsobanjaved
changed the title
|
After rebasing PR some thing went wrong. Fresh sandbox throwing error. Tasks list not showing instructor tasks. http://iamsobanjaved.sandbox.edx.org/ Another sandbox had the same issue but after multiple restarts and adding celery packages with our custom loggings soban sandbox now showing instructor tasks. but interesting thing it is showing these instructor tasks (16 in total ) multiple times, appearing in high queue, highmem queue and default queue also. |
|
@jmbowman This is the latest PR for the routing changes. Is it bad to make Koa without it? |
|
We have made sandbox with master and it was working fine, but after some restarts, it has started to showing I think the instructor task config is flaky here, as we are not importing tasks in AppConfig of this app. The output from this command: Sandbox: https://mastertest.sandbox.edx.org/ |
|
For a sandbox you can try these commands it will execute the instructor tasks 'generate-grade-csv' and show the result. |
|
@nedbat I'm pretty sure we want this in Koa, but at this point we should just cherrypick the commit once we get the whole problem sorted out. It shouldn't block any testing unless somebody wants to verify that tasks are still going to the correct queues (since without this they all go to the default queue). @iamsobanjaved Yeah, the terrible part of this is that the problem is intermittent even within the same deployment, and celery gives no useful information whatsoever to help in debugging it. I think we need to get some results from BOM-2086 before we can safely merge either this or the virtualenv upgrade again. |
awais786
force-pushed
the
iamsobanajved/celery-routing-update
branch
4 times, most recently
from
a323a33 to
2d66aff
Compare
iamsobanjaved
force-pushed
the
iamsobanajved/celery-routing-update
branch
from
2d66aff to
0674753
Compare
…ate' into timmc/celery-debug PR #25567 at 2020-12-08 # Conflicts: # cms/envs/production.py # lms/envs/production.py
|
Now that https://github.com/edx/edx-platform/pull/25840 has been merged, we should be clear to rebase and merge this. That PR made some changes to these files, so you may need to refactor a bit. |
- Used routing function istead of class - Move task queues to Djano settings - Removed routing_key parameter
iamsobanjaved
force-pushed
the
iamsobanajved/celery-routing-update
branch
from
0674753 to
f28b218
Compare
|
Your PR has finished running tests. There were no failures. |
|
After the fix, rebased it with the latest master and refactored it. Tested it on multiple sandboxes and Sandbox: https://celeryroute2.sandbox.edx.org/ |
feanil
left a comment
feanil
left a comment
There was a problem hiding this comment.
This looks great. It looks as a result of this change, how people write celery tasks for LMS and CMS will change. I think a follow-on task to write a new platform level docs/howto(How to add new celery tasks) would be a valuable doc to add.
Please make sure that that work is ticketed and done soon.
|
EdX Release Notice: This PR has been deployed to the staging environment in preparation for a release to production. |
|
EdX Release Notice: This PR has been deployed to the production environment. |
|
@nedbat Do we want to cherry pick this into Koa to fix celery task routing? |
* Update celery routing - Used routing function instead of class - Move task queues dictionary to Django settings - Removed routing_key parameter - Refactored routing for singleton celery instantiation Co-authored-by: Awais Qureshi <awais.qureshi@arbisoft.com>
* Update celery routing - Used routing function instead of class - Move task queues dictionary to Django settings - Removed routing_key parameter - Refactored routing for singleton celery instantiation Co-authored-by: Awais Qureshi <awais.qureshi@arbisoft.com>
* Update celery routing - Used routing function instead of class - Move task queues dictionary to Django settings - Removed routing_key parameter - Refactored routing for singleton celery instantiation Co-authored-by: Awais Qureshi <awais.qureshi@arbisoft.com>
* Update celery routing - Used routing function instead of class - Move task queues dictionary to Django settings - Removed routing_key parameter - Refactored routing for singleton celery instantiation Co-authored-by: Awais Qureshi <awais.qureshi@arbisoft.com> (cherry picked from commit e3b4d23)
* [SE-4101] fix: address VisibleBlocks caching race condition (openedx#27359) (#349) * fix: address VisibleBlocks caching race condition * sets visual block creation in an atomic transaction * refactor: add logging statement to bulk create Co-authored-by: Raul Gallegos <raul@opencraft.com> Co-authored-by: Raul Gallegos <raul@opencraft.com> * BB-3954 Add toggle for enrollment behavior (#351) Adds toggle REDIRECT_UNAUTHENTICATED_USER_TO_LOGIN_ON_ENROLL for enrollment behaviour for unauthenticated user. If true, the user will be redirected to 'signin_user' route. Co-authored-by: Arjun Singh Yadav <arjun@opencraft.com> * [BB-3622] feat:Restrict user create course (#319) (#352) * Add course creation condition for organization The condition added to ensure that if the feature is enabled user will not be able to create the course outside of the organization in which they belong. Signed-off-by: Farhaan Bukhsh <farhaan@opencraft.com> * fix: update the xblock-lti-consumer commit * Added new setting CERTIFICATE_DATE_FORMAT for easy customization of (#354) certificate issued date (cherry picked from commit 421e661) * fix: Produce grade report when subsections have future start dates When getting a subsection grade for a user, instead of failing if the user can't access that subsection, fallback to the collected structure. * Revert "Unhide student-generated certificates toggle" This reverts commit 8910ccf. Our clients are no longer using this feature, and edX won't accept this upstream as-is currently. See https://github.com/edx/edx-platform/pull/23735 for more information. Reverting to reduce code drift from upstream. * chore: update Arabic translations * fix:Fix function call to check MFE (#359) Signed-off-by: Farhaan Bukhsh <farhaan@opencraft.com> * fix: don't cache ajax request * feat: add site language config Cherry-picked from: https://github.com/edx/edx-platform/pull/27696 * Revert "[FAL-1813] fix: codejail issue when using matplotlib (#343)" (#365) This reverts commit 2143f84. * Update celery routing for celery 4+ (openedx#25567) * Update celery routing - Used routing function instead of class - Move task queues dictionary to Django settings - Removed routing_key parameter - Refactored routing for singleton celery instantiation Co-authored-by: Awais Qureshi <awais.qureshi@arbisoft.com> (cherry picked from commit e3b4d23) * fix: add missing set_code_owner_attribute imports (cherry picked from commit d1060d2) * chore: bump edx-django-utils to 3.12.0 The required `set_code_owner_attribute` decorator was introduced in v3.12.0, therefore we need to bump the dependency. (cherry picked from commit f52b84e) * [SE-4482] Allow delete course content in Studio only for admin users (#360) Co-authored-by: Nizar Mahmoud <nizarmah@hotmail.com> * fix: Password reset page throwing not found error (#364) Signed-off-by: Farhaan Bukhsh <farhaan@opencraft.com> * fix: change buttons in wiki modal to anchor tags The action buttons in wiki modal have href attribute but does not have an event listener for click. This PR changes the buttons to anchor tags so that they work as expected when clicked. * fix: use high priority queue for celery heartbeat check Makes HIGH_PRIORITY_QUEUE a derived setting, which allows HEARTBEAT_CELERY_ROUTING_KEY to use the correct config variant default. (cherry picked from commit 2a9067a) * fix: prevent invalidation of allowlisted certificates The allowlisted certificates were getting invalidated upon visiting the Course Progress page by users. This is a rough backport of the Lilac fix (edx#26356). In Lilac, this is gated by the `certificates_revamp.use_allowlist` Waffle flag. In post-Lilac branches, this is working out of the box (the flag has been removed in edx#27576). Jira ticket: BB-4287 * feat: add celery beat configuration Co-authored-by: Raul Gallegos <raul@opencraft.com> Co-authored-by: Farhaan Bukhsh <farhaan@opencraft.com> Co-authored-by: Arjun Singh Yadav <arjun@opencraft.com> Co-authored-by: pkulkark <pooja@opencraft.com> Co-authored-by: Shimul Chowdhury <shimul@opencraft.com> Co-authored-by: João Cabrita <joao.cabrita@opencraft.com> Co-authored-by: Samuel Walladge <samuel@opencraft.com> Co-authored-by: Giovanni Cimolin da Silva <giovannicimolin@gmail.com> Co-authored-by: 0x29a <demid@opencraft.com> Co-authored-by: Dmitry Gamanenko <dmitry.gamanenko@raccoongang.com> Co-authored-by: Jillian Vogel <jill@opencraft.com> Co-authored-by: Muhammad Soban Javed <58461728+iamsobanjaved@users.noreply.github.com> Co-authored-by: Sandeep Choudhary <sandeep@opencraft.com> Co-authored-by: Nizar Mahmoud <nizarmah@hotmail.com> Co-authored-by: Agrendalath <piotr@surowiec.it>
* Update celery routing - Used routing function instead of class - Move task queues dictionary to Django settings - Removed routing_key parameter - Refactored routing for singleton celery instantiation Co-authored-by: Awais Qureshi <awais.qureshi@arbisoft.com>
This is the new PR for celery routing as that revert PR's branch name had too many reverts in it.
We are currently facing anUnregisteredTaskexception on the sandbox after rebasing with the latest master. Although this exception disappears after restarting sandbox bysudo /edx/bin/supervisorctl restart allmultiple timesNow that fix is merged we can merge this.
This reverts commit c1fe3c3.
(cherry picked from commit db4c3b1)
https://github.com/edx/edx-platform/pull/25565