Skip to content

[SE-4304] [SE-4495] feat: add celerybeat configuration#171

Merged
gabor-boros merged 1 commit intoesme-release/koa.3from
gabor/backport-celery-updates-koa.3
Jul 14, 2021
Merged

[SE-4304] [SE-4495] feat: add celerybeat configuration#171
gabor-boros merged 1 commit intoesme-release/koa.3from
gabor/backport-celery-updates-koa.3

Conversation

@gabor-boros
Copy link

The current implementation of celerybeat usage works properly if and
only if one concurrent worker process runs per queue using the nested
beat process started by the --beat flag. In case the the celerybeat
process is running as a separate process not as part of the workers,
that partially solves the problem. Either in that case, if multiple
instances are running in a cluster, without proper process supervising
across instances will result in (at least) dumplicated scheduling,
meaning that the scheduler will cause more work for the workers and
results will be duplicated.

Taking the above into consideration, to solve this issue, this commit
introduces single-beat package that wraps the celerybeat process and
keeps track of the process that acquired the lock in redis. In case one
of the instances get killed or the locking process crashes, the lock

refactor: use boolean flag to enable or disable celery beat
Co-authored-by: Joseph Mulloy jdmulloy@users.noreply.github.com

refactor: use boolean flag to enable or disable celery beat
Co-authored-by: Joseph Mulloy jdmulloy@users.noreply.github.com

Unindent a few entries that needd to be unindented (openedx-unsupported#6450) (openedx-unsupported#6467)
(cherry picked from commit 945fe9d)
Co-authored-by: Chris Pappas christopappas@users.noreply.github.com

Configuration Pull Request

Make sure that the following steps are done before merging

  • A devops team member has commented with 👍
  • are you adding any new default values that need to be overridden when this goes live?
    • Open a ticket (DEVOPS) to make sure that they have been added to secure vars.
    • Add an entry to the CHANGELOG.
  • Are you adding/updating any variables that need to be added/updated to a specific configuration-secure repo?

The current implementation of celerybeat usage works properly if and
only if one concurrent worker process runs per queue using the nested
beat process started by the `--beat` flag. In case the the celerybeat
process is running as a separate process not as part of the workers,
that partially solves the problem. Either in that case, if multiple
instances are running in a cluster, without proper process supervising
across instances will result in (at least) dumplicated scheduling,
meaning that the scheduler will cause more work for the workers and
results will be duplicated.

Taking the above into consideration, to solve this issue, this commit
introduces single-beat package that wraps the celerybeat process and
keeps track of the process that acquired the lock in redis. In case one
of the instances get killed or the locking process crashes, the lock

refactor: use boolean flag to enable or disable celery beat
Co-authored-by: Joseph Mulloy <jdmulloy@users.noreply.github.com>

refactor: use boolean flag to enable or disable celery beat
Co-authored-by: Joseph Mulloy <jdmulloy@users.noreply.github.com>

Unindent a few entries that needd to be unindented (openedx-unsupported#6450) (openedx-unsupported#6467)
(cherry picked from commit 945fe9d)
Co-authored-by: Chris Pappas <christopappas@users.noreply.github.com>
Copy link

@nizarmah nizarmah left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

  • I tested this by testing open-craft/openedx-platform#374.
  • I made sure the cherry-picked commit mentions the commit is was cherry-picked from.
  • I checked for accessibility issues
  • Includes documentation

@gabor-boros gabor-boros merged commit 129280c into esme-release/koa.3 Jul 14, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants