Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: add maintenance mode for upgrades #2211

Merged
merged 31 commits into from
Nov 14, 2024
Merged

feat: add maintenance mode for upgrades #2211

merged 31 commits into from
Nov 14, 2024

Conversation

iamKunalGupta
Copy link
Member

@iamKunalGupta iamKunalGupta commented Nov 1, 2024 •
edited
Loading

  • Introduces Maintenance mode (status is available via dynamic config: PEERDB_MAINTENANCE_MODE_ENABLED)
  • Maintenance mode consists of 2 Workflows:
    • StartMaintenance - for pre-upgrade, responsible for
      • Waiting for running snapshots
      • Updating dynamic config to true
      • Pausing and backing up currently running mirrors
    • EndMaintenance - for post-upgrade, responsible for
      • Resuming backed up mirrors
      • Updating dynamic config to false
  • During the upgrade (between Start and End), mirrors cannot be mutated/created in any way,
  • There is also an instance info API which returns Ready/Maintenance which can be used for UI changes later.

There are 2 ways to trigger these 2 workflows:

  1. API call to flow-api
  2. Running the new maintenance entrypoint with the respective args

A new task queue is added so that the maintenance tasks can be spun up even during pre-upgrade hooks (from version earlier than ones containing this PR) and this also ensures that always the latest version of the maintenance flows run irrespective of the old version.

@iamKunalGupta iamKunalGupta marked this pull request as ready for review November 1, 2024 04:16
@iamKunalGupta iamKunalGupta requested a review from a team November 1, 2024 04:24
@iamKunalGupta iamKunalGupta changed the title feat: add maintenance mode feat: add maintenance mode for upgrades Nov 1, 2024
heavycrystal
heavycrystal reviewed Nov 1, 2024
View reviewed changes
flow/activities/maintenance_activity.go Outdated Show resolved Hide resolved
flow/activities/maintenance_activity.go
logEvery time.Duration,
alertEvery time.Duration,
) (protos.FlowStatus, error) {
// In case a mirror was just kicked off, it shows up in the running state, we wait for a bit before checking for snapshot
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this really shouldn't happen, seems like a bug

protos/route.proto Outdated Show resolved Hide resolved
flow/activities/maintenance_activity.go Outdated Show resolved Hide resolved
flow/activities/maintenance_activity.go Outdated Show resolved Hide resolved
flow/activities/maintenance_activity.go Outdated Show resolved Hide resolved
flow/activities/maintenance_activity.go Outdated Show resolved Hide resolved
flow/cmd/maintenance.go Outdated Show resolved Hide resolved
docker-bake.hcl Show resolved Hide resolved
flow/cmd/handler.go Show resolved Hide resolved
@iamKunalGupta
fix: nit
870ff64
@iamKunalGupta
fix: switch to proto for activity output and other nit fixes
b2a8575
@iamKunalGupta
fix: unroll loop
0833d4a
@iamKunalGupta
refactor: remove extra alerting and move env read to local activity
7c33505 
- also shift to tickers for waiting
@iamKunalGupta
refactor: use last start maintenance output from catalog
05a0ae6
@iamKunalGupta
fix: local test and fixes
67cc527
@iamKunalGupta
chore: lint fixes
2dc5a6f
@iamKunalGupta
feat: add assume-skipped flag for first time runs
be8b336
@iamKunalGupta
feaet: add additional k8s service existence check
969808d
@iamKunalGupta
feat: exit on error if error running app
2649f47
@iamKunalGupta
chore: nit fix
c39e070
@iamKunalGupta
fix: add missing flag
344ecd5
@iamKunalGupta
chore: use if not exists with index
8fd53ed
@iamKunalGupta
feat: embed peerdb version as arg
81c8075
@iamKunalGupta
chore: add peerdb sha to other docker images as well
620819f
@iamKunalGupta
fix: nit
d9d738c
@iamKunalGupta
chore: return ctx.Error to re-run on reschedules
dd670b2
@iamKunalGupta
chore: resolve comments
b19ba2c
@iamKunalGupta
chore: nit refactoring and fix
5625151
@iamKunalGupta
chore: small string fix
51e8a6e
@iamKunalGupta
chore: combine function call error check
f55a599
@iamKunalGupta
chore: update slog key to prevent datadog log level change
545d475
@iamKunalGupta
fix: refresh paused status
02cbd5e
@iamKunalGupta
fix: add heartbeat timeout for pausing
cd06ef8
@iamKunalGupta
feat: add more timeouts for activities
17f8c61
@iamKunalGupta
fix: err checking
c382119
@iamKunalGupta iamKunalGupta enabled auto-merge (squash) November 14, 2024 14:50
@iamKunalGupta iamKunalGupta merged commit 42b0208 into main Nov 14, 2024
10 checks passed
@iamKunalGupta iamKunalGupta deleted the feat/maintenance-mode branch November 14, 2024 14:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@heavycrystal heavycrystal heavycrystal approved these changes

@iskakaushik iskakaushik Awaiting requested review from iskakaushik

@serprex serprex Awaiting requested review from serprex

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Add maintenance mode for upgrades
3 participants
@iamKunalGupta @heavycrystal @serprex