Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GKE upgrade breaks postgres cluster #182

Open
vedantthapa opened this issue Jan 10, 2024 · 2 comments
Open

GKE upgrade breaks postgres cluster #182

vedantthapa opened this issue Jan 10, 2024 · 2 comments
Labels
dev ops CI/CD, infrastructure, etc k8s

Comments

@vedantthapa
Copy link
Contributor

There was an occurrence previously when a maintenance window was added to the GKE Autopilot Cluster. After maintenance was completed, the cloudnative-pg cluster ended up in a broken state. This might be due to the fact that cloudnative-pg operator expects to have nodeMaintenanceWindow.inProgress field set to true on the Cluster resource whenever there's a kubernetes upgrade (See docs).

We've removed the maintenance window for now so that no automated updates are triggered, however, a solution is needed to either automatically update the field in the Cluster resource before a GKE maintenance window and toggle it back to the orginial value once the upgrade is finished OR alert the user to the availability of an upgrade so they can modify the field, start the upgrade, and then toggle the field back to its initial value.

@vedantthapa
Copy link
Contributor Author

Setting nodeMaintenanceWindow.inProgress to true might cause issues with the Cluster as per the warning mentioned on this page: https://cloudnative-pg.io/documentation/1.15/kubernetes_upgrade/#kubernetes-upgrade

@vedantthapa vedantthapa added dev ops CI/CD, infrastructure, etc k8s labels Jan 10, 2024
@vedantthapa
Copy link
Contributor Author

Another idea is to set the maintenance window and have a github action workflow run on a schedule before and after the maintenance window to update the relevant field in the manifest.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
dev ops CI/CD, infrastructure, etc k8s
Projects
None yet
Development

No branches or pull requests

1 participant