[DPE-5827] Set all nodes to synchronous replicas #784

dragomirp · 2024-11-21T13:44:35Z

Port of canonical/postgresql-operator#672
Implements DA135 configuration for synchronous node count.

Adds logic to update the number of synchronous standbys and sets all replicas to sync.
Adds all and majority sync standby policies and ability to set constant number of sync standbys.

codecov · 2024-11-21T13:45:58Z

Codecov Report

Attention: Patch coverage is 68.75000% with 10 lines in your changes missing coverage. Please review.

Project coverage is 75.48%. Comparing base (62f5987) to head (f628e2c).
Report is 5 commits behind head on main.

Files with missing lines	Patch %	Lines
src/charm.py	53.84%	6 Missing ⚠️
src/patroni.py	73.33%	3 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #784      +/-   ##
==========================================
- Coverage   75.56%   75.48%   -0.09%     
==========================================
  Files          12       12              
  Lines        3127     3157      +30     
  Branches      467      471       +4     
==========================================
+ Hits         2363     2383      +20     
- Misses        623      632       +9     
- Partials      141      142       +1

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

dragomirp · 2024-11-21T23:16:53Z

tests/integration/ha_tests/helpers.py

+async def get_leader(model: Model, application_name: str) -> str:
+    """Get the standby leader name.
+
+    Args:
+        model: the model instance.
+        application_name: the name of the application to get the value for.
+
+    Returns:
+        the name of the standby leader.
+    """
+    status = await model.get_status()
+    first_unit_ip = list(status["applications"][application_name]["units"].values())[0]["address"]
+    cluster = get_patroni_cluster(first_unit_ip)
+    for member in cluster["members"]:
+        if member["role"] == "leader":
+            return member["name"]


get_primary from the main helpers is not model aware.

dragomirp · 2024-11-21T23:17:28Z

tests/integration/ha_tests/test_async_replication.py

-    new_sync_standby = await get_sync_standby(first_model, DATABASE_APP_NAME)
-    logger.info(f"New sync-standby: {new_sync_standby}")
-    assert new_sync_standby != sync_standby, "Sync-standby is the same as before"


Sync standby will not change, since all the replicas are sync standbys.

taurus-forever

The great job done! Thank you!

* try using pg 16 rock * try fix integration tests * remove argument on helper * more fixes * fix test issues * REVERT LATER: Adapt build_charm function * remove base=CHARM_BASE from deploy * remove juju 2.9 tests * grat permissions to public schema * fix linting and unit test * try refactor the permissions change * Update dependency uv to v0.5.27 (#847) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * grant public access to public schemas * [DPE-6053] Fix ports (#846) * Fix ports Signed-off-by: Marcelo Henrique Neppel <marcelo.neppel@canonical.com> * Remove unused import Signed-off-by: Marcelo Henrique Neppel <marcelo.neppel@canonical.com> * Remove unused charm library and fix unit tests Signed-off-by: Marcelo Henrique Neppel <marcelo.neppel@canonical.com> * Fix identation Signed-off-by: Marcelo Henrique Neppel <marcelo.neppel@canonical.com> * Bump libs * Conditional set_ports logic * Unit test * Don't expect juju managed resources --------- Signed-off-by: Marcelo Henrique Neppel <marcelo.neppel@canonical.com> Co-authored-by: Marcelo Henrique Neppel <marcelo.neppel@canonical.com> * mark smoke test as unstable * test release charm to 16/edge * fix test release workflow * fix typo * Sync docs from Discourse (#801) Co-authored-by: GitHub Actions <41898282+github-actions[bot]@users.noreply.github.com> * Update dependency uv to v0.5.29 (#848) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * [MISC] Define charm utility properties (#842) * Update data-platform-workflows to v29.1.0 (#849) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * revert metadata.yaml * nits * Use `charmcraft test` & concierge (#852) * [MISC] Split topology script (#853) * Update charmcraft.yaml build tools (#851) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update dependency uv to v0.5.31 (#858) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update canonical/data-platform-workflows action to v30 (#859) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Use _promote_charm.yaml (#860) Use `charmcraft promote` and auto-generate release notes * Update dependency cryptography to v44.0.1 [SECURITY] (#855) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Sync docs from Discourse (#854) Co-authored-by: GitHub Actions <41898282+github-actions[bot]@users.noreply.github.com> * Update data-platform-workflows to v30.0.2 (#863) * Update canonical/has-signed-canonical-cla action to v2 (#865) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * [DPE-5827] Set all nodes to synchronous replicas (#784) * Update patroni configuration * Update test assertion * Copy update_synchronous_node_count from VM * Add unit test * Set sync node count during upgrade * Fix tls test * Switchover primary * Add different helper to get leader * Add config boilerplate * Use config value when setting sync node count * Escape tuple * Add policy values * Add integration test * Fix casting * Fix test * Update to spec * Bump retry timout * Switch to planned units * Fix generator * Update conf description * Spread task * Pass the charm * [DPE-6484] Add scope to promote to primary (#850) * Promote unit action * Tweaks for failing REST calls * VM parity * [MISC] Enable RBAC in tests (#861) * Enable RBAC * Bump cosl * Add trust * Bump libs * [MISC] Define charm constants (#862) * Lock file maintenance Python dependencies (#835) * Lock file maintenance Python dependencies * Backoff boto3 1.36 --------- Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Dragomir Penev <dragomir.penev@canonical.com> * Update canonical/data-platform-workflows action to v30.1.3 (#856) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update charmcraft.yaml build tools (#868) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Handle missing stanza output (#866) * Update ghcr.io/canonical/charmed-postgresql:14.15-22.04_edge Docker digest to 71d49b6 (#870) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * [MISC] Disable storage test on arm (#872) * Disable storage test on Arm * Bump libs * Correct import path * Update charmcraft.yaml build tools (#871) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * [MISC] Add degraded status to primary message (#874) * Add degraded status on to primary message * Use const running tests * [MISC] Sanitize PostgreSQL extra-user-roles arg (#876) * [MISC] Fix PostgreSQL lib function signature (#879) * [MISC] Skip backup tests without creds (#884) * Bump libs * Skip backups tests if no creds are set * Check that the cloud config is set to a value * Typo * Update tests/integration/conftest.py Co-authored-by: Carl Csaposs <carl.csaposs@canonical.com> * Switch all checks to gets * Typo --------- Co-authored-by: Carl Csaposs <carl.csaposs@canonical.com> * [DPE-6679] PostgreSQL Config Improvement (#867) * config improvement * config improvement * config improvement * config improvement * config improvement * config improvement * config improvement * config improvement * config improvement * config improvement * Update dependency jinja2 to v3.1.6 [SECURITY] (#882) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update canonical/data-platform-workflows action to v30.2.0 (#887) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Reduce required approvals on Renovate pull requests by 1 (#880) * Update dependency uv to v0.6.5 (#878) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Don't run juju2 and legacy tests * Use correct charm * Dual branch config * Remove 14 specific tests * Remove eer data secrets * Skip upgrade from stable * Upgrade on Noble * Build and deploy noble * Linting * Disable pgaudit during tests * Linting * Use common deploy for smoke test * Tweaks * Remove legacy rels * Tweak tests * Cleanup markers * Deploy without pgaudit in plugin tests * Update ghcr.io/canonical/charmed-postgresql Docker tag to v14.17 (#877) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Correct remove relation * Bump rock * Remove check for port forwarding --------- Signed-off-by: Marcelo Henrique Neppel <marcelo.neppel@canonical.com> Co-authored-by: Lucas Gameiro Borges <lucas.borges@canonical.com> Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Dragomir Penev <6687393+dragomirp@users.noreply.github.com> Co-authored-by: Marcelo Henrique Neppel <marcelo.neppel@canonical.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Sinclert Pérez <sinclert.perez@canonical.com> Co-authored-by: lucas.gameiro-borges <lucas.gameiro-borges@student-cs.fr> Co-authored-by: Carl Csaposs <carl.csaposs@canonical.com> Co-authored-by: Dragomir Penev <dragomir.penev@canonical.com> Co-authored-by: Danylo Bereznenko <pepsiqqfanta@gmail.com>

* Update dependency uv to v0.5.27 (#847) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * [DPE-6053] Fix ports (#846) * Fix ports Signed-off-by: Marcelo Henrique Neppel <marcelo.neppel@canonical.com> * Remove unused import Signed-off-by: Marcelo Henrique Neppel <marcelo.neppel@canonical.com> * Remove unused charm library and fix unit tests Signed-off-by: Marcelo Henrique Neppel <marcelo.neppel@canonical.com> * Fix identation Signed-off-by: Marcelo Henrique Neppel <marcelo.neppel@canonical.com> * Bump libs * Conditional set_ports logic * Unit test * Don't expect juju managed resources --------- Signed-off-by: Marcelo Henrique Neppel <marcelo.neppel@canonical.com> Co-authored-by: Marcelo Henrique Neppel <marcelo.neppel@canonical.com> * Sync docs from Discourse (#801) Co-authored-by: GitHub Actions <41898282+github-actions[bot]@users.noreply.github.com> * Update dependency uv to v0.5.29 (#848) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * [MISC] Define charm utility properties (#842) * Update data-platform-workflows to v29.1.0 (#849) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Use `charmcraft test` & concierge (#852) * [MISC] Split topology script (#853) * Update charmcraft.yaml build tools (#851) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update dependency uv to v0.5.31 (#858) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update canonical/data-platform-workflows action to v30 (#859) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Use _promote_charm.yaml (#860) Use `charmcraft promote` and auto-generate release notes * Update dependency cryptography to v44.0.1 [SECURITY] (#855) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Sync docs from Discourse (#854) Co-authored-by: GitHub Actions <41898282+github-actions[bot]@users.noreply.github.com> * Update data-platform-workflows to v30.0.2 (#863) * Update canonical/has-signed-canonical-cla action to v2 (#865) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * [DPE-5827] Set all nodes to synchronous replicas (#784) * Update patroni configuration * Update test assertion * Copy update_synchronous_node_count from VM * Add unit test * Set sync node count during upgrade * Fix tls test * Switchover primary * Add different helper to get leader * Add config boilerplate * Use config value when setting sync node count * Escape tuple * Add policy values * Add integration test * Fix casting * Fix test * Update to spec * Bump retry timout * Switch to planned units * Fix generator * Update conf description * Spread task * Pass the charm * [DPE-6484] Add scope to promote to primary (#850) * Promote unit action * Tweaks for failing REST calls * VM parity * [MISC] Enable RBAC in tests (#861) * Enable RBAC * Bump cosl * Add trust * Bump libs * [MISC] Define charm constants (#862) * Lock file maintenance Python dependencies (#835) * Lock file maintenance Python dependencies * Backoff boto3 1.36 --------- Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Dragomir Penev <dragomir.penev@canonical.com> * Update canonical/data-platform-workflows action to v30.1.3 (#856) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update charmcraft.yaml build tools (#868) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Handle missing stanza output (#866) * Update ghcr.io/canonical/charmed-postgresql:14.15-22.04_edge Docker digest to 71d49b6 (#870) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * [MISC] Disable storage test on arm (#872) * Disable storage test on Arm * Bump libs * Correct import path * Update charmcraft.yaml build tools (#871) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * [MISC] Add degraded status to primary message (#874) * Add degraded status on to primary message * Use const running tests * [MISC] Sanitize PostgreSQL extra-user-roles arg (#876) * [MISC] Fix PostgreSQL lib function signature (#879) * [MISC] Skip backup tests without creds (#884) * Bump libs * Skip backups tests if no creds are set * Check that the cloud config is set to a value * Typo * Update tests/integration/conftest.py Co-authored-by: Carl Csaposs <carl.csaposs@canonical.com> * Switch all checks to gets * Typo --------- Co-authored-by: Carl Csaposs <carl.csaposs@canonical.com> * [DPE-6679] PostgreSQL Config Improvement (#867) * config improvement * config improvement * config improvement * config improvement * config improvement * config improvement * config improvement * config improvement * config improvement * config improvement * Update dependency jinja2 to v3.1.6 [SECURITY] (#882) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update canonical/data-platform-workflows action to v30.2.0 (#887) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Reduce required approvals on Renovate pull requests by 1 (#880) * Update dependency uv to v0.6.5 (#878) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update ghcr.io/canonical/charmed-postgresql Docker tag to v14.17 (#877) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update dependency uv to v0.6.7 (#890) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update canonical/data-platform-workflows action to v31 (#891) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Try the new timescale * Check PR on */edge * Enable cache, disable pgaudit * Use the right base * Arm tests --------- Signed-off-by: Marcelo Henrique Neppel <marcelo.neppel@canonical.com> Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Marcelo Henrique Neppel <marcelo.neppel@canonical.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Sinclert Pérez <sinclert.perez@canonical.com> Co-authored-by: Carl Csaposs <carl.csaposs@canonical.com> Co-authored-by: Danylo Bereznenko <pepsiqqfanta@gmail.com>

dragomirp added 4 commits November 20, 2024 17:51

Update patroni configuration

f50bb2c

Update test assertion

235e19d

Copy update_synchronous_node_count from VM

30a7f56

Add unit test

c6339ce

github-actions bot added the Libraries: OK label Nov 21, 2024

dragomirp added 5 commits November 21, 2024 16:42

Set sync node count during upgrade

9bb565e

Fix tls test

6e60993

Switchover primary

025bfb2

Merge branch 'main' into dpe-5827-all-sync

0aa9850

Add different helper to get leader

f88c956

dragomirp marked this pull request as ready for review November 21, 2024 23:14

dragomirp commented Nov 21, 2024

View reviewed changes

dragomirp requested review from a team, lucasgameiroborges, marceloneppel and taurus-forever and removed request for a team November 21, 2024 23:18

lucasgameiroborges approved these changes Nov 22, 2024

View reviewed changes

marceloneppel approved these changes Nov 22, 2024

View reviewed changes

taurus-forever approved these changes Nov 25, 2024

View reviewed changes

dragomirp added 8 commits November 29, 2024 16:06

Add config boilerplate

39817f6

Merge branch 'main' into dpe-5827-all-sync

2d5ab35

Merge branch 'dpe-5827-all-sync' into dpe-5827-all-sync-config

5752281

Use config value when setting sync node count

43412d1

Escape tuple

0b1c289

Add policy values

707a014

Add integration test

562dad5

Fix casting

048e720

dragomirp added 5 commits December 2, 2024 20:18

Fix test

1f65187

Update to spec

88541f1

Bump retry timout

95c80de

Switch to planned units

a74eae1

Merge branch 'main' into dpe-5827-all-sync

b3e9684

github-actions bot added Libraries: Out of sync and removed Libraries: OK labels Dec 5, 2024

dragomirp added 3 commits December 19, 2024 15:01

Merge branch 'main' into dpe-5827-all-sync

a5f8f2a

Fix generator

0a5916b

Merge branch 'main' into dpe-5827-all-sync

ac006d1

github-actions bot added Libraries: OK and removed Libraries: Out of sync labels Jan 8, 2025

Merge branch 'main' into dpe-5827-all-sync

26c536c

github-actions bot added Libraries: Out of sync and removed Libraries: OK labels Jan 8, 2025

dragomirp added 3 commits January 15, 2025 22:45

Merge branch 'main' into dpe-5827-all-sync

91d533b

Merge branch 'main' into dpe-5827-all-sync

372048d

Merge branch 'main' into dpe-5827-all-sync

b63c5d1

dragomirp added the enhancement New feature, UI change, or workload upgrade label Feb 14, 2025

dragomirp added 4 commits February 14, 2025 11:02

Merge branch 'main' into dpe-5827-all-sync

017fbb6

Update conf description

faee124

Spread task

ee6093f

Pass the charm

f628e2c

dragomirp merged commit 842aa47 into main Feb 18, 2025
96 checks passed

dragomirp deleted the dpe-5827-all-sync branch February 18, 2025 22:08

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[DPE-5827] Set all nodes to synchronous replicas #784

[DPE-5827] Set all nodes to synchronous replicas #784

Uh oh!

dragomirp commented Nov 21, 2024 •

edited

Loading

Uh oh!

codecov bot commented Nov 21, 2024 •

edited

Loading

Uh oh!

dragomirp Nov 21, 2024

Uh oh!

dragomirp Nov 21, 2024

Uh oh!

taurus-forever left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

[DPE-5827] Set all nodes to synchronous replicas #784

[DPE-5827] Set all nodes to synchronous replicas #784

Uh oh!

Conversation

dragomirp commented Nov 21, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Nov 21, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

dragomirp Nov 21, 2024

Choose a reason for hiding this comment

Uh oh!

dragomirp Nov 21, 2024

Choose a reason for hiding this comment

Uh oh!

taurus-forever left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

dragomirp commented Nov 21, 2024 •

edited

Loading

codecov bot commented Nov 21, 2024 •

edited

Loading