Enable Sync Mode #134

marceloneppel · 2023-04-10T17:13:31Z

Issue

Jira ticket: DPE-1456
Sync mode is not enabled, which can cause data loss.

Solution

Enable sync mode in the Patroni configuration and set the synchronous node code to the cluster minority.

In src/charm.py:

Added a health check to avoid multiple primaries when freezing the PostgreSQL DB process (I could't figure out how to make it work with TLS, so I create a ticket to investigate that and file a bug if needed).
Added a check for all members ready before restarting PostgreSQL due to TLS settings change (without the check, some failures happen).

In templates/patroni.yml.j2:

Update Sync mode configuration in the Patroni configuration.
Also redirected the logs to a file in the logs directory to avoid losing logs (pebble logs are limited to 100kb).

In tests/integration/ha_tests/application-charm/*:

Updated the tester charm with some improvements from the VM charm to make it more stable.

In tests/integration/helpers.py:

Added a new service to be checked in the tests.
Fixed a helper function that was not correctly returning whether the primary changed (sometimes, while failing over, there was no primary, and the helper function returned that the primary changed to a new one).

In tests/integration/test_tls.py:

Fixed the test to force pg_rewind to run again after having the new role: sync_standby.
Changed the test code to read whether pg_rewind is using TLS from the log file instead of the pebble logs.

src/charm.py

tests/integration/test_tls.py

tests/integration/ha_tests/helpers.py

dragomirp · 2023-04-11T09:58:38Z

tests/integration/ha_tests/test_self_healing.py

-PATRONI_PROCESS = "/usr/local/bin/patroni"
+APP_NAME = METADATA["name"]
+PATRONI_PROCESS = "/usr/bin/patroni"
 POSTGRESQL_PROCESS = "postgres"


We are likely hitting the same pkill issues as the VM charm here, though we may want to deal with that in a separate PR

Good catch! I'll check it and update the code.

Updated on https://github.com/canonical/postgresql-k8s-operator/pull/134/files/b79aa19b4a9f197c575d25345c780b47bb9f2d0d..c29e5cd37540872c7e19973598d8c77576d5388a. I used the same logic that we have in the VM charm (SIGINT). I'm going to open another PR to reenable the PG process freeze db test.

dragomirp · 2023-04-11T10:01:20Z

tests/integration/ha_tests/helpers.py

+    return resp.json()
+
+
 async def change_master_start_timeout(ops_test: OpsTest, seconds: Optional[int]) -> None:


We should switch this to primary since we are now on patroni 3

Removed on 84d810d, as the master_start_timeout is not being used. DPE-1407 is in the backlog to fix the wrong wording in the other places.

I reverted it due to changes needed to address the above comment.

This reverts commit 84d810d.

https://warthogs.atlassian.net/browse/DPE-1456 --------- Co-authored-by: Dragomir Penev <dragomir.penev@canonical.com>

dragomirp and others added 19 commits March 14, 2023 11:46

Enable sync mode

5efd7fd

Merge branch 'main' into dpe-1456-sync-mode

848e807

Merge branch 'main' into dpe-1456-sync-mode

359ebd9

Merge branch 'main' into dpe-1456-sync-mode

02ae4ad

Remove custom conf

426a00c

Merge branch 'main' into dpe-1456-sync-mode

349beaf

Add new expected k8s endpoint to helper function

aeec50d

Fix TLS test

ce1d6d9

Add pebble health check

4637469

Fix freeze db test

63cb1c2

Enable restart db test

ac75151

Remove commented code

4fb09d9

Fix unit tests

1ace69c

Merge branch 'main' into dpe-1456-sync-mode

7dd6f95

Improve TLS management

0ef7f51

Add health check update

17455b6

Remove health check

7e60fcb

Add health check

f12651e

Prevent multiple primaries

8dedcce

github-actions bot added the Libraries: OK label Apr 10, 2023

Remove unused code

4efaf74

marceloneppel marked this pull request as ready for review April 10, 2023 18:48

marceloneppel requested review from delgod, dragomirp and taurus-forever April 10, 2023 18:52

delgod reviewed Apr 11, 2023

View reviewed changes

src/charm.py Outdated Show resolved Hide resolved

delgod requested changes Apr 11, 2023

View reviewed changes

tests/integration/test_tls.py Outdated Show resolved Hide resolved

tests/integration/ha_tests/helpers.py Outdated Show resolved Hide resolved

tests/integration/ha_tests/helpers.py Show resolved Hide resolved

dragomirp reviewed Apr 11, 2023

View reviewed changes

Check max number written

d01c7e8

marceloneppel added 8 commits April 11, 2023 08:26

Check writes on all instances

3cc2f06

Remove master_start_timeout setting

84d810d

Improve logs retrieval

2278f80

Add CA chain to trusted certificates

364adae

Revert "Remove master_start_timeout setting"

b79aa19

This reverts commit 84d810d.

Use the right PG process in the HA tests

ed77452

Fix order in test

f89342e

Remove unused call to SIGCONT

c29e5cd

marceloneppel requested review from delgod and dragomirp April 11, 2023 21:43

dragomirp approved these changes Apr 12, 2023

View reviewed changes

delgod approved these changes Apr 12, 2023

View reviewed changes

delgod merged commit 8b9d456 into main Apr 12, 2023

delgod deleted the dpe-1456-sync-mode branch April 12, 2023 06:35

BON4 pushed a commit to BON4/postgresql-k8s-operator that referenced this pull request May 20, 2024

Enable Sync Mode (canonical#134)

ae91d62

https://warthogs.atlassian.net/browse/DPE-1456 --------- Co-authored-by: Dragomir Penev <dragomir.penev@canonical.com>

github-actions bot added a commit to canonical/test-runners-2-is-x64-postgresql-k8s-operator that referenced this pull request Aug 4, 2024

Allure report canonical#134

09fc7e8

github-actions bot added a commit to canonical/test-runners-2-is-arm64-postgresql-k8s-operator that referenced this pull request Aug 5, 2024

Allure report canonical#134

ec743dc

github-actions bot added a commit to canonical/test-runners-2-github-x64-postgresql-k8s-operator that referenced this pull request Aug 5, 2024

Allure report canonical#134

310338e

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Enable Sync Mode #134

Enable Sync Mode #134

Uh oh!

marceloneppel commented Apr 10, 2023 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dragomirp Apr 11, 2023

Uh oh!

marceloneppel Apr 11, 2023

Uh oh!

marceloneppel Apr 11, 2023

Uh oh!

dragomirp Apr 11, 2023

Uh oh!

marceloneppel Apr 11, 2023

Uh oh!

marceloneppel Apr 11, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		return resp.json()


		async def change_master_start_timeout(ops_test: OpsTest, seconds: Optional[int]) -> None:

Enable Sync Mode #134

Enable Sync Mode #134

Uh oh!

Conversation

marceloneppel commented Apr 10, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Issue

Solution

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dragomirp Apr 11, 2023

Choose a reason for hiding this comment

Uh oh!

marceloneppel Apr 11, 2023

Choose a reason for hiding this comment

Uh oh!

marceloneppel Apr 11, 2023

Choose a reason for hiding this comment

Uh oh!

dragomirp Apr 11, 2023

Choose a reason for hiding this comment

Uh oh!

marceloneppel Apr 11, 2023

Choose a reason for hiding this comment

Uh oh!

marceloneppel Apr 11, 2023

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

marceloneppel commented Apr 10, 2023 •

edited

Loading