Freeze DB process test #39

marceloneppel · 2022-10-10T14:11:46Z

Issue

Jira issues: DPE-538
The PostgreSQL charm should self heal the workload when freezing the DB process (and later unfreezing it).

Solution

Add a test that freezes the DB processes (from Patroni and from PostgreSQL), unfreezing it after some time and later check that the workload recovers itself from that situation.

Context

This test can also be run on an existing cluster like in the previous implemented test from Add kill db process test #35.
An mechanism to get the other instances REST API URL was created on src/cluster.py to enable the charm to find the primary (that is used to update the relation application databag with the primary and the replicas list) when the unit that had the process frozen is the leader.

Testing

The test was added on tests/integration/ha_tests/test_self_healing.py. It tests the charm in a very similar way the Charmed MongoDB Operator is tested.
The test freezes both Patroni and PostgreSQL OS process, one at a time.

Release Notes

Add freeze db process test.

codecov · 2022-10-10T14:13:34Z

Codecov Report

Merging #39 (29eae5b) into main (c611ab6) will increase coverage by 2.13%.
The diff coverage is 90.90%.

@@            Coverage Diff             @@
##             main      #39      +/-   ##
==========================================
+ Coverage   60.32%   62.45%   +2.13%     
==========================================
  Files           6        6              
  Lines         809      815       +6     
  Branches      119      122       +3     
==========================================
+ Hits          488      509      +21     
+ Misses        293      276      -17     
- Partials       28       30       +2

Impacted Files	Coverage Δ
src/cluster.py	`67.78% <90.90%> (+11.84%)`	⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

shayancanonical

looks great!

MiaAltieri

wow so clean and very well implemented 🤩! Great job Marcelo! Minor comments only

MiaAltieri · 2022-10-11T16:44:00Z

src/cluster.py

+                    if member["name"] == member_name:
+                        ip = member["host"]
+                        break
+                return ip


Suggested change

if member["name"] == member_name:

ip = member["host"]

break

return ip

if member["name"] == member_name:

return member["host"]

Updated on 29eae5b.

MiaAltieri · 2022-10-11T16:45:17Z

src/cluster.py

+                        break
        return primary


Suggested change

break

return primary

return primary

Updated on 29eae5b.

MiaAltieri · 2022-10-11T16:46:09Z

src/cluster.py

        return primary

+    def _get_alternative_server_url(self, attempt: AttemptManager) -> str:
+        """Get an alternative URL from another member each time."""


what is an alternative URL and why is it needed? Sorry I am a PG noob 😅

Sorry Mia. Now I added an improved description. The alternative server URL method is just a way to call the REST API from another cluster member when, for example, the current member DB process is frozen (not able to receive requests). In that case the API is still working in the other members, so the URL from one of them is used. Updated on 2108c4a.

fascinating! Great descriptions thanks Marcelo! :)

MiaAltieri · 2022-10-11T16:53:06Z

tests/integration/ha_tests/test_self_healing.py

+        await send_signal_to_process(ops_test, primary_name, process, "SIGCONT")
+
+        # Verify that the database service got restarted and is ready in the old primary.
+        assert await postgresql_ready(ops_test, primary_name)


add checks that verify:

that the old primary is now the secondary

all units are in the same replica set

Great suggestion. Added both on 6a1e39b. I also added those checks on the kill DB process test.

tests/unit/test_cluster.py

MiaAltieri

really awesome work here Marcelo!

MiaAltieri · 2022-10-12T08:58:26Z

tests/integration/ha_tests/helpers.py

+                else:
+                    raise MemberNotUpdatedOnClusterError()
+    except RetryError:
+        return False


great function 🤩

* Add alternative servers for primary and members retrieval * Test working * Test working * Cleanup the code * More cleanup * Small adjustments * Add unit tests * Improve comments * Use down unit * Improve alternative URL description * Add additional checks * Improve returns

marceloneppel added 4 commits October 7, 2022 14:49

Add alternative servers for primary and members retrieval

79daddd

Test working

7811a0b

Test working

a4b76c8

Cleanup the code

ba63682

marceloneppel added 6 commits October 10, 2022 12:52

More cleanup

b48a3bd

Small adjustments

800ad8f

Merge branch 'main' into freeze-db-process-test

5d679df

Add unit tests

9ab9bf3

Improve comments

1c0d12d

Use down unit

ec6705d

marceloneppel changed the title ~~Freeze db process test~~ Freeze DB process test Oct 11, 2022

marceloneppel marked this pull request as ready for review October 11, 2022 15:20

marceloneppel requested review from Mehdi-Bendriss, MiaAltieri, WRFitch, averma-canonical, delgod, marcoppenheimer, paulomach, shayancanonical, welpaolo and zmraul October 11, 2022 15:21

shayancanonical approved these changes Oct 11, 2022

View reviewed changes

MiaAltieri reviewed Oct 11, 2022

View reviewed changes

marceloneppel added 3 commits October 11, 2022 14:32

Improve alternative URL description

2108c4a

Add additional checks

6a1e39b

Improve returns

29eae5b

marceloneppel requested a review from MiaAltieri October 12, 2022 01:59

MiaAltieri approved these changes Oct 12, 2022

View reviewed changes

marceloneppel merged commit 4994516 into main Oct 12, 2022

marceloneppel deleted the freeze-db-process-test branch October 12, 2022 10:55

marceloneppel mentioned this pull request Dec 15, 2022

Add freeze db process test canonical/postgresql-k8s-operator#49

Merged

github-actions bot added a commit to canonical/test-runners-2-github-x64-postgresql-operator that referenced this pull request May 21, 2024

Allure report canonical#39

e4aee37

github-actions bot added a commit to canonical/test-runners-2-is-x64-postgresql-operator that referenced this pull request May 22, 2024

Allure report canonical#39

9f23118

github-actions bot added a commit to canonical/test-runners-2-azure-arm64-postgresql-operator that referenced this pull request May 22, 2024

Allure report canonical#39

e79fbc0

Freeze DB process test #39

Freeze DB process test #39

Uh oh!

Conversation

marceloneppel commented Oct 10, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Issue

Solution

Context

Testing

Release Notes

Uh oh!

codecov bot commented Oct 10, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

shayancanonical left a comment

Choose a reason for hiding this comment

Uh oh!

MiaAltieri left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MiaAltieri Oct 11, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

MiaAltieri left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

marceloneppel commented Oct 10, 2022 •

edited

Loading

codecov bot commented Oct 10, 2022 •

edited

Loading

MiaAltieri Oct 11, 2022 •

edited

Loading