Add kill db process test

[marceloneppel]

Issue

• Jira issues: DPE-535 and DPE-537
• The PostgreSQL charm should self heal the workload when the DB process is killed.

Solution

• Add a test that kills the DB processes (from Patroni and from PostgreSQL) and later check that the workload recovers itself from that situation.

Context

• We can run the test on an existing cluster passing the model name when running the integration tests (the PostgreSQL application from an existing cluster is retrieved through the app_name helper function).
• The master_start_timeout fixture runs the continuous writes test charm that was merged on Add continuous writes test charm #36 and now is used to ensure the database is healthy after each shakedown test.
• The master_start_timeout fixture is used to speed up the fail-over process when the DB process is killed in the former primary.

Testing

• The test was added on tests/integration/ha_tests/test_self_healing.py. It tests the charm in a very similar way the Charmed MongoDB Operator is tested.
• Both Patroni and PostgreSQL OS process are killed, one at a time.

Release Notes

• Add kill db process test.

[codecov-commenter]

Codecov Report

Merging #35 (5aaf4bc) into main (ceeba02) will increase coverage by 0.20%.
The diff coverage is 61.39%.

@@            Coverage Diff             @@
##             main      #35      +/-   ##
==========================================
+ Coverage   60.11%   60.32%   +0.20%     
==========================================
  Files           6        6              
  Lines         692      809     +117     
  Branches      103      119      +16     
==========================================
+ Hits          416      488      +72     
- Misses        256      293      +37     
- Partials       20       28       +8     

Impacted Files	Coverage Δ
src/cluster.py	55.94% <55.17%> (+1.46%)	⬆️
src/charm.py	53.25% <60.33%> (+1.08%)	⬆️
src/constants.py	100.00% <100.00%> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[WRFitch]

In the release pipeline you could call the CI with the following code:

# release.yaml
jobs:
  test/ci/whatever you want to call this part:
    uses: ./.github/workflows/ci.yaml

# ci.yaml
on:
  workflow_call:

This way, you only have to write these test runs out once.

[marceloneppel]

Amazing suggestion! I updated that on 5aaf4bc.

[WRFitch]

Perhaps these test application-charms could be renamed to be more descriptive? not essential for this PR though.

[marceloneppel]

Good suggestion! I created a ticket for that on Jira.