SST test

[marceloneppel]

Issue

• Jira issues: DPE-546
• The PostgreSQL charm should self heal the workload when restarting the DB process without data or transaction logs (SST test).

Solution

• Add a test that deletes the PostgreSQL data directory files (including transaction logs), restarts the DB processe and later check that the workload recovers itself from that situation.

Context

• This test can also be run on an existing cluster.

Testing

• The test was added on tests/integration/ha_tests/test_self_healing.py.

Release Notes

• Add SST test.

[codecov]

Codecov Report

Merging #43 (3497cf3) into main (855dab5) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main      #43   +/-   ##
=======================================
  Coverage   64.05%   64.05%           
=======================================
  Files           6        6           
  Lines         818      818           
  Branches      121      121           
=======================================
  Hits          524      524           
  Misses        264      264           
  Partials       30       30           

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[shayancanonical]

i believe that for this test, we need to remove the data directory while the postgress process is down (we need to extend the systemd restart timeout like mongodb here)

an excerpt of a message from mykola:

for SST test (in MySQL):
1) stop pebble/systemd on one member, remove all files in /var/lib/mysql [data directory] (simulate HDD failure)
2) write data to new primary
3) rotate binlog and remove rotated binlog ON ALL alive members. literally remove data written on step 2 (with such we simulate looong period of downtime)
4) run mysql on member from step 1)

[marceloneppel]

Thanks for the additional details, Shayan! I'll update the PR to have a similar approach on PostgreSQL.

[marceloneppel]

Hey @shayancanonical! I updated the code to have the right steps that simulate the needed scenario.

I haven't changed the systemd service restart timeout as after stopping the service it was not being restarted until I request to start it again.

I also added a check to ensure the WAL files (the equivalent to MysQL binlog) are correctly rotated (a new one is created - in fact, more than one new WAL file is kept due to some settings that enabled the old ones to be removed).

[shayancanonical]

Looks great!