-
Notifications
You must be signed in to change notification settings - Fork 27
Enable backup on replicas and hide replica from PgBouncer while doing backup #114
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
b8eae3d
to
578e05e
Compare
3b2c06c
to
9ff452a
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Well done!
P.S. Would be nice to create backup logic diagram for users/community.
With PgBackrest-server it is no longer trivial in cluster setup with TLS requirements.
Good idea, Alex! I created two Jira tickets for creating those diagrams on K8S and VM charms. |
… backup (canonical#114) * Initial code for backing up on replicas * Add initial TLS settings on primary * Adjust pgBackRest configuration to use standbys * Implement working TLS server * Fix TLS errors * Fix stanza retrieval, restore test and unit test * Enable Mattermost test * Fix backup settings check * Revert S3 settings * Add TLS check * Reorganize TLS server logic * Fix multiple primaries situation * Remove log calls * Turn off connectivity only on replicas * Use planned units when checking whether there are replicas * Revert last change * Improve TLS flag documentation * Improve logic that controls backups on standbys
Issue
Jira ticket: DPE-1270
It's not possible to run backups on replicas, only in the primary.
Also, when performing a backup, the unit is accessible by PgBouncer and other applications (those who can face performance issues while the backup is being created).
Solution
Create and start a pebble service calling
pgbackrest server
, which is a TLS server that is used by the replicas to communicate with pgBackRest in the primary and request that missing WAL files are sent to the repository (S3) when the replica starts a backup process (so it backups the most updated data).Also, while doing a backup in a replica, create some
pg_hba
reject rules to drop new connections to the replica.Context
The pgBackRest server is a requirement to backup from replicas. If TLS is not enabled, it's allowed to backup only from the primary.
In
src/backups.py
:_is_primary_pgbackrest_service_running
is needed to confirm that the primary unit started the TLS server because in the replicas the service will fail if it cannot connect to the primary TLS server._change_connectivity_to_database
changes the connectivity to the database from external host (pg_hba reject rule).start_stop_pgbackrest_service
starts or stops the pgbackrest TLS server, depending on whether TLS is enabled (relation to TLS certificates operator) and there are replicas.In
src/charm.py
:tls
key was added to unit relation data to fix DPE-1473 and to not generate errors when relating to TLS certificates operator (which is needed for this PR).In
templates/pgbackrest.conf.j2
:In
tests/integration/test_tls.py
:Testing
Changed the existing backup and restore integration test to backup from the replica.
Tested manually the connection (refused) to the replica while doing the backup on it.
Release Notes
Enable backup on replicas.
Hide replica from PgBouncer while doing backup.