Skip to content

Conversation

marceloneppel
Copy link
Member

@marceloneppel marceloneppel commented Mar 17, 2023

Issue

Jira ticket: DPE-1270

It's not possible to run backups on replicas, only in the primary.

Also, when performing a backup, the unit is accessible by PgBouncer and other applications (those who can face performance issues while the backup is being created).

Solution

Create and start a pebble service calling pgbackrest server, which is a TLS server that is used by the replicas to communicate with pgBackRest in the primary and request that missing WAL files are sent to the repository (S3) when the replica starts a backup process (so it backups the most updated data).

Also, while doing a backup in a replica, create some pg_hba reject rules to drop new connections to the replica.

Context

The pgBackRest server is a requirement to backup from replicas. If TLS is not enabled, it's allowed to backup only from the primary.

In src/backups.py:

  • _is_primary_pgbackrest_service_running is needed to confirm that the primary unit started the TLS server because in the replicas the service will fail if it cannot connect to the primary TLS server.
  • _change_connectivity_to_database changes the connectivity to the database from external host (pg_hba reject rule).
  • start_stop_pgbackrest_service starts or stops the pgbackrest TLS server, depending on whether TLS is enabled (relation to TLS certificates operator) and there are replicas.

In src/charm.py:

  • tls key was added to unit relation data to fix DPE-1473 and to not generate errors when relating to TLS certificates operator (which is needed for this PR).

In templates/pgbackrest.conf.j2:

  • Some TLS server related configs were added.

In tests/integration/test_tls.py:

  • TLS test was enabled again.

Testing

Changed the existing backup and restore integration test to backup from the replica.

Tested manually the connection (refused) to the replica while doing the backup on it.

Release Notes

Enable backup on replicas.
Hide replica from PgBouncer while doing backup.

@marceloneppel marceloneppel force-pushed the hide-replica-on-backup branch from b8eae3d to 578e05e Compare March 21, 2023 14:06
@marceloneppel marceloneppel changed the title Hide replica on backup Enable backup on replicas and hide cluster from PgBouncer while doing backup Mar 23, 2023
@marceloneppel marceloneppel changed the title Enable backup on replicas and hide cluster from PgBouncer while doing backup Enable backup on replicas and hide replica from PgBouncer while doing backup Mar 23, 2023
@marceloneppel marceloneppel marked this pull request as ready for review March 23, 2023 12:22
@marceloneppel marceloneppel force-pushed the hide-replica-on-backup branch from 3b2c06c to 9ff452a Compare March 23, 2023 21:56
@marceloneppel marceloneppel requested a review from delgod March 23, 2023 21:58
Copy link
Contributor

@taurus-forever taurus-forever left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well done!

P.S. Would be nice to create backup logic diagram for users/community.
With PgBackrest-server it is no longer trivial in cluster setup with TLS requirements.

@marceloneppel
Copy link
Member Author

create backup logic diagram for users/community.
With PgBackrest-server it is no longer trivial in cluster setup with TLS requirements.

Good idea, Alex! I created two Jira tickets for creating those diagrams on K8S and VM charms.

@marceloneppel marceloneppel merged commit ea781c1 into main Mar 24, 2023
@marceloneppel marceloneppel deleted the hide-replica-on-backup branch March 24, 2023 11:33
BON4 pushed a commit to BON4/postgresql-k8s-operator that referenced this pull request May 20, 2024
… backup (canonical#114)

* Initial code for backing up on replicas

* Add initial TLS settings on primary

* Adjust pgBackRest configuration to use standbys

* Implement working TLS server

* Fix TLS errors

* Fix stanza retrieval, restore test and unit test

* Enable Mattermost test

* Fix backup settings check

* Revert S3 settings

* Add TLS check

* Reorganize TLS server logic

* Fix multiple primaries situation

* Remove log calls

* Turn off connectivity only on replicas

* Use planned units when checking whether there are replicas

* Revert last change

* Improve TLS flag documentation

* Improve logic that controls backups on standbys
github-actions bot added a commit to canonical/test-runners-2-is-x64-postgresql-k8s-operator that referenced this pull request Jul 16, 2024
github-actions bot added a commit to canonical/test-runners-2-is-arm64-postgresql-k8s-operator that referenced this pull request Jul 17, 2024
github-actions bot added a commit to canonical/test-runners-2-github-x64-postgresql-k8s-operator that referenced this pull request Jul 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants