diff --git a/docs/explanation/charm-versions/legacy-charm.md b/docs/explanation/charm-versions/legacy-charm.md index 17c758e882..e1753305aa 100644 --- a/docs/explanation/charm-versions/legacy-charm.md +++ b/docs/explanation/charm-versions/legacy-charm.md @@ -4,7 +4,7 @@ The legacy PostgreSQL charm is a [Reactive charm](https://documentation.ubuntu.c The PostgreSQL 16 charm does not support the same endpoints as the legacy charm. To migrate from legacy to PostgreSQL 16, you must implement the modern `database` endpoint for the `postgresql_client` interface on your charm. -To read more about implementing compatible endpoints, see: [](/how-to/development/integrate-with-your-charm) +To read more about implementing compatible endpoints, see: [](/how-to/integrate-with-your-charm) To read more about the legacy charm see the {doc}`PostgreSQL 14 documentation `. diff --git a/docs/explanation/roles.md b/docs/explanation/roles.md index 1eff6097f1..0eae363642 100644 --- a/docs/explanation/roles.md +++ b/docs/explanation/roles.md @@ -1,3 +1,4 @@ +(roles)= # Roles There are several definitions of roles in Charmed PostgreSQL: diff --git a/docs/how-to/data-migration/index.md b/docs/how-to/data-migration/index.md new file mode 100644 index 0000000000..fbf3c55d84 --- /dev/null +++ b/docs/how-to/data-migration/index.md @@ -0,0 +1,18 @@ +# Migrate data + +For guidance about moving data from a Charmed PostgreSQL 14 database to Charmed PostgreSQL 16, start here: + +```{toctree} +:titlesonly: + +Migrate data from PostgreSQL 14 to 16 +``` + +More generic data migration guides: + +```{toctree} +:titlesonly: + +Migrate data via `pg_dump` +Migrate data via backup/restore +``` \ No newline at end of file diff --git a/docs/how-to/data-migration/migrate-data-from-14-to-16.md b/docs/how-to/data-migration/migrate-data-from-14-to-16.md new file mode 100644 index 0000000000..9eb18ae8ee --- /dev/null +++ b/docs/how-to/data-migration/migrate-data-from-14-to-16.md @@ -0,0 +1,110 @@ +# Migrate data from PostgreSQL 14 to 16 + +There are two possible ways to migrate data from PostgreSQL 14 to 16 depending on how {ref}`roles` are managed: + +**In case of admin roles management**, all database objects ownership are handled manually by the user. In this case, see the more general guide {ref}`migrate-data-via-pg-dump`. Note that you must set `extra-user-roles` to `charmed-admin` once a Juju relation is requested the new database. + +**In the case of charm roles management**, all the database objects ownership will be handled by charm automatically. This guide covers how to migrate data from Charmed PostgreSQL 14 to 16 using the new charm roles management setup for client applications managed by Juju. The migrated data from PostgreSQL 14 will be mapped to the corresponding ownership in PostgreSQL 16. + +## Prepare PostgreSQL 14 data + +First, in order to make sure the latest data is included, remove the relation between the client app and Charmed PostgreSQL 14. + +Then, define the following variables for the old database: + +```shell +TMP_PATH="~/old-db-dump/" +OLD_DB_NAME="postgresql_test_app_database" +OLD_IP="10.218.34.229" +OLD_USER="operator" +OLD_PASSWORD="fJQYljbthEo2T1gj" +``` + +Create a dump of the old PostgreSQL 14 database with `pg_dump`: + +```shell +mkdir -p "${TMP_PATH}" +PGPASSWORD="${OLD_PASSWORD}" pg_dump -j 4 -Fd -h "${OLD_IP}" -U "${OLD_USER}" -d "${OLD_DB_NAME}" -f "${TMP_PATH}" --compress 9 +``` + +## Set up new PostgreSQL 16 charm + +Deploy one unit of Charmed PostgreSQL 16. This will simplify the migration and can be scaled later. + +```shell +juju deploy postgresql --channel 16/stable +``` + +Define the following variables for the new database: + +```shell +NEW_DB_NAME="postgresql_test_app_database123" +NEW_IP="10.218.34.56" +NEW_USER="operator" +NEW_PASSWORD="RnnijCiotVeW8O5I" +NEW_OWNER="charmed_${NEW_DB_NAME}_owner" +``` + +Migrate the following charm features from the old 14 charm to the new 16 charm: +* any necessary charm config options +* enabled charm extensions/plugins + +```{note} +Config options and extensions *must* be migrated before restoring the data dump +``` + +## Create a new database on PostgreSQL 16 + +```shell +PGPASSWORD="${NEW_PASSWORD}" psql -h "${NEW_IP}" -U "${NEW_USER}" -d postgres -c "CREATE DATABASE ${NEW_DB_NAME}" +``` + +Create new roles by running the function `set_up_predefined_catalog_roles()` in all databases except `postgres` and `template1`. It will create roles like `charmed__owner`, `..._dml` and others: + +```shell +PGPASSWORD="${NEW_PASSWORD}" psql -h "${NEW_IP}" -U "${NEW_USER}" -d "${NEW_DB_NAME}" -c "SELECT set_up_predefined_catalog_roles();" +PGPASSWORD="${NEW_PASSWORD}" psql -h "${NEW_IP}" -U "${NEW_USER}" -d "${NEW_DB_NAME}" -c "ALTER DATABASE ${NEW_DB_NAME} OWNER TO charmed_databases_owner;" +PGPASSWORD="${NEW_PASSWORD}" psql -h "${NEW_IP}" -U "${NEW_USER}" -d "${NEW_DB_NAME}" -c "ALTER SCHEMA public OWNER TO ${NEW_OWNER};" +``` + +## Migrate data from PostgreSQL 14 to 16 + +Restore the PostgreSQL 14 database dump into the new 16 database: + +```shell +PGPASSWORD="${NEW_PASSWORD}" pg_restore -j 4 -h "${NEW_IP}" -U "${NEW_USER}" -d "${NEW_DB_NAME}" -Fd "${TMP_PATH}" --no-owner +``` + +### Set up new database ownership + +Verify and modify the ownership for each database object in each schema to be equal to `charmed__owner` (`${NEW_OWNER}` above). + +For example, to find and fix ownership for all tables, sequences, and views: + +```shell +PGPASSWORD="${NEW_PASSWORD}" psql -h "${NEW_IP}" -U "${NEW_USER}" -d "${NEW_DB_NAME}" + +mydb=> DO $$ +DECLARE + r record; +BEGIN + FOR r IN + SELECT format('ALTER %s %I.%I OWNER TO %I;', + CASE c.relkind + WHEN 'r' THEN 'TABLE' + WHEN 'v' THEN 'VIEW' + WHEN 'm' THEN 'MATERIALIZED VIEW' + WHEN 'S' THEN 'SEQUENCE' + WHEN 'p' THEN 'TABLE' + ELSE NULL END, + n.nspname, c.relname, 'charmed__owner') + FROM pg_class c + JOIN pg_namespace n ON n.oid = c.relnamespace + WHERE n.nspname = 'public' + LOOP + EXECUTE r.format; + END LOOP; +END$$; +``` + +At this stage, the database has been completely imported. The cluster can be scaled, and the client app can be related to the new PostgreSQL 16 database. \ No newline at end of file diff --git a/docs/how-to/development/migrate-data-via-backup-restore.md b/docs/how-to/data-migration/migrate-data-via-backup-restore.md similarity index 75% rename from docs/how-to/development/migrate-data-via-backup-restore.md rename to docs/how-to/data-migration/migrate-data-via-backup-restore.md index f4e5277133..5d35eaaa0e 100644 --- a/docs/how-to/development/migrate-data-via-backup-restore.md +++ b/docs/how-to/data-migration/migrate-data-via-backup-restore.md @@ -1,8 +1,9 @@ -# Migrate database data using ‘backup/restore’ +(migrate-data-via-backup-restore)= +# Migrate data via backup/restore -This is a guide for migrating data from modern charms. To migrate [legacy charms](/explanation/charm-versions/legacy-charm) data, refer to the guide [Migrate data via pg_dump](/how-to/development/migrate-data-via-pg-dump). +This is a guide for migrating data from modern charms. To migrate [legacy charms](/explanation/charm-versions/legacy-charm) data, refer to the guide [Migrate data via pg_dump](/how-to/data-migration/migrate-data-via-pg-dump). -This Charmed PostgreSQL operator is able to restore its own[backups](/how-to/back-up-and-restore/restore-a-backup) stored on [S3-compatible storage](/how-to/back-up-and-restore/configure-s3-aws). The same restore approach is applicable to restore [foreign backups](/how-to/back-up-and-restore/migrate-a-cluster) made by different Charmed PostgreSQL installation or even another PostgreSQL charm. The backup have to be created manually using [pgBackRest](https://pgbackrest.org/)! +This Charmed PostgreSQL operator is able to restore its own [backups](/how-to/back-up-and-restore/restore-a-backup) stored on [S3-compatible storage](/how-to/back-up-and-restore/configure-s3-aws). The same restore approach is applicable to restore [foreign backups](/how-to/back-up-and-restore/migrate-a-cluster) made by different Charmed PostgreSQL installation or even another PostgreSQL charm. The backup has to be created manually using [pgBackRest](https://pgbackrest.org/)! ```{caution} The Canonical Data Team describes here the general approach and does NOT support nor guarantee the restoration results. @@ -19,7 +20,7 @@ Below is the *general approach* to the migration (see warning above!): 1. Retrieve root/admin level credentials from legacy charm. - See examples in [](/how-to/development/migrate-data-via-pg-dump). + See examples in [](/how-to/data-migration/migrate-data-via-pg-dump). 2. Install [pgBackRest](https://pgbackrest.org/) inside the old charm OR nearby. diff --git a/docs/how-to/development/migrate-data-via-pg-dump.md b/docs/how-to/data-migration/migrate-data-via-pg-dump.md similarity index 82% rename from docs/how-to/development/migrate-data-via-pg-dump.md rename to docs/how-to/data-migration/migrate-data-via-pg-dump.md index 2311662b3e..3d38c11ad0 100644 --- a/docs/how-to/development/migrate-data-via-pg-dump.md +++ b/docs/how-to/data-migration/migrate-data-via-pg-dump.md @@ -1,21 +1,8 @@ -# Migrate database data using `pg_dump` / `pg_restore` +(migrate-data-via-pg-dump)= +# Migrate data via `pg_dump` -This guide describes database **data** migration only. To migrate charms on new juju interfaces, refer to the guide [How to integrate a database with my charm](/how-to/development/integrate-with-your-charm). +This guide describes database **data** migration only. To migrate charms on new Juju interfaces, refer to the guide [How to integrate a database with my charm](/how-to/integrate-with-your-charm). -## Do you need to migrate? - -A database migration is only required if the output of the following command is `latest/stable`: - -```text -juju show-application postgresql | yq '.[] | .channel' -``` -Migration is **not** necessary if the output above is `14/stable`! - -This guide can be used to copy data between different installations of the same (modern) charm `postgresql`, but the [backup/restore](/how-to/development/migrate-data-via-backup-restore) is more recommended for migrations between modern charms. - -## Summary - -The legacy VM charm archived in the `latest/stable` channel, read more [here](/explanation/charm-versions/legacy-charm). A minor difference in commands might be necessary for different revisions and/or Juju versions, but the general logic remains: * Deploy the modern charm nearby @@ -42,13 +29,16 @@ Always test migration in a safe environment before performing it in production! To obtain credentials for existing databases, execute the following commands for **each** database that will be migrated. Take note of these credentials for future steps. First, define and tune your application and db (database) names. For example: + ```text CLIENT_APP=< my-application/0 > OLD_DB_APP=< legacy-postgresql/leader | postgresql/0 > NEW_DB_APP=< new-postgresql/leader | postgresql/0 > DB_NAME=< your_db_name_to_migrate > ``` + Then, obtain the username from the existing legacy database via its relation info: + ```text OLD_DB_USER=$(juju show-unit ${CLIENT_APP} | yq '.[] | .relation-info | select(.[].endpoint == "db") | .[0].application-data.user') ``` @@ -60,13 +50,16 @@ Deploy new PostgreSQL database charm: ```text juju deploy postgresql ${NEW_DB_APP} --channel 16/stable ``` + Obtain `operator` user password of new PostgreSQL database from PostgreSQL charm: + ```text NEW_DB_USER=operator NEW_DB_PASS=$(juju run ${NEW_DB_APP} get-password | yq '.password') ``` ## Migrate database + Use the credentials and information obtained in previous steps to perform the database migration with the following procedure. ```{note} @@ -74,47 +67,63 @@ Make sure no new connections were made and that the database has not been altere ``` ### Create dump from legacy charm + Remove the relation between application charm and legacy charm: + ```text juju remove-relation ${CLIENT_APP} ${OLD_DB_APP} ``` Connect to the database VM of a legacy charm: + ```text juju ssh ${OLD_DB_APP} bash ``` + Create a dump via Unix socket using credentials from the relation: + ```text mkdir -p /srv/dump/ OLD_DB_DUMP="legacy-postgresql-${DB_NAME}.sql" pg_dump -Fc -h /var/run/postgresql/ -U ${OLD_DB_USER} -d ${DB_NAME} > "/srv/dump/${OLD_DB_DUMP}" ``` + Exit the database VM: + ```text exit ``` ### Upload dump to new charm + Fetch dump locally and upload it to the new Charmed PostgreSQL charm: + ```text juju scp ${OLD_DB_APP}:/srv/dump/${OLD_DB_DUMP} ./${OLD_DB_DUMP} juju scp ./${OLD_DB_DUMP} ${NEW_DB_APP}:. ``` + ssh into new Charmed PostgreSQL charm and create a new database (using `${NEW_DB_PASS}`): + ```text juju ssh ${NEW_DB_APP} bash createdb -h localhost -U ${NEW_DB_USER} --password ${DB_NAME} ``` + Restore the dump (using `${NEW_DB_PASS}`): + ```text pg_restore -h localhost -U ${NEW_DB_USER} --password -d ${DB_NAME} --no-owner --clean --if-exists ${OLD_DB_DUMP} ``` ## Integrate with modern charm + Integrate (formerly "relate" in `juju v.2.9`) your application and new PostgreSQL database charm (using the modern `database` endpoint) + ```text juju integrate ${CLIENT_APP} ${NEW_DB_APP}:database ``` If the `database` endpoint (from the `postgresql_client` interface) is not yet supported, use instead the `db` endpoint from the legacy `pgsql` interface: + ```text juju integrate ${CLIENT_APP} ${NEW_DB_APP}:db ``` diff --git a/docs/how-to/development/index.md b/docs/how-to/development/index.md deleted file mode 100644 index 710279e2d9..0000000000 --- a/docs/how-to/development/index.md +++ /dev/null @@ -1,9 +0,0 @@ -# Development - -```{toctree} -:titlesonly: - -Integrate with your charm -Migrate data via pg_dump -Migrate data via backup/restore -``` \ No newline at end of file diff --git a/docs/how-to/index.md b/docs/how-to/index.md index 515c37861a..54877613bf 100644 --- a/docs/how-to/index.md +++ b/docs/how-to/index.md @@ -54,7 +54,7 @@ Monitoring (COS) ## Refresh (upgrade) -In-place upgrades to higher revisions of Charmed PostgreSQL 16: +Instructions for performing an in-place application refresh: ```{toctree} :titlesonly: @@ -62,6 +62,17 @@ In-place upgrades to higher revisions of Charmed PostgreSQL 16: Refresh (upgrade) ``` +## Data migration + +For charm developers looking to support PostgreSQL integrations with their charm: + +```{toctree} +:maxdepth: 2 +:titlesonly: + +Data migration +``` + ## Cross-regional (cluster-cluster) async replication Walkthrough of a cluster-cluster deployment and its essential operations: @@ -84,13 +95,16 @@ How to replicate a subset of data to another PostgreSQL cluster: Logical replication ``` -## Development +## Charm development -For charm developers looking to support PostgreSQL integrations with their charm: +For charm developers looking to support PostgreSQL integrations with their charm ```{toctree} -:maxdepth: 2 :titlesonly: -Development +Integrate PostgreSQL with your charm ``` + +Other relevant guides: +* {ref}`migrate-data-via-pg-dump` +* {ref}`migrate-data-via-backup-restore` \ No newline at end of file diff --git a/docs/how-to/integrate-with-another-application.md b/docs/how-to/integrate-with-another-application.md index 73976c7eb8..21a34180e2 100644 --- a/docs/how-to/integrate-with-another-application.md +++ b/docs/how-to/integrate-with-another-application.md @@ -4,7 +4,7 @@ This guide shows how to integrate Charmed PostgreSQL with both charmed and non-charmed applications. -For developer information about how to integrate your own charmed application with PostgreSQL, see [Development > How to integrate with your charm](/how-to/development/integrate-with-your-charm). +For developer information about how to integrate your own charmed application with PostgreSQL, see [Development > How to integrate with your charm](/how-to/integrate-with-your-charm). ## Integrate with a charmed application diff --git a/docs/how-to/development/integrate-with-your-charm.md b/docs/how-to/integrate-with-your-charm.md similarity index 100% rename from docs/how-to/development/integrate-with-your-charm.md rename to docs/how-to/integrate-with-your-charm.md diff --git a/docs/reference/contacts.md b/docs/reference/contacts.md index dbbe12750e..bd11cbfd2d 100644 --- a/docs/reference/contacts.md +++ b/docs/reference/contacts.md @@ -13,5 +13,4 @@ Useful links: * [Git sources for Charmed PostgreSQL](https://github.com/canonical/postgresql-operator) * [Canonical Data on Launchpad](https://launchpad.net/~data-platform) * [Canonical Data on Matrix](https://matrix.to/#/#charmhub-data-platform:ubuntu.com) -* [Mailing list on Launchpad](https://lists.launchpad.net/data-platform/)