Add PostgreSQL 14 to 16 data migration guide

docs/explanation/charm-versions/legacy-charm.md

@@ -4,7 +4,7 @@ The legacy PostgreSQL charm is a [Reactive charm](https://documentation.ubuntu.c
 
 The PostgreSQL 16 charm does not support the same endpoints as the legacy charm. To migrate from legacy to PostgreSQL 16, you must implement the modern `database` endpoint for the `postgresql_client` interface on your charm.
 
-To read more about implementing compatible endpoints, see: [](/how-to/development/integrate-with-your-charm)
+To read more about implementing compatible endpoints, see: [](/how-to/integrate-with-your-charm)
 
 To read more about the legacy charm see the {doc}`PostgreSQL 14 documentation <postgresql-14:explanation/charm-versions/legacy-charm>`.
 

docs/explanation/roles.md

@@ -1,3 +1,4 @@
+(roles)=
 # Roles 
 
 There are several definitions of roles in Charmed PostgreSQL:

docs/how-to/data-migration/index.md

@@ -0,0 +1,18 @@
+# Migrate data
+
+For guidance about moving data from a Charmed PostgreSQL 14 database to Charmed PostgreSQL 16, start here:
+
+```{toctree}
+:titlesonly:
+
+Migrate data from PostgreSQL 14 to 16 <migrate-data-from-14-to-16>
+```
+
+More generic data migration guides:
+
+```{toctree}
+:titlesonly:
+
+Migrate data via `pg_dump` <migrate-data-via-pg-dump>
+Migrate data via backup/restore <migrate-data-via-backup-restore>
+```

docs/how-to/data-migration/migrate-data-from-14-to-16.md

@@ -0,0 +1,110 @@
+# Migrate data from PostgreSQL 14 to 16
+
+There are two possible ways to migrate data from PostgreSQL 14 to 16 depending on how {ref}`roles` are managed:
+
+**In case of admin roles management**, all database objects ownership are handled manually by the user. In this case, see the more general guide {ref}`migrate-data-via-pg-dump`. Note that you must set `extra-user-roles` to `charmed-admin` once a Juju relation is requested the new database.
+
+**In the case of charm roles management**, all the database objects ownership will be handled by charm automatically. This guide covers how to migrate data from Charmed PostgreSQL 14 to 16 using the new charm roles management setup for client applications managed by Juju. The migrated data from PostgreSQL 14 will be mapped to the corresponding ownership in PostgreSQL 16. 
+
+## Prepare PostgreSQL 14 data
+
+First, in order to make sure the latest data is included, remove the relation between the client app and Charmed PostgreSQL 14.
+
+Then, define the following variables for the old database:
+
+```shell
+TMP_PATH="~/old-db-dump/"
+OLD_DB_NAME="postgresql_test_app_database"
+OLD_IP="10.218.34.229"
+OLD_USER="operator"
+OLD_PASSWORD="fJQYljbthEo2T1gj"
+```
+
+Create a dump of the old PostgreSQL 14 database with `pg_dump`:
+
+```shell
+mkdir -p "${TMP_PATH}"
+PGPASSWORD="${OLD_PASSWORD}" pg_dump -j 4 -Fd -h "${OLD_IP}" -U "${OLD_USER}" -d "${OLD_DB_NAME}" -f "${TMP_PATH}" --compress 9
+```
+
+## Set up new PostgreSQL 16 charm
+
+Deploy one unit of Charmed PostgreSQL 16. This will simplify the migration and can be scaled later.
+
+```shell
+juju deploy postgresql --channel 16/stable
+```
+
+Define the following variables for the new database:
+
+```shell
+NEW_DB_NAME="postgresql_test_app_database123"
+NEW_IP="10.218.34.56"
+NEW_USER="operator"
+NEW_PASSWORD="RnnijCiotVeW8O5I"
+NEW_OWNER="charmed_${NEW_DB_NAME}_owner"
+```
+
+Migrate the following charm features from the old 14 charm to the new 16 charm: 
+* any necessary charm config options
+* enabled charm extensions/plugins
+
+```{note}
+Config options and extensions *must* be migrated before restoring the data dump
+```
+
+## Create a new database on PostgreSQL 16
+
+```shell
+PGPASSWORD="${NEW_PASSWORD}" psql -h "${NEW_IP}" -U "${NEW_USER}" -d postgres -c "CREATE DATABASE ${NEW_DB_NAME}"
+```
+
+Create new roles by running the function `set_up_predefined_catalog_roles()` in all databases except `postgres` and `template1`.  It will create roles like `charmed_<database-name>_owner`, `..._dml` and others:
+
+```shell
+PGPASSWORD="${NEW_PASSWORD}" psql -h "${NEW_IP}" -U "${NEW_USER}" -d "${NEW_DB_NAME}" -c "SELECT set_up_predefined_catalog_roles();"
+PGPASSWORD="${NEW_PASSWORD}" psql -h "${NEW_IP}" -U "${NEW_USER}" -d "${NEW_DB_NAME}" -c "ALTER DATABASE ${NEW_DB_NAME} OWNER TO charmed_databases_owner;"
+PGPASSWORD="${NEW_PASSWORD}" psql -h "${NEW_IP}" -U "${NEW_USER}" -d "${NEW_DB_NAME}" -c "ALTER SCHEMA public OWNER TO ${NEW_OWNER};"
+```
+
+## Migrate data from PostgreSQL 14 to 16
+
+Restore the PostgreSQL 14 database dump into the new 16 database:
+
+```shell
+PGPASSWORD="${NEW_PASSWORD}" pg_restore -j 4 -h "${NEW_IP}" -U "${NEW_USER}" -d "${NEW_DB_NAME}" -Fd "${TMP_PATH}" --no-owner
+```
+
+### Set up new database ownership
+
+Verify and modify the ownership for each database object in each schema to be equal to `charmed_<database-name>_owner` (`${NEW_OWNER}` above).
+
+For example, to find and fix ownership for all tables, sequences, and views:  
+
+```shell
+PGPASSWORD="${NEW_PASSWORD}" psql -h "${NEW_IP}" -U "${NEW_USER}" -d "${NEW_DB_NAME}"
+
+mydb=> DO $$
+DECLARE
+  r record;
+BEGIN
+  FOR r IN
+    SELECT format('ALTER %s %I.%I OWNER TO %I;',
+                  CASE c.relkind
+                    WHEN 'r' THEN 'TABLE'
+                    WHEN 'v' THEN 'VIEW'
+                    WHEN 'm' THEN 'MATERIALIZED VIEW'
+                    WHEN 'S' THEN 'SEQUENCE'
+                    WHEN 'p' THEN 'TABLE'
+                    ELSE NULL END,
+                  n.nspname, c.relname, 'charmed_<database-name>_owner')
+    FROM pg_class c
+    JOIN pg_namespace n ON n.oid = c.relnamespace
+    WHERE n.nspname = 'public'
+  LOOP
+    EXECUTE r.format;
+  END LOOP;
+END$$;
+```
+
+At this stage, the database has been completely imported. The cluster can be scaled, and the client app can be related to the new PostgreSQL 16 database.

docs/how-to/development/migrate-data-via-backup-restore.md → docs/how-to/data-migration/migrate-data-via-backup-restore.md

@@ -1,8 +1,9 @@
-# Migrate database data using ‘backup/restore’
+(migrate-data-via-backup-restore)=
+# Migrate data via backup/restore
 
-This is a guide for migrating data from modern charms. To migrate [legacy charms](/explanation/charm-versions/legacy-charm) data, refer to the guide [Migrate data via pg_dump](/how-to/development/migrate-data-via-pg-dump).
+This is a guide for migrating data from modern charms. To migrate [legacy charms](/explanation/charm-versions/legacy-charm) data, refer to the guide [Migrate data via pg_dump](/how-to/data-migration/migrate-data-via-pg-dump).
 
-This Charmed PostgreSQL operator is able to restore its own[backups](/how-to/back-up-and-restore/restore-a-backup) stored on [S3-compatible storage](/how-to/back-up-and-restore/configure-s3-aws). The same restore approach is applicable to restore [foreign backups](/how-to/back-up-and-restore/migrate-a-cluster) made by different Charmed PostgreSQL installation or even another PostgreSQL charm. The backup have to be created manually using [pgBackRest](https://pgbackrest.org/)!
+This Charmed PostgreSQL operator is able to restore its own [backups](/how-to/back-up-and-restore/restore-a-backup) stored on [S3-compatible storage](/how-to/back-up-and-restore/configure-s3-aws). The same restore approach is applicable to restore [foreign backups](/how-to/back-up-and-restore/migrate-a-cluster) made by different Charmed PostgreSQL installation or even another PostgreSQL charm. The backup has to be created manually using [pgBackRest](https://pgbackrest.org/)!
 
 ```{caution}
 The Canonical Data Team describes here the general approach and does NOT support nor guarantee the restoration results. 
@@ -19,7 +20,7 @@ Below is the *general approach* to the migration (see warning above!):
 
 1. Retrieve root/admin level credentials from legacy charm. 
 
-   See examples in [](/how-to/development/migrate-data-via-pg-dump).
+   See examples in [](/how-to/data-migration/migrate-data-via-pg-dump).
 
 2. Install [pgBackRest](https://pgbackrest.org/) inside the old charm OR nearby. 
 

docs/how-to/development/migrate-data-via-pg-dump.md → docs/how-to/data-migration/migrate-data-via-pg-dump.md

@@ -1,21 +1,8 @@
-# Migrate database data using `pg_dump` / `pg_restore`
+(migrate-data-via-pg-dump)=
+# Migrate data via `pg_dump`
 
-This guide describes database **data** migration only. To migrate charms on new juju interfaces, refer to the guide [How to integrate a database with my charm](/how-to/development/integrate-with-your-charm). 
+This guide describes database **data** migration only. To migrate charms on new Juju interfaces, refer to the guide [How to integrate a database with my charm](/how-to/integrate-with-your-charm). 
 
-## Do you need to migrate?
-
-A database migration is only required if the output of the following command is `latest/stable`:
-
-```text
-juju show-application postgresql | yq '.[] | .channel'
-```
-Migration is **not** necessary if the output above is `14/stable`! 
-
-This guide can be used to copy data between different installations of the same (modern) charm `postgresql`, but the [backup/restore](/how-to/development/migrate-data-via-backup-restore) is more recommended for migrations between modern charms.
-
-## Summary
-
-The legacy VM charm archived in the `latest/stable` channel, read more [here](/explanation/charm-versions/legacy-charm).
 A minor difference in commands might be necessary for different revisions and/or Juju versions, but the general logic remains:
 
 * Deploy the modern charm nearby
@@ -42,13 +29,16 @@ Always test migration in a safe environment before performing it in production!
 To obtain credentials for existing databases, execute the following commands for **each** database that will be migrated. Take note of these credentials for future steps.
 
 First, define and tune your application and db (database) names. For example:
+
 ```text
 CLIENT_APP=< my-application/0 >
 OLD_DB_APP=< legacy-postgresql/leader | postgresql/0 >
 NEW_DB_APP=< new-postgresql/leader | postgresql/0 >
 DB_NAME=< your_db_name_to_migrate >
 ```
+
 Then, obtain the username from the existing legacy database via its relation info:
+
 ```text
 OLD_DB_USER=$(juju show-unit ${CLIENT_APP} | yq '.[] | .relation-info | select(.[].endpoint == "db") | .[0].application-data.user')
 ```
@@ -60,61 +50,80 @@ Deploy new PostgreSQL database charm:
 ```text
 juju deploy postgresql ${NEW_DB_APP} --channel 16/stable
 ```
+
 Obtain `operator` user password of new PostgreSQL database from PostgreSQL charm:
+
 ```text
 NEW_DB_USER=operator
 NEW_DB_PASS=$(juju run ${NEW_DB_APP} get-password | yq '.password')
 ```
 
 ## Migrate database
+
 Use the credentials and information obtained in previous steps to perform the database migration with the following procedure.
 
 ```{note}
 Make sure no new connections were made and that the database has not been altered!
 ```
 
 ### Create dump from legacy charm
+
 Remove the relation between application charm and legacy charm:
+
 ```text
 juju remove-relation  ${CLIENT_APP}  ${OLD_DB_APP}
 ```
 Connect to the database VM of a legacy charm:
+
 ```text
 juju ssh ${OLD_DB_APP} bash
 ```
+
 Create a dump via Unix socket using credentials from the relation:
+
 ```text
 mkdir -p /srv/dump/
 OLD_DB_DUMP="legacy-postgresql-${DB_NAME}.sql"
 pg_dump -Fc -h /var/run/postgresql/ -U ${OLD_DB_USER} -d ${DB_NAME} > "/srv/dump/${OLD_DB_DUMP}"
 ```
+
 Exit the database VM:
+
 ```text
 exit
 ```
 ### Upload dump to new charm
+
 Fetch dump locally and upload it to the new Charmed PostgreSQL charm:
+
 ```text
 juju scp ${OLD_DB_APP}:/srv/dump/${OLD_DB_DUMP}  ./${OLD_DB_DUMP}
 juju scp ./${OLD_DB_DUMP}  ${NEW_DB_APP}:.
 ```
+
 ssh into new Charmed PostgreSQL charm and create a new database (using `${NEW_DB_PASS}`):
+
 ```text
 juju ssh ${NEW_DB_APP} bash
 createdb -h localhost -U ${NEW_DB_USER} --password ${DB_NAME}
 ```
+
 Restore the dump (using `${NEW_DB_PASS}`):
+
 ```text
 pg_restore -h localhost -U ${NEW_DB_USER} --password -d ${DB_NAME} --no-owner --clean --if-exists ${OLD_DB_DUMP}
 ```
 
 ## Integrate with modern charm
+
 Integrate (formerly "relate" in `juju v.2.9`) your application and new PostgreSQL database charm (using the modern `database` endpoint)
+
 ```text
 juju integrate ${CLIENT_APP}  ${NEW_DB_APP}:database
 ```
 
 If the `database` endpoint (from the `postgresql_client` interface) is not yet supported, use instead the `db` endpoint from the legacy `pgsql` interface:
+
 ```text
 juju integrate ${CLIENT_APP}  ${NEW_DB_APP}:db
 ```

docs/how-to/index.md

@@ -54,14 +54,25 @@ Monitoring (COS) <monitoring-cos/index>
 
 ## Refresh (upgrade)
 
-In-place upgrades to higher revisions of Charmed PostgreSQL 16:
+Instructions for performing an in-place application refresh:
 
 ```{toctree}
 :titlesonly:
 
 Refresh (upgrade) <refresh>
 ```
 
+## Data migration
+
+For charm developers looking to support PostgreSQL integrations with their charm:
+
+```{toctree}
+:maxdepth: 2
+:titlesonly:
+
+Data migration <data-migration/index>
+```
+
 ## Cross-regional (cluster-cluster) async replication
 
 Walkthrough of a cluster-cluster deployment and its essential operations:
@@ -84,13 +95,16 @@ How to replicate a subset of data to another PostgreSQL cluster:
 Logical replication <logical-replication/index>
 ```
 
-## Development
+## Charm development
 
-For charm developers looking to support PostgreSQL integrations with their charm:
+For charm developers looking to support PostgreSQL integrations with their charm
 
 ```{toctree}
-:maxdepth: 2
 :titlesonly:
 
-Development <development/index>
+Integrate PostgreSQL with your charm <integrate-with-your-charm>
 ```
+
+Other relevant guides:
+* {ref}`migrate-data-via-pg-dump`
+* {ref}`migrate-data-via-backup-restore`

docs/how-to/integrate-with-another-application.md

@@ -4,7 +4,7 @@
 
 This guide shows how to integrate Charmed PostgreSQL with both charmed and non-charmed applications.
 
-For developer information about how to integrate your own charmed application with PostgreSQL, see [Development > How to integrate with your charm](/how-to/development/integrate-with-your-charm).
+For developer information about how to integrate your own charmed application with PostgreSQL, see [Development > How to integrate with your charm](/how-to/integrate-with-your-charm).
 
 ## Integrate with a charmed application
 

docs/reference/contacts.md

@@ -13,5 +13,4 @@ Useful links:
 * [Git sources for Charmed PostgreSQL](https://github.com/canonical/postgresql-operator)
 * [Canonical Data on Launchpad](https://launchpad.net/~data-platform)
 * [Canonical Data on Matrix](https://matrix.to/#/#charmhub-data-platform:ubuntu.com) 
-* [Mailing list on Launchpad](https://lists.launchpad.net/data-platform/)