-
Notifications
You must be signed in to change notification settings - Fork 27
Add PostgreSQL 14 to 16 data migration guide #1153
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
5 commits
Select commit
Hold shift + click to select a range
046da9c
refactor development section
a-velasco 65dc0e2
add content to migration guide
a-velasco e3b58b8
fixes
a-velasco 4c789bd
remove launchpad mailing list
a-velasco cadd562
Update deployment channel to 16/edge for consistency
a-velasco File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,18 @@ | ||
| # Migrate data | ||
|
|
||
| For guidance about moving data from a Charmed PostgreSQL 14 database to Charmed PostgreSQL 16, start here: | ||
|
|
||
| ```{toctree} | ||
| :titlesonly: | ||
|
|
||
| Migrate data from PostgreSQL 14 to 16 <migrate-data-from-14-to-16> | ||
| ``` | ||
|
|
||
| More generic data migration guides: | ||
|
|
||
| ```{toctree} | ||
| :titlesonly: | ||
|
|
||
| Migrate data via `pg_dump` <migrate-data-via-pg-dump> | ||
| Migrate data via backup/restore <migrate-data-via-backup-restore> | ||
| ``` |
110 changes: 110 additions & 0 deletions
110
docs/how-to/data-migration/migrate-data-from-14-to-16.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,110 @@ | ||
| # Migrate data from PostgreSQL 14 to 16 | ||
|
|
||
| There are two possible ways to migrate data from PostgreSQL 14 to 16 depending on how roles are managed: | ||
|
|
||
| **In case of admin roles management**, all database objects ownership are handled manually by the user. In this case, see the more general guide {ref}`migrate-data-via-pg-dump`. Note that you must set `extra-user-roles` to `charmed-admin` once a Juju relation is requested the new database. | ||
|
|
||
| **In the case of charm roles management**, all the database objects ownership will be handled by charm automatically. This guide covers how to migrate data from Charmed PostgreSQL 14 to 16 using the new charm roles management setup for client applications managed by Juju. The migrated data from PostgreSQL 14 will be mapped to the corresponding ownership in PostgreSQL 16. | ||
|
|
||
| ## Prepare PostgreSQL 14 data | ||
|
|
||
| First, in order to make sure the latest data is included, remove the relation between the client app and Charmed PostgreSQL 14. | ||
|
|
||
| Then, define the following variables for the old database: | ||
|
|
||
| ```shell | ||
| TMP_PATH="~/old-db-dump/" | ||
| OLD_DB_NAME="postgresql_test_app_database" | ||
| OLD_IP="10.218.34.229" | ||
| OLD_USER="operator" | ||
| OLD_PASSWORD="fJQYljbthEo2T1gj" | ||
| ``` | ||
|
|
||
| Create a dump of the old PostgreSQL 14 database with `pg_dump`: | ||
|
|
||
| ```shell | ||
| mkdir -p "${TMP_PATH}" | ||
| PGPASSWORD="${OLD_PASSWORD}" pg_dump -j 4 -Fd -h "${OLD_IP}" -U "${OLD_USER}" -d "${OLD_DB_NAME}" -f "${TMP_PATH}" --compress 9 | ||
| ``` | ||
|
|
||
| ## Set up new PostgreSQL 16 charm | ||
|
|
||
| Deploy one unit of Charmed PostgreSQL 16. This will simplify the migration and can be scaled later. | ||
|
|
||
| ```shell | ||
| juju deploy postgresql-k8s --channel 16/edge --trust | ||
| ``` | ||
|
|
||
| Define the following variables for the new database: | ||
|
|
||
| ```shell | ||
| NEW_DB_NAME="postgresql_test_app_database123" | ||
| NEW_IP="10.218.34.56" | ||
| NEW_USER="operator" | ||
| NEW_PASSWORD="RnnijCiotVeW8O5I" | ||
| NEW_OWNER="charmed_${NEW_DB_NAME}_owner" | ||
| ``` | ||
|
|
||
| Migrate the following charm features from the old 14 charm to the new 16 charm: | ||
| * any necessary charm config options | ||
| * enabled charm extensions/plugins | ||
|
|
||
| ```{note} | ||
| Config options and extensions *must* be migrated before restoring the data dump | ||
| ``` | ||
|
|
||
| ## Create a new database on PostgreSQL 16 | ||
|
|
||
| ```shell | ||
| PGPASSWORD="${NEW_PASSWORD}" psql -h "${NEW_IP}" -U "${NEW_USER}" -d postgres -c "CREATE DATABASE ${NEW_DB_NAME}" | ||
| ``` | ||
|
|
||
| Create new roles by running the function `set_up_predefined_catalog_roles()` in all databases except `postgres` and `template1`. It will create roles like `charmed_<database-name>_owner`, `..._dml` and others: | ||
|
|
||
| ```shell | ||
| PGPASSWORD="${NEW_PASSWORD}" psql -h "${NEW_IP}" -U "${NEW_USER}" -d "${NEW_DB_NAME}" -c "SELECT set_up_predefined_catalog_roles();" | ||
| PGPASSWORD="${NEW_PASSWORD}" psql -h "${NEW_IP}" -U "${NEW_USER}" -d "${NEW_DB_NAME}" -c "ALTER DATABASE ${NEW_DB_NAME} OWNER TO charmed_databases_owner;" | ||
| PGPASSWORD="${NEW_PASSWORD}" psql -h "${NEW_IP}" -U "${NEW_USER}" -d "${NEW_DB_NAME}" -c "ALTER SCHEMA public OWNER TO ${NEW_OWNER};" | ||
| ``` | ||
|
|
||
| ## Migrate data from PostgreSQL 14 to 16 | ||
|
|
||
| Restore the PostgreSQL 14 database dump into the new 16 database: | ||
|
|
||
| ```shell | ||
| PGPASSWORD="${NEW_PASSWORD}" pg_restore -j 4 -h "${NEW_IP}" -U "${NEW_USER}" -d "${NEW_DB_NAME}" -Fd "${TMP_PATH}" --no-owner | ||
| ``` | ||
|
|
||
| ### Set up new database ownership | ||
|
|
||
| Verify and modify the ownership for each database object in each schema to be equal to `charmed_<database-name>_owner` (`${NEW_OWNER}` above). | ||
|
|
||
| For example, to find and fix ownership for all tables, sequences, and views: | ||
|
|
||
| ```shell | ||
| PGPASSWORD="${NEW_PASSWORD}" psql -h "${NEW_IP}" -U "${NEW_USER}" -d "${NEW_DB_NAME}" | ||
|
|
||
| mydb=> DO $$ | ||
| DECLARE | ||
| r record; | ||
| BEGIN | ||
| FOR r IN | ||
| SELECT format('ALTER %s %I.%I OWNER TO %I;', | ||
| CASE c.relkind | ||
| WHEN 'r' THEN 'TABLE' | ||
| WHEN 'v' THEN 'VIEW' | ||
| WHEN 'm' THEN 'MATERIALIZED VIEW' | ||
| WHEN 'S' THEN 'SEQUENCE' | ||
| WHEN 'p' THEN 'TABLE' | ||
| ELSE NULL END, | ||
| n.nspname, c.relname, 'charmed_<database-name>_owner') | ||
| FROM pg_class c | ||
| JOIN pg_namespace n ON n.oid = c.relnamespace | ||
| WHERE n.nspname = 'public' | ||
| LOOP | ||
| EXECUTE r.format; | ||
| END LOOP; | ||
| END$$; | ||
| ``` | ||
|
|
||
| At this stage, the database has been completely imported. The cluster can be scaled, and the client app can be related to the new PostgreSQL 16 database. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file was deleted.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
File renamed without changes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
it should be 16/stable here