Migration documentation

.github/styles/Vocab/Timescale/accept.txt

@@ -10,7 +10,9 @@ https?
 [Bb]itemporal
 [Bb]lockchains?
 [Bb]oolean
+BSD
 [Cc]allouts?
+COPY
 [Cc]lickstreams?
 Cloudformation
 Cloudwatch
@@ -32,7 +34,10 @@ Ethereum
 [Gg]apfill(?:ing)?
 [Gg]eospatial
 GitHub
+GNU
 Grafana
+GUC
+gsed
 gzip(?:ped)?
 Hasura
 HipChat
@@ -48,6 +53,8 @@ Indri
 [Ii]ntraday
 [Iin]validation
 jpg
+JDBC
+JDK
 JSON
 Kafka
 Kaggle
@@ -70,9 +77,12 @@ Outflux
 Patroni
 Paulo
 [Pp]erformant
+pg_dump
+pg_restore
 [Pp]laintext
 Plotly
 pre
+POSIX
 PostgreSQL
 [Pp]ooler?
 Prometheus
@@ -83,15 +93,18 @@ Protobuf
 psql
 [Qq]uantiles?
 qStudio
+RDS
 [Rr]edistribut(?:e|able)
 [Rr]eindex(?:ed)?
+reltuples
 [Rr]eusability
 [Rr]unbooks?
 [Ss]crollable
 Sequelize
 [Ss]ignups?
 [Ss]iloed
 (?i)smallint
+sed
 src
 [Ss]ubquer(?:y|ies)
 [Ss]ubsets?
@@ -105,6 +118,7 @@ Threema
 Timescale(?:DB)?
 tobs
 [Tt]ransactionally
+tsdbadmin
 Uber
 [Uu]nary
 [Uu]ncomment
@@ -126,6 +140,7 @@ URLs?
 URIs?
 versionContent
 [Vv]irtualenv
+WAL
 [Ww]ebsockets?
 Worldmap
 www

_partials/_migrate_dual_write_backfill_getting_help.md

@@ -0,0 +1,13 @@
+import OpenSupportRequest from "versionContent/_partials/_migrate_open_support_request.mdx"
+
+<Highlight type="tip">
+
+If you get stuck, you can get help by either opening a support request, or take
+your issue to the `#migration` channel in the [community slack], where the
+developers of this migration method are there to help.
+
+<OpenSupportRequest />
+
+</Highlight>
+
+[community slack]: https://slack.timescale.com/

_partials/_migrate_dual_write_dump_database_roles.md

@@ -0,0 +1,50 @@
+```bash
+pg_dumpall -d "$SOURCE" \
+  --quote-all-identifiers \
+  --roles-only \
+  --file=roles.sql
+```
+
+Timescale services do not support roles with superuser access. If your SQL
+dump includes roles that have such permissions, you'll need to modify the file
+to be compliant with the security model.
+
+You can use the following `sed` command to remove unsupported statements and
+permissions from your roles.sql file:
+
+```bash
+sed -i -E \
+-e '/CREATE ROLE "postgres";/d' \
+-e '/ALTER ROLE "postgres"/d' \
+-e 's/(NO)*SUPERUSER//g' \
+-e 's/(NO)*REPLICATION//g' \
+-e 's/(NO)*BYPASSRLS//g' \
+-e 's/GRANTED BY "[^"]*"//g' \
+roles.sql
+```
+
+<Highlight type="note">
+This command works only with the GNU implementation of sed (sometimes referred
+to as gsed). For the BSD implementation (the default on macOS), you need to
+add an extra argument to change the `-i` flag to `-i ''`.
+
+To check the sed version, you can use the command `sed --version`. While the
+GNU version explicitly identifies itself as GNU, the BSD version of sed
+generally doesn't provide a straightforward --version flag and simply outputs
+an "illegal option" error.
+</Highlight>
+
+A brief explanation of this script is:
+
+- `CREATE ROLE "postgres"`; and `ALTER ROLE "postgres"`: These statements are
+  removed because they require superuser access, which is not supported
+  by Timescale.
+
+- `(NO)SUPERUSER` | `(NO)REPLICATION` | `(NO)BYPASSRLS`: These are permissions
+  that require superuser access.
+
+- `GRANTED BY role_specification`: The GRANTED BY clause can also have permissions that
+  require superuser access and should therefore be removed. Note: Per the
+  TimescaleDB documentation, the GRANTOR in the GRANTED BY clause must be the
+  current user, and this clause mainly serves the purpose of SQL compatibility.
+  Therefore, it's safe to remove it.

_partials/_migrate_dual_write_step1.md

@@ -0,0 +1,12 @@
+import OpenSupportRequest from "versionContent/_partials/_migrate_open_support_request.mdx"
+
+## 1. Set up a target database instance in Timescale
+
+[Create a database service in Timescale][create-service].
+
+If you intend on migrating more than 400 GB, open a support request to
+ensure that enough disk is pre-provisioned on your Timescale instance.
+
+<OpenSupportRequest />
+
+[create-service]: /use-timescale/:currentVersion:/services/create-a-service/

_partials/_migrate_dual_write_step2.md

@@ -0,0 +1,14 @@
+## 2. Modify the application to write to the target database
+
+How exactly to do this is dependent on the language that your application is
+written in, and on how exactly your ingestion and application function. In the
+simplest case, you simply execute two inserts in parallel. In the general case,
+you must think about how to handle the failure to write to either the source or
+target database, and what mechanism you want to or can build to recover from
+such a failure.
+
+You may also want to execute the same read queries on the source and target
+database to evaluate the correctness and performance of the results which the
+queries deliver. Bear in mind that the new database spends a certain amount of
+time without all data being present, so you should expect that the results are
+not the same for some period (potentially a number of days).

_partials/_migrate_dual_write_step4.md

@@ -0,0 +1,4 @@
+## 4. Start application in dual-write mode
+
+With the target database set up, your application can now be started in
+dual-write mode.

_partials/_migrate_dual_write_step5.md

@@ -0,0 +1,53 @@
+## 5. Determine the completion point `T`
+
+After dual-writes have been executing for a while, the target hypertable
+contains data in three time ranges: missing writes, late-arriving data, and the
+"consistency" range
+
+<img
+class="main-content__illustration"
+width={1375} height={944}
+src="https://assets.timescale.com/docs/images/hypertable_backfill_consistency.png"
+alt="Hypertable dual-write ranges"
+/>
+
+### Missing writes
+
+If the application is made up of multiple writers, and these writers did not
+all simultaneously start writing into the target hypertable, there is a period
+of time in which not all writes have made it into the target hypertable. This
+period starts when the first writer begins dual-writing, and ends when the last
+writer begins dual-writing.
+
+### Late-arriving data
+
+Some applications have late-arriving data: measurements which have a timestamp
+in the past, but which weren't written yet (for example from devices which had
+intermittent connectivity issues). The window of late-arriving data is between
+the present moment, and the maximum lateness.
+
+### Consistency range
+
+The consistency range is the range in which there are no missing writes, and in
+which all data has arrived, that is between the end of the missing writes range
+and the beginning of the late-arriving data range.
+
+The length of these ranges is defined by the properties of the application,
+there is no one-size-fits-all way to determine what they are.
+
+### Completion point
+
+The completion point `T` is an arbitrarily chosen time in the consistency range.
+It is the point in time to which data can safely be backfilled, ensuring that
+there is no data loss.
+
+The completion point should be expressed as the type of the `time` column of
+the hypertables to be backfilled. For instance, if you're using a `TIMESTAMPTZ`
+`time` column, then the completion point may be `2023-08-10T12:00:00.00Z`. If
+you're using a `BIGINT` column it may be `1695036737000`.
+
+If you are using a mix of types for the `time` columns of your hypertables, you
+must determine the completion point for each type individually, and backfill
+each set of hypertables with the same type independently from those of other
+types.
+

_partials/_migrate_dual_write_switch_production_workload.md

@@ -0,0 +1,5 @@
+Once you've validated that all the data is present, and that the target
+database can handle the production workload, the final step is to switch to the
+target database as your primary. You may want to continue writing to the source
+database for a period, until you are certain that the target database is
+holding up to all production traffic.

_partials/_migrate_dual_write_validate_production_load.md

@@ -0,0 +1,6 @@
+Now that dual-writes have been in place for a while, the target database should
+be holding up to production write traffic. Now would be the right time to
+determine if the target database can serve all production traffic (both reads
+_and_ writes). How exactly this is done is application-specific and up to you
+to determine.
+

_partials/_migrate_explain_pg_dump_flags.md

@@ -0,0 +1,12 @@
+- `--no-tablespaces` is required because Timescale does not support
+  tablespaces other than the default. This is a known limitation.
+
+- `--no-owner` is required because Timescale's `tsdbadmin` user is not a
+  superuser and cannot assign ownership in all cases. This flag means that
+  everything is owned by the user used to connect to the target, regardless of
+  ownership in the source. This is a known limitation.
+
+- `--no-privileges` is required because Timescale's `tsdbadmin` user is not a
+  superuser and cannot assign privileges in all cases. This flag means that
+  privileges assigned to other users must be reassigned in the target database
+  as a manual clean-up task. This is a known limitation.

_partials/_migrate_from_timescaledb_version.md

@@ -0,0 +1,30 @@
+It is very important that the version of the TimescaleDB extension in the
+target database is the same as it was in the source database.
+
+You can determine the version of TimescaleDB in the source database with the
+following command:
+
+```bash
+psql $SOURCE -c "SELECT extversion FROM pg_extension WHERE extname = 'timescaledb';"
+```
+
+You can alter the extension version in Timescale with the following query:
+
+```bash
+psql $TARGET -c "ALTER EXTENSION timescaledb UPDATE TO '<version here>';"
+```
+
+The extension version must be present on Timescale in order for this to be
+successful. The following table gives an overview of the PostgreSQL version and
+lowest available TimescaleDB extension version:
+
+| pg12  | pg13  | pg14  | pg15  |
+|-------|-------|-------|-------|
+| 1.7.5 | 2.1.0 | 2.5.0 | 2.9.0 |
+
+[//]: # (Note: to update this table, consult https://timescale.slab.com/posts/migrations-and-version-compatibility-5red287x)
+
+By default, Timescale instances run on PostgreSQL version 15. If you require a
+lower version for a migration, open a support request.
+
+<OpenSupportRequest />

_partials/_migrate_open_support_request.md

@@ -0,0 +1,4 @@
+You can open a support request directly from the [Timescale console][support-link],
+or by email to [support@timescale.com](mailto:support@timescale.com).
+
+[support-link]: https://console.cloud.timescale.com/dashboard/support

_partials/_migrate_pg_dump_do_not_recommend_for_large_migration.md

@@ -0,0 +1,13 @@
+import OpenSupportRequest from "versionContent/_partials/_migrate_open_support_request.mdx"
+
+We do not recommend using this migration method to migrate more than
+100 GB of data, primarily because of the amount of downtime that it
+implies for your application, instead use the [dual-write and backfill]
+low-downtime migration solution. Should you nonetheless wish to migrate more
+than 400 GB of data with this method, open a support request to ensure
+that enough disk is pre-provisioned on your Timescale instance.
+
+<OpenSupportRequest />
+
+[dual-write and backfill]: /migrate/:currentVersion:/dual-write-and-backfill
+

_partials/_migrate_pg_dump_minimal_downtime.md

@@ -0,0 +1,4 @@
+For minimal downtime, the `pg_dump` and `psql` commands should be run from a
+machine with a low-latency, high-throughput link to the database that they are
+connected to. As Timescale instances run in the Amazon cloud, use an AWS EC2
+instance in the same region as your Timescale instance.

_partials/_migrate_set_up_source_and_target.md

@@ -0,0 +1,10 @@
+<Highlight type="note">
+For the sake of convenience, connection strings to the source and target
+databases are referred to as `$SOURCE` and `$TARGET` throughout this guide.
+This can be set in your shell, for example:
+
+```bash
+export SOURCE=postgres://<user>@<source host>:<source port>
+export TARGET=postgres://<user>@<target host>:<target port>
+```
+</Highlight>

_partials/_migrate_source_target_note.md

@@ -0,0 +1,5 @@
+<Highlight type="note">
+In the context of migrations, your existing production database is referred to
+as the "source" database, while the new Timescale database that you intend to
+migrate your data to is referred to as the "target" database.
+</Highlight>

api/timescaledb_post_restore.md

@@ -24,4 +24,4 @@ Prepare the database for normal use after a restore:
 SELECT timescaledb_post_restore();
 ```
 
-[backup-restore]: /use-timescale/:currentVersion:/migration/pg-dump-and-restore/
+[backup-restore]: /migrate/:currentVersion:/pg-dump-and-restore/

api/timescaledb_pre_restore.md

@@ -34,5 +34,5 @@ Prepare to restore the database:
 SELECT timescaledb_pre_restore();
 ```
 
-[backup-restore]: /use-timescale/:currentVersion:/migration/pg-dump-and-restore/
+[backup-restore]: /migrate/:currentVersion:/pg-dump-and-restore/
 [timescaledb_post_restore]: /api/:currentVersion:/administration/timescaledb_post_restore/