Add tt migrations #4499

p7nov · 2024-09-12T11:57:20Z

Resolves https://github.com/tarantool/enterprise_doc/issues/269, https://github.com/tarantool/enterprise_doc/issues/277, https://github.com/tarantool/enterprise_doc/issues/278, https://github.com/tarantool/enterprise_doc/issues/279

Add new tt reference page Managing migrations
Revise and update Migrations page
Write tutorials on using tt migrations: Centralized migrations with tt:
- Basic migration tutorial
- Migrations with space.upgrade
- Extending the cluster
- Troubleshooting migrations

Deployment: https://docs.d.tarantool.io/en/doc/gh-ee269-tt-migrations/tooling/tt_cli/migrations/

p7nov · 2024-09-30T08:46:50Z

doc/tooling/tt_cli/migrations.rst

+Q: can any migrations in a batch complete successfully? If I apply 2 migrations and call
+`tt migrations stop` after the first one is finished without errors, what are migration statuses?


@DifferentialOrange please provide the answer.

If the first migration has been applied and the second one is interrupted with tt migrations stop, the first one will have APPLIED status and the second one will be in APPLY_STARTED.

p7nov · 2024-09-30T08:46:56Z

doc/tooling/tt_cli/migrations.rst

+-   command options ``--config-storage-username`` and ``--config-storage-password``
+-   the etcd URI, for example, ``https://user:pass@localhost:2379/myapp``
+
+Q: which way has a higher priority?


@DifferentialOrange please provide the answer.

The priority of credentials: environment variables < command flags < URL credentials.

https://github.com/tarantool/tt-ee/blob/4c59e56529d6e4f4e4e514391510ebeef67f9d10/cli/migrations/cmd.go#L85-L86

DifferentialOrange

Thanks for the PR! Seems valid in general, have left several comments.

doc/tooling/tt_cli/migrations.rst

DifferentialOrange · 2024-09-30T10:33:32Z

doc/tooling/tt_cli/migrations.rst

+.. warning::
+
+    Using the options that ignore checks when publishing migration may cause
+    migration inconsistency in the etcd storage.


It's not about etcd storage, it's more about the whole cluster. For example, using the --ignore-order-violation option in the case described above may result in the following:

replicaset 1 (was in the cluster from the start) has migration 003 applied before 002

replicaset 2 (was added later) has migration 002 applied before 003

For example, if 002 adds some data and 003 reverts it, the data will be different between replicasets. etcd storage will be fine (btw, it's just a key-value storage, it will always be fine), but the cluster schema isn't.

DifferentialOrange · 2024-09-30T10:38:06Z

doc/tooling/tt_cli/migrations.rst

+Q: can any migrations in a batch complete successfully? If I apply 2 migrations and call
+`tt migrations stop` after the first one is finished without errors, what are migration statuses?


If the first migration has been applied and the second one is interrupted with tt migrations stop, the first one will have APPLIED status and the second one will be in APPLY_STARTED.

DifferentialOrange · 2024-09-30T10:50:46Z

doc/platform/ddl_dml/migrations/basic_migrations_tt.rst

+The centralized migration mechanism works with Tarantool EE clusters that:
+
+-   use etcd as a centralized configuration storage
+-   use the `CRUD <https://github.com/tarantool/crud>`__ module for data distribution


Once again we're in a weird situation when the module is actually out of support (in favor of crud-ee), but it's replacement is unavailable for users (it's removed from customer zone SDKs due to recent Team Product policies).

doc/platform/ddl_dml/migrations/upgrade_migrations_tt.rst

DifferentialOrange · 2024-09-30T11:14:47Z

doc/platform/ddl_dml/migrations/troubleshoot_migrations_tt.rst

+                          --force-reapply
+
+If execution of the incorrect migration version has failed, you may also need to add
+the ``--ignore-preceding-status`` option:


--ignore-preceding-status is needed only in case we've got something like this:

000004 is FAILED

000005 is FAILED

we try to reapply 000005 and, for some reason, don't care about 000004 for now.

doc/platform/ddl_dml/migrations/troubleshoot_migrations_tt.rst

DifferentialOrange · 2024-09-30T11:19:04Z

doc/platform/ddl_dml/migrations/troubleshoot_migrations_tt.rst

+    $ tt migrations stop "http://app_user:config_pass@localhost:2379/myapp" \
+                          --tarantool-username=client --tarantool-password=secret
+
+To avoid such situations in the future, restrict the maximum migration execution time


Well, any unfinished migration is bad, since APPLY_STARTED is a cluster inconsistency. Moreover, timeouted migration may be even a bit worse, since iproto requests are not cancelled when the client gets a timeout. It is highly likely that you'll need to call tt migrations stop after a timeout failure to do something.

Co-authored-by: Georgy Moiseev <moiseev.georgii@gmail.com>

xuniq

LGTM, left some comments

xuniq · 2024-10-03T07:26:44Z

doc/platform/ddl_dml/migrations/index.rst

+        local users = box.space.users
+        local fmt = users:format()
+
+        table.insert(fmt, { name = 'age', type = 'number', is_nullable = true })
+        users:format(fmt)


Nit

The example doesn't show the initial space format before the insertion. It may be confusing for the reader: some field is added to the end of a space, but you don't see the actual space it is added to.

In other migration docs, you use the writers space as an example. I suggest using it here as well (for consistency).

Not sure the initial format is important for this example. We're showing how easy it is to add a field (just 4 lines). With the initial format, the example would look much heavier without additional meaning.

Will use writers, thanks.

xuniq · 2024-10-03T07:45:01Z

doc/platform/ddl_dml/migrations/troubleshoot_migrations_tt.rst

+..  _centralized_migrations_tt_troubleshoot:
+
+Troubleshooting migrations
+--------------------------


Level 1 heading: "==========="

xuniq · 2024-10-03T07:45:12Z

doc/platform/ddl_dml/migrations/troubleshoot_migrations_tt.rst

+..  _centralized_migrations_tt_troubleshoot_publish:
+
+Incorrect migration published
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~


Level 2 heading: "----------"

doc/platform/ddl_dml/migrations/index.rst

doc/tooling/tt_cli/migrations.rst

doc/platform/ddl_dml/migrations/basic_migrations_tt.rst

Co-authored-by: Kseniia Antonova <73473519+xuniq@users.noreply.github.com>

Resolves tarantool/enterprise_doc#269, tarantool/enterprise_doc#277, tarantool/enterprise_doc#278, tarantool/enterprise_doc#279 Co-authored-by: Georgy Moiseev <moiseev.georgii@gmail.com> Co-authored-by: Kseniia Antonova <73473519+xuniq@users.noreply.github.com>

p7nov added 8 commits September 12, 2024 18:57

Add tt migrations

d9a8b4d

Migration tutorial draft

1e54dae

Migration tutorial draft

377c0d5

Migration tutorial draft

0463c1c

Migration tutorial draft

4b30f69

Migrations structure

ba2a9fd

Fix

0d20803

Fix

ab1867c

p7nov requested a review from DifferentialOrange September 30, 2024 08:45

p7nov commented Sep 30, 2024

View reviewed changes

DifferentialOrange approved these changes Sep 30, 2024

View reviewed changes

p7nov and others added 4 commits October 1, 2024 12:35

Fix missing newlines

d9827e5

Apply suggestions from code review

8aa777c

Co-authored-by: Georgy Moiseev <moiseev.georgii@gmail.com>

Review fixes

6f9be5e

Fix

c1405b6

p7nov requested a review from xuniq October 1, 2024 07:36

xuniq approved these changes Oct 3, 2024

View reviewed changes

p7nov and others added 4 commits October 3, 2024 16:29

Apply suggestions from code review

c616d6f

Co-authored-by: Kseniia Antonova <73473519+xuniq@users.noreply.github.com>

Fix

52b7b98

Fix

5350608

Fix list numbering

5a4a414

p7nov merged commit 344ba58 into latest Oct 3, 2024
1 check passed

p7nov deleted the gh-ee269-tt-migrations branch October 3, 2024 14:56

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add tt migrations #4499

Add tt migrations #4499

p7nov commented Sep 12, 2024 •

edited

Loading

p7nov Sep 30, 2024

DifferentialOrange Sep 30, 2024

p7nov Sep 30, 2024

DifferentialOrange Sep 30, 2024

DifferentialOrange left a comment

DifferentialOrange Sep 30, 2024

DifferentialOrange Sep 30, 2024

DifferentialOrange Sep 30, 2024

DifferentialOrange Sep 30, 2024

DifferentialOrange Sep 30, 2024

xuniq left a comment

xuniq Oct 3, 2024

p7nov Oct 3, 2024

xuniq Oct 3, 2024

xuniq Oct 3, 2024

		Q: can any migrations in a batch complete successfully? If I apply 2 migrations and call
		`tt migrations stop` after the first one is finished without errors, what are migration statuses?

Add tt migrations #4499

Add tt migrations #4499

Conversation

p7nov commented Sep 12, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DifferentialOrange left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xuniq left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

p7nov commented Sep 12, 2024 •

edited

Loading