tt: add command `tt upgrade` #936

mandesero · 2024-09-03T18:27:28Z

`tt upgrade` command steps:

For each replicaset:
- On the master instance:
  1. Execute the following commands sequentially:
```
box.schema.upgrade()
box.snapshot()
```
- On each replica:
  1. Wait until the replica applies all transactions produced by box.schema.upgrade() on the master by comparing the vector clocks (vclock).
  2. Once synchronization is confirmed, execute the following command on the replica:
```
box.snapshot()
```

If any errors occur during the upgrade process, the process will stop and an error report will be generated.

The replica is waiting for synchronization for Timeout seconds. The default value for Timeout is 5 seconds, but you can specify it manually using the --timeout option.

$tt upgrade [<APP_NAME>] --timeout 10

You can also specify which replicaset(s) to upgrade by using the --replicaset option.

$tt upgrade [<APP_NAME>] --replicaset <RS_NAME_1> -r <RS_NAME_2> ...

mandesero · 2024-09-06T15:44:03Z

Examples:

Case 1: OK

INSTANCE            STATUS   PID    MODE
 app2:storage-002-b  RUNNING  12556  RO
 app2:router-001-a   RUNNING  12560  RW
 app2:storage-001-a  RUNNING  12567  RO
 app2:storage-001-b  RUNNING  12554  RO
 app2:storage-002-a  RUNNING  12555  RW

$ tt upgrade app2
• storage-001: ok
• storage-002: ok
• router-001: ok

Case 2: More than one master in the same replicaset

INSTANCE            STATUS   PID    MODE
app2:storage-002-b  RUNNING  12556  RO
app2:router-001-a   RUNNING  12560  RW
app2:storage-001-a  RUNNING  12567  RO
app2:storage-001-b  RUNNING  12554  RW
app2:storage-002-a  RUNNING  12555  RW

$ tt upgrade app2
• storage-001: error
  ⨯ [storage-001]: app2:storage-001-a and app2:storage-001-b are both masters

Case 3: LSN didn't update

$ tt upgrade app2
• storage-001: error
   ⨯ [storage-001]: LSN wait timeout: error waiting LSN 2003085 in vclock component 1 on app2:storage-001-b: time quota 5 seconds exceeded

Case 4: There is a replicaset that does not have a master

 INSTANCE            STATUS   PID    MODE
 app2:storage-002-b  RUNNING  12556  RO
 app2:router-001-a   RUNNING  12560  RW
 app2:storage-001-a  RUNNING  12567  RO
 app2:storage-001-b  RUNNING  12554  RO
 app2:storage-002-a  RUNNING  12555  RO

$ tt upgrade app2
• storage-001: error
   ⨯ [storage-001]: has not master instance

Case 5: A non-existent replicaset was specified

$ tt upgrade app2 --replicaset foo
   ⨯ replicaset with alias "foo" doesn't exist

cli/upgrade/upgrade.go

00000000000000000000.snap

test/integration/replicaset/test_replicaset_upgrade.py

cli/replicaset/cmd/upgrade.go

oleg-jukovec · 2024-10-25T16:55:17Z

Please, rebase on the latest commit in the master branch.

mandesero · 2024-10-28T08:21:41Z

Please, rebase on the latest commit in the master branch.

Rebased.

oleg-jukovec

Thank you for the update. I have a couple of non-critical comments.
Please, rebase to the master too.

cli/replicaset/cmd/lua/upgrade.lua

cli/replicaset/cmd/upgrade.go

test/integration/replicaset/test_replicaset_upgrade.py

cli/replicaset/cmd/lua/upgrade.lua

cli/replicaset/cmd/upgrade.go

test/integration/replicaset/single-t2-app/init.lua

cli/replicaset/cmd/upgrade.go

@TarantoolBot

Part of tarantool#924 @TarantoolBot document Title: `tt replicaset upgrade` upgrades database schema. The `tt replicaset upgrade` command allows for a automate upgrade of each replicaset in a Tarantool cluster. The process is performed sequentially on the master instance and its replicas to ensure data consistency. Below are the steps involved: For Each Replicaset: - **On the Master Instance**: 1. Run the following commands in sequence to upgrade the schema and take a snapshot: ```lua box.schema.upgrade() box.snapshot() ``` - **On Each Replica**: 1. Wait for the replica to apply all transactions produced by the `box.schema.upgrade()` command executed on the master. This is done by monitoring the vector clocks (vclock) to ensure synchronization. 2. Once the repica has caught up, run the following command to take a snapshot: ```lua box.snapshot() ``` > **Error Handling**: If any errors occur during the upgrade process, the operation will halt, and an error report will be generated. --- - Timeout for Synchronization Replicas will wait for synchronization for a maximum of `Timeout` seconds. The default timeout is set to 5 seconds, but this can be adjusted manually using the `--timeout` option. **Example:** ```bash $ tt replicaset upgrade [<APP_NAME>] --timeout 10 ``` - Selecting Replicasets for Upgrade You can specify which replicaset(s) to upgrade by using the `--replicaset` or `-r` option to target specific replicaset names. **Example:** ```bash $ tt replicaset upgrade [<APP_NAME> | <URI>] --replicaset <RS_NAME_1> -r <RS_NAME_2> ... ``` This provides flexibility in upgrading only the desired parts of the cluster without affecting the entire system.

mandesero · 2024-11-18T11:07:24Z

It was found that sometimes the replicaset name might not reach tt for some reason. I simplified the check in the test so that the test wouldn't be flaky.

For example:

• 5b3a3d0d-5ee2-40d1-989e-d4d68687581e: ok
• router-001: ok
• storage-001: ok

But should be:

• storage-001: ok
• storage-002: ok
• router-001: ok

mandesero force-pushed the mandesero/gh-924-upgrade-db-schema branch 3 times, most recently from b10d9bd to 34a3589 Compare September 6, 2024 15:31

mandesero force-pushed the mandesero/gh-924-upgrade-db-schema branch 4 times, most recently from 2b33e94 to f2406ee Compare September 8, 2024 12:06

mandesero changed the title ~~tt: add command tt upgrade~~ tt: add command tt upgrade [WIP] Sep 9, 2024

mandesero force-pushed the mandesero/gh-924-upgrade-db-schema branch 3 times, most recently from 234b980 to e12fe07 Compare September 10, 2024 11:52

mandesero requested review from Totktonada and psergee September 10, 2024 13:01

Totktonada reviewed Sep 10, 2024

View reviewed changes