Upgrade Downgrade testing for backups

[frouioui]

Description

This pull request adds the upgrade downgrade tests for the backup and restore functionalities of Vitess. The goal of the upgrade downgrade tests is to make sure Vitess functionalities continue working when certain components have different versions (version n for the current commit, and version n-1 for the previous release of Vitess).

Two new workflows are added:

• Upgrade Downgrade Testing - Backups - E2E uses the already-existing end-to-end tests
• Upgrade Downgrade Testing - Backups - Manual does the backup and restore while downgrading and upgrading VTTablet manually on a live cluster. The steps are detailed in this comment)

Upgrade Downgrade Testing - Backups - E2E

This test runs the end-to-end tests contained in packages go/test/endtoend/backup/vtbackup & go/test/endtoend/backup/transform. It runs once with vttablet n-1 and vtbackup n, then runs a second time with vttablet n and vtbackup n-1.

Upgrade Downgrade Testing - Backups - Manual

Setup

1. The test creates a sharded cluster with every component using version n.
2. It inserts some data in every table.
3. Proceeds to backup the whole cluster.
4. And then inserts one row in every table. The purpose is to make sure that when we restore the backup the new rows we inserted are not present in the tables.

Downgrade (n-1 restore from n)

1. After the setup, the test shutdowns all vttablets and make sure to remove the tablet directories from vtdataroot.
2. Then it re-creates the tablets using the version n-1 of vttablet. The tablets restore from the backup we made.
3. The test asserts that the correct number of rows are present in every table using a select count().
4. The test inserts one more row in every table.
5. It proceeds to backup the whole cluster again.
6. Then the test inserts one more row in every table again, these rows should not be present the next time we restore from the backup we just made.

Upgrade (n restore from n-1)

1. Once this is done, the test shutdowns all the vttablets. Again, it makes sure to remove the tablet directories from vttdataroot.
2. New vttablets are created by the test using the version n of vttablet. The tablets restore from the backup we made using vttablet n-1 in the Downgrade phase.
3. The test then asserts that the number of rows in every table is correct.

Related Issue(s)

• Following Add upgrade downgrade tests for reparent #9473 (cluster management) and Upgrade Downgrade Testing #9300 (query serving)
• Description of upgrade downgrade testing: Migration testing in CI #7344

Checklist

• Tests were added or are not required
• Documentation was added or is not required

[deepthi]

This is very good work.
I'm just wondering if the cluster we are creating needs to be this complicated. We are running MoveTables and Reshard on that cluster before we even start the backup flow.
Can we not simply create the sharded cluster we want without going through all these workflows? A problem in any of them can fail the upgrade/downgrade test (false positive).
I'm willing to consider having this merged as-is if you can simplify it later.

[GuptaManan100]

Apart from the requested changes, everything else looks good to me!

[frouioui]

@deepthi @GuptaManan100 - Thank you both for the review. Actually @deepthi you are right, the setup of the cluster was inefficient, via commit da541cd I simplified the whole process, going from ~2-5 mins to <1 min to set up the cluster and with a process that is less prone to errors