Online DDL: followups in multiple trajectories

[shlomi-noach]

Adding functionality to online DDL:

• TabletAlias identity, now persisted in backend _vt.schema_migrations table tablet column, indicating which tablet originated the migration. This will be useful when identifying, and recovering from, mid-migration failovers
• retries column, counting the number of retries for a specific migration

Updated changes summary in followup comment

[shlomi-noach]

This PR begins to accumulate multiple changes, so I'd like to get it reviewed now before it grows too much, and then followup on each of the changes in future PRs. Changes in this PR:

• Online DDL tracks identify of tablet (as string, e.g. zone1-0000000100) which spawned the migration
  column: tablet
• Online DDL tracks number of retries for a migration
  column: retries
• Online DDL tracks whether a migration failed due to tablet failure
  column: tablet_failure
• Online DDL executor identifies migrations which have failed due to tablet failure. For example, a master failover due to host crash fails the tablet and the migration. Executor identifies this situation and marks the migration both as failed as well as tablet_failure=1
• Online DDL Executor automatically retries a migration under the following conditions:
  • the migration failed
  • the migration is marked as tablet_failure=1, ie. it was started by a different tablet
  • the migration has not been retried yet (this is a temporary safety latch, to make sure we don't go infinite or too-many retries)
• Don't overwrites artifacts identities upon retry. Always append new artifacts. Unset cleanup_timestamp upon retry
  all this is done to avoid a situation where we retry a migration before the previously failed migration's artifacts have been cleaned up.
• Online DDL identifies to throttler using app=... and with the values online-ddl:gh-ost:<uuid> or online-ddl:pt-osc:<uuid>
• Initial support in throttler to throttling app by token. Such that we can throttle anything that is online-ddl or anything that is gh-ost or of course a specific migration by its UUID

[shlomi-noach]

Tracking issue: #6926

[shlomi-noach]

This PR will be accompanied with a documentation PR, followup in future comment.

[shlomi-noach]

Documentation in vitessio/website#571

[rohit-nayak-ps]

lgtm

[shlomi-noach]

Question: this PR introduces a new column (multiple, actually) in schema_migrations table. As part of VExec, vtctld runs an INSERT, and uses a value for the new column. That INSERT gets transported to vttablet which, again, via VExec, executes it on the underlying table.

Now, the issue is what happens if someone upgrades their vtctld but not yet vttablet. Then vttablet sees a query with a new column, which it cannot execute.

I seem to recall we stated that we support partial upgrades.

I wonder how I should go about solving this problem.

[deepthi]

Question: this PR introduces a new column (multiple, actually) in schema_migrations table. As part of VExec, vtctld runs an INSERT, and uses a value for the new column. That INSERT gets transported to vttablet which, again, via VExec, executes it on the underlying table.

Now, the issue is what happens if someone upgrades their vtctld but not yet vttablet. Then vttablet sees a query with a new column, which it cannot execute.

I seem to recall we stated that we support partial upgrades.

I wonder how I should go about solving this problem.

I think this pushes us in the direction of version-aware components. vtctld needs to know the version of the vttablets it is talking to and modify its behavior. OTOH we did say that online DDL is experimental in 8.0 so we could document this as an expected incompatibility. Though that just postpones the time when we would have to deal with it, so it might be a good idea to work out a canonical solution right now.

[shlomi-noach]

vttablet's VExec receiver is now capable of injecting new columns into insert statements. I've refactored the code to mitigate my concerns in #6901 (comment)

As of now, this PR does not change the query passed between vtctld and vttablet, and vttablet injects the missing column. Fortunately, the value of that column is known to vttablet (indeed, it is a tablet column, so as it stand, vttablet is the best authority for the value of that column).

This does not solve all possible futuristic problems: in the future we may actually want to inject more data on vtctld side, and will have to find a way to solve the Vexec compatibility issue. But for this PR this is enough, and paves the way for more flexibility.

[deepthi]

vttablet's VExec receiver is now capable of injecting new columns into insert statements. I've refactored the code to mitigate my concerns in #6901 (comment)

As of now, this PR does not change the query passed between vtctld and vttablet, and vttablet injects the missing column. Fortunately, the value of that column is known to vttablet (indeed, it is a tablet column, so as it stand, vttablet is the best authority for the value of that column).

This does not solve all possible futuristic problems: in the future we may actually want to inject more data on vtctld side, and will have to find a way to solve the Vexec compatibility issue. But for this PR this is enough, and paves the way for more flexibility.

Oh, this is clever 💯

[deepthi]

lgtm