migration: add scaffolding for the migrations manager

[irfansharif]

The commits in this PR were pulled out of our original prototype of the
migrations manager (#56107). These are the "boring bits", and simply introduces
the scaffolding for the manager without materially hooking it up to anything.
It will be fleshed out in future PRs, following the direction set by our
prototype above. For more details, see the RFC (#48843).

    sql: add scaffolding for version upgrade hook

    This callback will be called after validating a `SET CLUSTER SETTING
    version` but before executing it. It will be used in future PRs to
    execute arbitrary migrations to allow us to eventually remove code to
    support legacy behavior.

    This diff was pulled out of the long-running migrations prototype
    (#56107). For more details, see the RFC (#48843).

    migration: introduce pkg/migrations

    Package migration captures the facilities needed to define and execute
    migrations for a crdb cluster. These migrations can be arbitrarily long
    running, are free to send out arbitrary requests cluster wide, change
    internal DB state, and much more. They're typically reserved for crdb
    internal operations and state. Each migration is idempotent in nature,
    is associated with a specific cluster version, and executed when the
    cluster version is made activate on every node in the cluster.

    Examples of migrations that apply would be migrations to move all raft
    state from one storage engine to another, or purging all usage of the
    replicated truncated state in KV. A "sister" package of interest is
    pkg/sqlmigrations.

    ---

    This commit only introduces the basic scaffolding and wiring from
    existing functionality. We'll flesh in the missing bits in future
    commits.

    migration: plumb in a dialer, executor, kv.DB, and liveness

    The migration manager will need all of the above in order to execute
    migrations. It'll need:
    - A `Dialer`, to send out migration RPCs to individual nodes in the
      cluster.
    - A handle on `Liveness`, to figure out which nodes are part of the
      cluster
    - An `Executor`, for migrations to inspect/set internal SQL state, and
      to log progress into a dedicated system table
    - A `kv.DB`, for migrations to inspect/set internal KV state, and to
      send out Migrate requests to ranges for execute below-Raft migrations

    For more details, see the RFC (#48843). The fully "fleshed" out version
    of this manager was originally prototyped in #56107. This PR is simply
    pulling out the boring bits from there to move things along.

[cockroach-teamcity]

This change is 

[tbg]

Reviewed 3 of 3 files at r1, 5 of 5 files at r2, 3 of 3 files at r3.
Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @irfansharif and @knz)

---

pkg/migration/manager.go, line 46 at r2 (raw file):

// Registry. Be sure to key it in with the new cluster version we just added.
// During cluster upgrades, once the operator is able to set a cluster version
// setting that's past the version that was introduced (typically the major

at or past, right? If I attach a migration to 21.1.0-185, it will be run at the step going from 21.1.0-184 to 21.1.0-185. If I introduce VersionRemoveLegacyTombstone, I want to have removed legacy tombstones when that version is active.

---

pkg/migration/manager.go, line 79 at r2 (raw file):

	// TODO(irfansharif): Should we inject every ctx here with specific labels
	// for each migration, so they log distinctly? Do we need an AmbientContext?
	_ = logtags.AddTag(ctx, "migration-mgr", nil)

I like the idea of annotating the context with the current work item. I think no ambient context is needed, the caller can pass a populated context already.

---

pkg/migration/manager.go, line 111 at r2 (raw file):

		// TODO(irfansharif): We'll also want a testing override here to be able
		// to stub out migrations as needed.
		_ = Registry[version]

I think the most canonical way to get this is to return &Manager{registry: Registry} in NewManager and in tests, you simply put another registry in. No need for knobs at all.

[knz]

I really like the idea of populating intermediate commits with API Stubs and "TODO" comments. This makes the review much easier. I will reuse your idea.

Reviewed 3 of 3 files at r1, 4 of 5 files at r2, 3 of 3 files at r3.
Reviewable status: complete! 1 of 0 LGTMs obtained (waiting on @irfansharif)

[irfansharif]

I will reuse your idea.

Heh, I remember getting this idea from you during a 1:1. Thanks for the reviews!

bors r+

Reviewable status: complete! 0 of 0 LGTMs obtained (and 1 stale) (waiting on @knz and @tbg)

---

pkg/migration/manager.go, line 46 at r2 (raw file):

Previously, tbg (Tobias Grieger) wrote…

at or past, right? If I attach a migration to 21.1.0-185, it will be run at the step going from 21.1.0-184 to 21.1.0-185. If I introduce VersionRemoveLegacyTombstone, I want to have removed legacy tombstones when that version is active.

yep, here the thing I was getting at was that the user would only be able to set major versions as the active cluster version (so 21.1.0 for e.g.), and not the intermediate versions directly. I still need to do some work to properly introduce these fence versions, and the semantics of wanting to have removed legacy tombstones once VersionRemoveLegacyTombstone is active is also what I'm aiming to go for. TODO.

[craig]

Build succeeded:

• GitHub CI (Cockroach)