Allow a migration to resurrect under a new gh-ost process

[jonahberquist]

We've previously discussed in passing the potential to do this. Our conclusion at the time was that it would be possible, though not particularly useful. The use case that I had thought of at the time was resuming if a gh-ost process died.

I have a more interesting use case now, which is the desire to change gh-ost itself during a long migration.

As an example, if during the course of a 25 hour migration, I decide I want to change the value of a configuration option to something less than a hard coded minimum, I am out of luck. There are few of these hard coded configuration options, but I can imagine other times where a behavior change that would require a change in gh-ost itself could be beneficial to an ongoing migration.

@shlomi-noach, imagine having this in conjunction with branch deploys.

I would be happy to work on the development of this feature.

[SuperQ]

I guess the easiest thing for this would be to write out a checkpoint file containing the necessary information. It could be written to a gh-ost checkpoint table in mysql, but I think a local file would be better.

[pbitty]

This would be a very nice feature to have. We've been running migrations as jobs on our compute cluster. We could treat this as a resumable job and wouldn't have to allocate a specific long-running host to run the job - if a host is set to be terminated and the job gets moved to a different host, it would just resume automagically.

I'd be happy to help develop/test this.

I would vote for having the checkpoint data stored in a table instead of a file, so the job itself doesn't need to keep any state.

[shlomi-noach]

Terminology wise let's find a different name for this since pause/resume are already used to describe throttling/unthrottling.

[SuperQ]

I would call it shutdown checkpoint and restart from checkpoint.

[shlomi-noach]

In order to make that happen, a new invocation needs to have the context to a previously interrupted migration. On the grander scale, this must include:

• The last processed copy-range
• The last applied bin log position

On the finer scale we are also interested in:

• reading the unique-key by which migration chooses to iterate
• "last processed copy-range" must be typed values
• last applied bin log position -- not too very important. Turns out RBR is idempotent. OK to resume from an already applied position and re-apply it.
• For that matter, same goes for rowcopy-range. This means we don't strictly need fsync or two-phase commit for everything. Periodic flush of status is good enough.
• all configuration
• all calculated values (number of rows? applied rows? time estimate?)

I would go as far as to suggest that the entire MigrationContext struct should be dumped and restored. (also see https://github.com/github/gh-ost/blob/master/go/cmd/gh-ost/main.go#L44)

Let's begin with dumping to file; let's begin with an interactive command such as hibernate or whatever, something orderly. Such that the next execution sees "oh there's a file indicating an interrupted migration". Let me read it, delete it and proceed.

[SuperQ]

👍 Seems like a sensible plan to me. Some google searching suggests that gob or json are the correct thing to write out.. I would suggest a text format like json.

[jonahberquist]

Dumping everything in MigrationContext makes sense to me. 👍

Something that I waned to note was that I mentioned wanting to change gh-ost and branch deploy mid-migration, and over the weekend I realized this was pure folly. Doing that throws away the whole concept we have of testing migrations and knowing that we have the correct data in the ghost table.

let's begin with an interactive command such as hibernate or whatever, something orderly

👍 hibernate works for me

As for file vs table, I am inclined to put this in a table. I think it makes this feature more flexible. @pbitty's use case is a good one. Another would be a host failure where gh-ost is running.

[shlomi-noach]

Following up on a discussion we had yesterday, an excellent use case for this feature is gh-ost bailing out on critical-load. Yesterday we had a 4 hour migration legitimately failing having met critical-load. That was during the cut-over phase -- at the very end of the migration. No problem, we ran the migration again and it worked -- but it would have been so nice if we could just resume from that point.

Point being: we don't see gh-ost crash unexpectedly, but it can bail out legitimately, in which case we might want to resume from same point in time.

[shlomi-noach]

Hey we need terminology. "resurrect"?

[jonahberquist]

resurrect

Haha, that's more zombie than ghost, but yes, I think we found it.