xtrabackup: Better support for large datasets

[enisoc]

Changes to improve the Vitess xtrabackup integration for use with large datasets:

• Stream stderr in the background instead of waiting until the end. This is needed for long-running backups so that the xtrabackup process doesn't block after the write buffer fills up. It's also nice for checking in on progress during a long upload.
• Use move-back instead of copy-back so the disk doesn't need 2x the space to restore. We download backups from remote storage on every restore, so there's no need to keep a copy of the original downloaded files on local disk.
• Store stream mode (tar vs xbstream) in the manifest so going forward it will be possible to restore from either one, regardless of the current flag setting for creating new backups.
• Support optional data striping to parallelize compression/decompression and file upload/download. The striping parameters are stored in the manifest so the flags for new backups don't have to match in order to restore an old one.

Fixes #5063

Signed-off-by: Anthony Yeh enisoc@planetscale.com

[deepthi]

looks good so far

[enisoc]

I added some more changes to hopefully fix this for long restores too. I'll retry the large test DB with both this and #5066 in a custom build.

I also changed how we search for the replication position in the xtrabackup log since in my first test it found "" (empty string) and considered that a valid position (empty GTID set). I don't think we want to allow that.

[enisoc]

The backup side of this worked well on our 250G/shard test DB, which takes 1.5hrs to back up. I'll report back once I've tested the restore side.

[enisoc]

This should be ready for review now. I ended up broadening the scope of this PR to generally supporting large backups/restores with xtrabackup. I've tested it on our 250GB/shard keyspace and it passed.

The data striping seems to get us back to parity with backup/restore times for the same keyspace using the built-in backup engine (which compresses and uploads each file in the data dir independently). Before adding striping, xtrabackup was between 2x and 8x slower on my test keyspace because decompression and upload/download were single-threaded.

The xbstream format could technically support parallel compression/decompression on its own without striping, but not without extra disk space to store the compressed and decompressed content of a given file at the same time. You either risk running out of disk space and failing to restore, or you run with extra disk space that's only used during restore and is wasted otherwise. Also, without striping, even parallel xbstream would still bottleneck into a single destination file upload/download. Blob stores like S3 and GCS get better throughput across multiple files than for a single file.

[deepthi]

Nice work! LGTM.