backupccl: RESTORE slowing at end

[maddyblue]

During a TPCC restore, we noticed that the speed of the restore of a table slows way down near the end. Looking at a goroutine dump, a single node (out of a 24 node cluster) had hundreds of goroutines in beginLimitedRequest, which throttles import requests to one at a time.

Next question was: why did that node (node 9) have so many requests after a scatter had run. Doing a range dump for the new table, we made a histogram with the number of leader leases per node of the new table:

This indeed does show node 9 having a much higher number of leader leases compared to other nodes.

Our guess is that scatter is indeed scattering, but since various restores have already happened, it is possible that scatter has no choice but to put the new ranges onto some underfull nodes, which makes the restore eventually take a long time on that node. We don't think there's any specific thing to do regarding this scatter problem at the current time.

However, we do think it's worth removing or upping the limit in beginLimitedRequest since we have some other limits during addsstable that may make the beginLimitedRequest throttling unnecessary.

Epic CRDB-6406

[maddyblue]

cockroach/pkg/ccl/storageccl/import.go

Line 40 in fe81631

var importRequestLimit = 1

should be changed to numcpu probably during the test.

[petermattis]

I think the problem here is that scattering a small number of ranges (hundreds) in a cluster containing tens of thousands of ranges will often be a no-op. See #23358 (comment) for a suggestion about plumbing a new flag down to SCATTER so that we can force replicas to be spread out across a cluster. This would be helpful for manually ensuring a table is spread across a cluster and very helpful for RESTORE.

[awoods187]

I observe this on all of my TPC-C restores of 10k. The last 0.2tb take ~40m when the first 2tb take 120m.

[tbg]

Saw this in the CPU usage graph in roachtest/restore2TB. Not sure if this is what was talked about before or if it's some replication hiccup, but here we go.

[awoods187]

I can see similar things on master. For example:

The job has been at 99% for at least the last 2 hours:

But it is, ever so glacially slowly, making progress.

[tbg]

That looks like a worse problem. Is that cluster still up?

…

On Wed, Nov 14, 2018, 03:34 Andy Woods ***@***.*** wrote: I can see similar things on master. For example: [image: image] <https://user-images.githubusercontent.com/22278911/48456107-cd8f6f80-e78b-11e8-8160-6994c56534f9.png> The job has been at 99% for at least the last 2 hours: [image: image] <https://user-images.githubusercontent.com/22278911/48456121-dbdd8b80-e78b-11e8-80eb-82337b9688f4.png> — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#23509 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AE135O4AFibgh5MdXOn6iFnNBb14VSdHks5uu4EygaJpZM4SffCa> .

[tbg]

@awoods187 just sent me a cluster of his that had the same symptom. Not sure it's the root cause of the behavior he's seeing but the cluster had all the signs of #31875 (comment) which is hopefully fixed once that PR lands.

[danhhz]

The time-bound iterator workaround (#32909) eliminates at least one class of these tail issues for incremental backup. We're also seeing this behavior in restore, import, and non-incremental backup, so that was clearly not the only cause. Just FYI that it was a partial fix.

[mwang1026]

cc @pbardea thoughts on how relevant this still is with the work you did in 20.2?

[pbardea]

There are a few things going on here:

• AdminScatter was augmented with a flag that forces the leases of the target range to be scatter (this was added a while ago and I don't know if it was introduced after this bug was reported)
• The 20.2 refactor should enable all of the nodes to process the ranges that they're scattered as soon as they're available (ie they're not bottlenecked on other nodes)
• Further improvements based on recent restore performance investigation suggested that we could do a better job sizing the chunks that we hand out to worker nodes based on cluster size. backupccl: create more chunks on larger clusters #64067 improves that.

A lot of this issue is also covered in #63925. One thing that hasn't been benchmarked is the performance of restoring a table into a cluster that already holds a significant amount of data. My hypothesis is that the RandomizeLeases flag on the AdminScatter request should help ensure that the entries are probably distributed, but I can run a benchmark to confirm. If we see pretty even AddSSTable traffic across the nodes in that benchmark I think we can close out this issue.

[shermanCRL]

We think we’ve made substantial improvements here, will close for now. There is probably more to be done, but not urgently, and can re-open if need be.