Avoid using Get interface to obtain Metadata when iterating data during migration

[caipengbo]

Motivation

When migrating data, I found that when migrating the same number of keys (about 10W), the migration time was very variable, sometimes very short, sometimes very long. When the migration took a long time, I found that the CPU was full.

Looking at the flame diagram below:

we can see that there is a deep call stack on the left.

Zoomed in it:

When we parsed and sent the full data (SlotMigrate::SendSnapshot()), we used the rocksdb::DB::Get() interface to get the metadata, which was CPU consuming.

How much does this affect migration speed?

1. Because we set the iterator to not fill the cache, when the amount of data is larger than cache, it is very likely that we will call rocksdb::DB::Get() to read disk, which greatly affects the performance!
2. If the retrieval in memtable is bad( it depends on workload), then it will consume more CPU, which will bring more serious impact.

Solution

In fact, we do not need to get the value through the rocksdb::DB::Get() interface. When we iterate data, value and key are together, so we can directly get the value through the iterator and then encode into metadata.

[git-hulk]

Yes, good catch.

[datavisorxiaobiaozhao]

I have tested and confirmed that this is an issue. The fix works and can speed up migration by 10-100 times.Looking forward to your PR

[shangxiaoxiong]

I have tested and confirmed that this is an issue. The fix works and can speed up migration by 10-100 times.Looking forward to your PR

More than 10 times? Are you serious?

[caipengbo]

The original purpose of using rocksdb::DB::Get() to Get Metadata, I guess is to get the latest metadata (but mistakenly using the old snapshot).

The purpose of this is: snapshot is created at the beginning of the migration. If a slot has been migrated for a long time, users may change the number of subkeys in complex types or change the expiration time. If we use metadata in very old snapshot to determine the expiration time, we may make a mistake. The subkey cannot be migrated to the target end. WAL only records metadata changes, not subkeys. In this case, subkeys will be completely lost.

However, as we discussed in #906, even if we get the latest snaphot, we still can't prevent users from modifying complex types of metadata. We still make errors in judgment.

As of now, this problem is unavoidable, and we welcome discussion of effective solutions.