lightning: optimize region split check logic (#30428)

[ti-srebot]

cherry-pick #30428 to release-5.3
You can switch your code base to this Pull Request by using git-extras:

# In tidb repo:
git pr https://github.com/pingcap/tidb/pull/30876

After apply modifications, you can push your change to this PR via:

git push git@github.com:ti-srebot/tidb.git pr/30876:release-5.3-92207005ec8e

---

What problem does this PR solve?

Issue Number: close #30018

Problem Summary: After tidb-lightning importing, there are a lot of empty region in tidb, and these regions won't automatically merge until 1h later.

What is changed and how it works?

• Decreate default region max-keys from 1_440_000 to 1_280_000.
• Compact the SSTs before ingest into pebble event if the data size is smaller so lightning can estimate region range better in this case.

Analysis:

In the current implementation, lightning will ingest a SST file into one region with at most 1440k kvs or 128MiB(with default region-split-size) in size. 1440k is the region auto split threshold that will cause tikv auto split.

With some data set, because lightning can accurately estimate the region range, so the kvs in a region range slightly exceeds the key count threshold, then lightning will ingest these keys into two region, one with 1440k kvs and another small region this pd will treat it as empty region. Thus after lightning import, the bigger region will cause tidb auto-split. Due to pd's region merge row, pd will prevent one of the new split region and the nearly empty region merging before 1h later.

Test result:
Before this PR, after lightning import there are a lot of empty region in grafana and tikv logs contains 1k+ log with following pattern:

[2021/11/29 21:47:16.871 +08:00] [INFO] [split_check.rs:289] ["update approximate size and keys with accurate value"] [keys=1440000] [size=132480000] [region_id=12554]

After this pr:

There are still empty regions after lightning import, but these regions can be merge in a few minutes. There are no region auto-split after import.

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• See PR description
• No code

Side effects

• Performance regression: Consumes more CPU
• Because we force tirgger compact even if the data set is relatively small (< 100GB), this will cause more CPU consume though is overhead is small.
• Performance regression: Consumes more Memory
• Breaking backward compatibility

Documentation

• Affects user behaviors
• Contains syntax changes
• Contains variable changes
• Contains experimental features
• Changes MySQL compatibility

Release note

Avoid tikv trigger auto region split by lower the ingest kv count threshold 

[ti-chi-bot]

[REVIEW NOTIFICATION]

This pull request has been approved by:

• glorv
• kennytm

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

[ti-srebot]

/run-all-tests

[ti-srebot]

@glorv you're already a collaborator in bot's repo.

[lichunzhu]

/merge

[ti-chi-bot]

This pull request has been accepted and is ready to merge.

Commit hash: cf38c22

[ti-chi-bot]

@ti-srebot: Your PR was out of date, I have automatically updated it for you.

At the same time I will also trigger all tests for you:

/run-all-tests

If the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.