br: reduce memory footprint and HWM (high water mark)

[BornChanger]

Enhancement

BR can consume large size of volume especially at SAAS scenarios due to huge number of db/tables, temporary cache etc.
In fact, most of those memory is one time usage and can be allocated in smaller batches and also be released sooner.
In this issue, we track improvement on this area.

Common

• br: support to stop gc memory limit tuner when restore #51082

The GC memory limit tuner would adjust the golang GC memory limit to a value close to TiDB server environment memory instead of BR's. Besides, backup/restore is a task with a lot of temporary memory, which requires to trigger GC frequently. Therefore, PR#51082 forbidden the GC memory limit tuner in BR binary.

• feature: make stats export/import execution distributed #52781

Make stats export/import under DXF.

• br: Add goleak test for br integration test #52741

Catch possible goroutine leak

• br: Add goleak test for br integration test #52741

Automatically adjust GOMEMLIMIT for br clp

• br: set memory limit #53793

Backup

• br: pipeline backup schemas #43003
• br: public the parameter --use-backupmeta-v2 and use it in default #47114

Before v7.1.0, when the upstream cluster had a large number of wide tables, it was possible for BR to consume a lot of memory during the backup process. During a backup process, BR would keep three copies of the table information in memory:

1. The InfoSchema maintained by the background domain.
2. The information of the databases and tables being prepared for backup.
3. The serialized schema information before uploading them to external storage.

PR#43003 removes the aforementioned second point of table/databse information. Instead, it adopts a traversal execution approach to promptly release the memory of information of backed up tables.

PR#47114 removes the aforementioned third point of table/databse information. It saves the schema information into some files, and the size of each file is at most 128 MB.

For the aforementioned first point, we will use BRIE via SQL on TiDB in future, and the TiDB shares the domain with BR task.

Restore

• br: backup/restore statistic data in the partition dimension #49628
• statistic: support fine-grained dump/load statistics #49973
• br: reduce memory usage when get incomplete ranges #57192

There might be a table with a large size of statistics (sometimes the table has many partitions). BR uses a lot of memory when backup/restore the table.

PR#49973 supports to dump/load statistics in partition dimension.
PR#49628 supports for BR to persist/restore the statistics data in partition dimension.
PR#57192 prevent preallocating too much items and uses too much memory.

Log Task

• br: no domain to run log command #52127

There is no need to start domain for br log operation except log restore. PR#52127 stops to start domain and creates etcd client by br itself.

• br: make sure connection is closed #52594
  Make sure connection to TiKV stores all closed finally

BR in SQL

• br: record the memory consumption of br sql command #48046

Put BR in SQL under the memory quota control framework

[BornChanger]

/component br