statistics: Do not create pseudo statistics for the auto-analysis check process

[Rustin170506]

What problem does this PR solve?

Issue Number: ref #50132

Problem Summary:

What changed and how does it work?

In this PR, we solved two problems:

1. We shouldn't create pseudo stats during auto-analyze check processing. We only need the real row count of tables, so no need for pseudo stats. This would help us to reduce memory consumption and alleviate GC pressure.
2. We should analyze static partitions one by one instead of analyzing the entry global table again and again.

So I made the following changes:

1. I changed the refresher to use GetPartitionStatsForAutoAnalyze and GetTableStatsForAutoAnalyze to avoid creating the pseudo stats.
2. I added a new type of job for the static partition. It will execute the analysis jobs in these ways:
   a. analyze table %n.%n partition %n
   b. analyze table %n.%n partition %n index %n

In the old implementation, we wrongly analyze the entire table.

I also did some refactorings to help us construct the job.

• NewStaticPartitionTableAnalysisJob
• NewNonPartitionedTableAnalysisJob
• NewDynamicPartitionTableAnalysisJob

I think I can use generic to reduce the confusion of the job structure, but this PR is already big enough. So I will do it in my following PR.

Check List

Tests

• Unit test
• Integration test
• Manual test See: statistics: Do not create pseudo statistics for the auto-analysis check process #51479 (comment)
• No need to test

  • I checked and no code files have been changed.

Side effects

• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Breaking backward compatibility

Documentation

• Affects user behaviors
• Contains syntax changes
• Contains variable changes
• Contains experimental features
• Changes MySQL compatibility

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

[ti-chi-bot]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[tiprow]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[codecov]

Codecov Report

Merging #51479 (1d025d1) into master (62afab3) will increase coverage by 2.2252%.
Report is 28 commits behind head on master.
The diff coverage is 71.7514%.

Additional details and impacted files

@@               Coverage Diff                @@
##             master     #51479        +/-   ##
================================================
+ Coverage   70.7337%   72.9590%   +2.2252%     
================================================
  Files          1462       1464         +2     
  Lines        435276     440243      +4967     
================================================
+ Hits         307887     321197     +13310     
+ Misses       108095      99013      -9082     
- Partials      19294      20033       +739     

Flag	Coverage Δ
integration	48.9482% <0.0000%> (?)
unit	70.5100% <71.7514%> (-0.0258%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	53.9957% <ø> (ø)
parser	∅ <ø> (∅)
br	51.8399% <ø> (+6.0073%)	⬆️

[Rustin170506]

🔢 Self-check (PR reviewed by myself and ready for feedback.)

[Rustin170506]

Tested locally:

1. Enable priority queue: SET GLOBAL tidb_enable_auto_analyze_priority_queue=ON;
2. Create a table and insert data:

import pymysql
import random

# Connect to TiDB
conn = pymysql.connect(host='localhost', port=4000, user='root', passwd='', db='test')
cursor = conn.cursor()

# Create partitioned table
cursor.execute("""
    CREATE TABLE users (
      id INT NOT NULL,
      name VARCHAR(50) NOT NULL,
      age INT NOT NULL,
      country VARCHAR(2) NOT NULL
    )
    PARTITION BY RANGE (age) (
      PARTITION p0 VALUES LESS THAN (20),
      PARTITION p1 VALUES LESS THAN (40),
      PARTITION p2 VALUES LESS THAN (60)
    );
""")

# Generate and insert data
for i in range(2000):
  id = i
  name = 'User {}'.format(i)
  age = random.randint(15, 55)
  country = random.choice(['US', 'CN', 'IN'])

  cursor.execute("INSERT INTO users VALUES (%s, %s, %s, %s)", (id, name, age, country))

conn.commit()

print("Partitioned table created and data inserted successfully!")

conn.close()

3. Check the log

[2024/03/05 15:13:12.300 +08:00] [INFO] [refresher.go:94] ["Auto analyze triggered"] [category=stats] [job="TableAnalysisJob: {AnalyzeType: partition, Partitions: p1, p2, Schema: test, Table: users, TableID: 104, TableStatsVer: 2, ChangePercentage: 1.00, Weight: 1.3792}"]
[2024/03/05 15:13:12.340 +08:00] [INFO] [save.go:200] ["incrementally update modifyCount"] [category=stats] [tableID=106] [curModifyCnt=3896] [results.BaseModifyCnt=3896] [modifyCount=0]
[2024/03/05 15:13:12.341 +08:00] [INFO] [save.go:222] ["directly update count"] [category=stats] [tableID=106] [results.Count=3896] [count=3896]
[2024/03/05 15:13:12.495 +08:00] [INFO] [analyze.go:745] ["analyze table `test`.`users` has finished"] [partition=p1] ["job info"="auto analyze table all columns with 256 buckets, 500 topn, 1 samplerate"] ["start time"=2024/03/05 15:13:12.315 +08:00] ["end time"=2024/03/05 15:13:12.493 +08:00] [cost=178.039666ms] ["sample rate reason"="use min(1, 110000/3896) as the sample-rate=1"]
[2024/03/05 15:13:12.497 +08:00] [INFO] [save.go:200] ["incrementally update modifyCount"] [category=stats] [tableID=107] [curModifyCnt=3118] [results.BaseModifyCnt=3118] [modifyCount=0]
[2024/03/05 15:13:12.497 +08:00] [INFO] [save.go:222] ["directly update count"] [category=stats] [tableID=107] [results.Count=3118] [count=3118]
[2024/03/05 15:13:12.620 +08:00] [INFO] [analyze.go:745] ["analyze table `test`.`users` has finished"] [partition=p2] ["job info"="auto analyze table all columns with 256 buckets, 500 topn, 1 samplerate"] ["start time"=2024/03/05 15:13:12.339 +08:00] ["end time"=2024/03/05 15:13:12.618 +08:00] [cost=278.853375ms] ["sample rate reason"="use min(1, 110000/3118) as the sample-rate=1"]
[2024/03/05 15:13:12.713 +08:00] [WARN] [global_stats.go:143] ["missing partition stats when merging global stats"] [table=users] [item=columns] [missing="[\"partition `p0` column `id` hist and topN\",\"partition `p0` column `name` hist and topN\",\"partition `p0` column `age` hist and topN\",\"partition `p0` column `country` hist and topN\"]"]

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: hawkingrei, qw4990

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [hawkingrei,qw4990]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ti-chi-bot]

[LGTM Timeline notifier]

Timeline:

• 2024-03-05 09:50:10.336478295 +0000 UTC m=+160637.358724683: ☑️ agreed by qw4990.
• 2024-03-06 03:49:44.440413639 +0000 UTC m=+225411.462660026: ☑️ agreed by hawkingrei.

[hawkingrei]

/retest