Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve the performance of partition table in extreme case #4474

Closed
windtalker opened this issue Mar 28, 2022 · 0 comments · Fixed by #4988
Closed

Improve the performance of partition table in extreme case #4474

windtalker opened this issue Mar 28, 2022 · 0 comments · Fixed by #4988
Assignees
@bestwoody
Labels
type/enhancement The issue or PR belongs to an enhancement.

Comments

@windtalker
Copy link
Contributor

windtalker commented Mar 28, 2022 •
edited
Loading

Enhancement

In the performance tests for mpp partition table, we found that the performance of partition table will decline sharply in some extreme case:
In the test case, there are two tables:
partition_table: partitioned by partition_date into 60 partitions(range partition, one day for each partition)
non_partition_table: the same as partition_table except that it is not partitioned.
The data size in both table are 3kw.

The query is like
select sum(xx), some_other_date from partition_table where some_other_date > '2022-03-25 00:00:00' and some_other_date < '2022-03-25 23:59:59' group by some_other_date
The size of result set is 20w
Note some_other_date is not the partition key.

The query time is

partition table non partiton table
1.5s 0.4s

After some debug, I found the root cause is although some_other_date is not the partition key, but it has strong correlation with the partition key: the result of select min(partition_date), max(partition_date) from partition_table where partition_date > '2022-03-25 00:00:00' and partition_date < '2022-03-25 23:59:59' is

min(partition_date) max(partition_date)
2022-03-24 07:34:59 2022-03-25 13:37:30

That is the useful data only exists in 2 partitions.

For non-partition table, the TiFlash will use max_threads's pipeline to execute tablescan->filter->aggregation concurrently, and since it is not partitioned, the data distribution between these pipeline evenly. The following log prove this:

[2022/03/28 13:45:08.655 +08:00] [TRACE] [<unknown>] ["MPPTask:<query:432130309036441603,task:4> ParallelAggregating Aggregated. 80158 to 40176 rows (from 5.176 MiB) in 0.193 sec. (414802.564 rows/sec., 26.786 MiB/sec.)"] [thread_id=5634]
[2022/03/28 13:45:08.655 +08:00] [TRACE] [<unknown>] ["MPPTask:<query:432130309036441603,task:4> ParallelAggregating Aggregated. 27797 to 21238 rows (from 1.795 MiB) in 0.193 sec. (143844.243 rows/sec., 9.290 MiB/sec.)"] [thread_id=5634]
[2022/03/28 13:45:08.655 +08:00] [TRACE] [<unknown>] ["MPPTask:<query:432130309036441603,task:4> ParallelAggregating Aggregated. 73483 to 38565 rows (from 4.745 MiB) in 0.193 sec. (380260.695 rows/sec., 24.554 MiB/sec.)"] [thread_id=5634]
[2022/03/28 13:45:08.655 +08:00] [TRACE] [<unknown>] ["MPPTask:<query:432130309036441603,task:4> ParallelAggregating Aggregated. 49704 to 31182 rows (from 3.210 MiB) in 0.193 sec. (257208.845 rows/sec., 16.609 MiB/sec.)"] [thread_id=5634]
[2022/03/28 13:45:08.655 +08:00] [TRACE] [<unknown>] ["MPPTask:<query:432130309036441603,task:4> ParallelAggregating Aggregated. 84147 to 40920 rows (from 5.434 MiB) in 0.193 sec. (435444.888 rows/sec., 28.120 MiB/sec.)"] [thread_id=5634]
[2022/03/28 13:45:08.655 +08:00] [TRACE] [<unknown>] ["MPPTask:<query:432130309036441603,task:4> ParallelAggregating Aggregated. 62534 to 35749 rows (from 4.038 MiB) in 0.193 sec. (323601.681 rows/sec., 20.898 MiB/sec.)"] [thread_id=5634]
[2022/03/28 13:45:08.655 +08:00] [TRACE] [<unknown>] ["MPPTask:<query:432130309036441603,task:4> ParallelAggregating Aggregated. 93678 to 42804 rows (from 6.050 MiB) in 0.193 sec. (484766.019 rows/sec., 31.305 MiB/sec.)"] [thread_id=5634]
[2022/03/28 13:45:08.655 +08:00] [TRACE] [<unknown>] ["MPPTask:<query:432130309036441603,task:4> ParallelAggregating Aggregated. 32482 to 23742 rows (from 2.098 MiB) in 0.193 sec. (168088.237 rows/sec., 10.856 MiB/sec.)"] [thread_id=5634]
[2022/03/28 13:45:08.655 +08:00] [TRACE] [<unknown>] ["MPPTask:<query:432130309036441603,task:4> ParallelAggregating Aggregated. 50272 to 31647 rows (from 3.246 MiB) in 0.193 sec. (260148.138 rows/sec., 16.799 MiB/sec.)"] [thread_id=5634]
[2022/03/28 13:45:08.655 +08:00] [TRACE] [<unknown>] ["MPPTask:<query:432130309036441603,task:4> ParallelAggregating Aggregated. 59692 to 34807 rows (from 3.855 MiB) in 0.193 sec. (308894.865 rows/sec., 19.948 MiB/sec.)"] [thread_id=5634]
[2022/03/28 13:45:08.655 +08:00] [TRACE] [<unknown>] ["MPPTask:<query:432130309036441603,task:4> ParallelAggregating Aggregated. 65118 to 36395 rows (from 4.205 MiB) in 0.193 sec. (336973.394 rows/sec., 21.760 MiB/sec.)"] [thread_id=5634]
[2022/03/28 13:45:08.655 +08:00] [TRACE] [<unknown>] ["MPPTask:<query:432130309036441603,task:4> ParallelAggregating Aggregated. 65060 to 36383 rows (from 4.201 MiB) in 0.193 sec. (336673.255 rows/sec., 21.740 MiB/sec.)"] [thread_id=5634]
[2022/03/28 13:45:08.655 +08:00] [TRACE] [<unknown>] ["MPPTask:<query:432130309036441603,task:4> ParallelAggregating Aggregated. 43371 to 28923 rows (from 2.801 MiB) in 0.193 sec. (224436.762 rows/sec., 14.494 MiB/sec.)"] [thread_id=5634]
[2022/03/28 13:45:08.655 +08:00] [TRACE] [<unknown>] ["MPPTask:<query:432130309036441603,task:4> ParallelAggregating Aggregated. 49528 to 31363 rows (from 3.198 MiB) in 0.193 sec. (256298.078 rows/sec., 16.551 MiB/sec.)"] [thread_id=5634]
[2022/03/28 13:45:08.655 +08:00] [TRACE] [<unknown>] ["MPPTask:<query:432130309036441603,task:4> ParallelAggregating Aggregated. 19973 to 16399 rows (from 1.290 MiB) in 0.193 sec. (103356.516 rows/sec., 6.675 MiB/sec.)"] [thread_id=5634]
[2022/03/28 13:45:08.655 +08:00] [TRACE] [<unknown>] ["MPPTask:<query:432130309036441603,task:4> ParallelAggregating Aggregated. 76145 to 39296 rows (from 4.918 MiB) in 0.193 sec. (394036.044 rows/sec., 25.447 MiB/sec.)"] [thread_id=5634]
[2022/03/28 13:45:08.655 +08:00] [TRACE] [<unknown>] ["MPPTask:<query:432130309036441603,task:4> ParallelAggregating Aggregated. 37302 to 26134 rows (from 2.409 MiB) in 0.193 sec. (193030.829 rows/sec., 12.466 MiB/sec.)"] [thread_id=5634]
[2022/03/28 13:45:08.655 +08:00] [TRACE] [<unknown>] ["MPPTask:<query:432130309036441603,task:4> ParallelAggregating Aggregated. 68534 to 37246 rows (from 4.426 MiB) in 0.193 sec. (354650.551 rows/sec., 22.902 MiB/sec.)"] [thread_id=5634]
[2022/03/28 13:45:08.655 +08:00] [TRACE] [<unknown>] ["MPPTask:<query:432130309036441603,task:4> ParallelAggregating Aggregated. 64184 to 36077 rows (from 4.145 MiB) in 0.193 sec. (332140.120 rows/sec., 21.449 MiB/sec.)"] [thread_id=5634]
[2022/03/28 13:45:08.655 +08:00] [TRACE] [<unknown>] ["MPPTask:<query:432130309036441603,task:4> ParallelAggregating Aggregated. 84239 to 41019 rows (from 5.440 MiB) in 0.193 sec. (435920.971 rows/sec., 28.150 MiB/sec.)"] [thread_id=5634]
[2022/03/28 13:45:08.655 +08:00] [TRACE] [<unknown>] ["MPPTask:<query:432130309036441603,task:4> ParallelAggregating Aggregated. 54490 to 33110 rows (from 3.519 MiB) in 0.193 sec. (281975.495 rows/sec., 18.209 MiB/sec.)"] [thread_id=5634]
[2022/03/28 13:45:08.655 +08:00] [TRACE] [<unknown>] ["MPPTask:<query:432130309036441603,task:4> ParallelAggregating Aggregated. 81639 to 40532 rows (from 5.273 MiB) in 0.193 sec. (422466.460 rows/sec., 27.285 MiB/sec.)"] [thread_id=5634]
[2022/03/28 13:45:08.655 +08:00] [TRACE] [<unknown>] ["MPPTask:<query:432130309036441603,task:4> ParallelAggregating Aggregated. 49431 to 31324 rows (from 3.192 MiB) in 0.193 sec. (255796.122 rows/sec., 16.517 MiB/sec.)"] [thread_id=5634]
[2022/03/28 13:45:08.655 +08:00] [TRACE] [<unknown>] ["MPPTask:<query:432130309036441603,task:4> ParallelAggregating Aggregated. 68340 to 37361 rows (from 4.413 MiB) in 0.193 sec. (353646.638 rows/sec., 22.836 MiB/sec.)"] [thread_id=5634]
[2022/03/28 13:45:08.655 +08:00] [TRACE] [<unknown>] ["MPPTask:<query:432130309036441603,task:4> ParallelAggregating Aggregated. 38872 to 26819 rows (from 2.510 MiB) in 0.193 sec. (201155.284 rows/sec., 12.991 MiB/sec.)"] [thread_id=5634]
[2022/03/28 13:45:08.655 +08:00] [TRACE] [<unknown>] ["MPPTask:<query:432130309036441603,task:4> ParallelAggregating Aggregated. 41743 to 28214 rows (from 2.696 MiB) in 0.193 sec. (216012.169 rows/sec., 13.950 MiB/sec.)"] [thread_id=5634]
[2022/03/28 13:45:08.655 +08:00] [TRACE] [<unknown>] ["MPPTask:<query:432130309036441603,task:4> ParallelAggregating Aggregated. 39917 to 27369 rows (from 2.578 MiB) in 0.193 sec. (206562.962 rows/sec., 13.338 MiB/sec.)"] [thread_id=5634]
[2022/03/28 13:45:08.655 +08:00] [TRACE] [<unknown>] ["MPPTask:<query:432130309036441603,task:4> ParallelAggregating Aggregated. 50773 to 31686 rows (from 3.279 MiB) in 0.193 sec. (262740.719 rows/sec., 16.968 MiB/sec.)"] [thread_id=5634]
[2022/03/28 13:45:08.655 +08:00] [TRACE] [<unknown>] ["MPPTask:<query:432130309036441603,task:4> ParallelAggregating Aggregated. 63204 to 35782 rows (from 4.082 MiB) in 0.193 sec. (327068.804 rows/sec., 21.123 MiB/sec.)"] [thread_id=5634]
[2022/03/28 13:45:08.655 +08:00] [TRACE] [<unknown>] ["MPPTask:<query:432130309036441603,task:4> ParallelAggregating Aggregated. 27021 to 20767 rows (from 1.745 MiB) in 0.193 sec. (139828.589 rows/sec., 9.029 MiB/sec.)"] [thread_id=5634]
[2022/03/28 13:45:08.655 +08:00] [TRACE] [<unknown>] ["MPPTask:<query:432130309036441603,task:4> ParallelAggregating Aggregated. 36620 to 25847 rows (from 2.365 MiB) in 0.193 sec. (189501.608 rows/sec., 12.239 MiB/sec.)"] [thread_id=5634]
[2022/03/28 13:45:08.655 +08:00] [TRACE] [<unknown>] ["MPPTask:<query:432130309036441603,task:4> ParallelAggregating Aggregated. 40333 to 27477 rows (from 2.605 MiB) in 0.193 sec. (208715.684 rows/sec., 13.479 MiB/sec.)"] [thread_id=5634]
[2022/03/28 13:45:08.655 +08:00] [TRACE] [<unknown>] ["MPPTask:<query:432130309036441603,task:4> ParallelAggregating Aggregated. 66681 to 36857 rows (from 4.307 MiB) in 0.193 sec. (345061.625 rows/sec., 22.286 MiB/sec.)"] [thread_id=5634]
[2022/03/28 13:45:08.655 +08:00] [TRACE] [<unknown>] ["MPPTask:<query:432130309036441603,task:4> ParallelAggregating Aggregated. 64698 to 36334 rows (from 4.178 MiB) in 0.193 sec. (334799.973 rows/sec., 21.620 MiB/sec.)"] [thread_id=5634]
[2022/03/28 13:45:08.655 +08:00] [TRACE] [<unknown>] ["MPPTask:<query:432130309036441603,task:4> ParallelAggregating Aggregated. 44512 to 29255 rows (from 2.875 MiB) in 0.193 sec. (230341.222 rows/sec., 14.876 MiB/sec.)"] [thread_id=5634]
[2022/03/28 13:45:08.655 +08:00] [TRACE] [<unknown>] ["MPPTask:<query:432130309036441603,task:4> ParallelAggregating Aggregated. 41776 to 28275 rows (from 2.698 MiB) in 0.193 sec. (216182.937 rows/sec., 13.960 MiB/sec.)"] [thread_id=5634]

For partitioned table, the pipeline is first divided between partitions, since there is only 2 partition has data after filter, the data in the aggregation stage is very skewed, the following log shows it clearly:

[2022/03/28 13:45:00.019 +08:00] [TRACE] [<unknown>] ["MPPTask:<query:432130306537684995,task:1> ParallelAggregating Aggregated. 76324 to 39182 rows (from 4.929 MiB) in 1.138 sec. (67070.054 rows/sec., 4.332 MiB/sec.)"] [thread_id=440]
[2022/03/28 13:45:00.019 +08:00] [TRACE] [<unknown>] ["MPPTask:<query:432130306537684995,task:1> ParallelAggregating Aggregated. 0 to 0 rows (from 0.000 MiB) in 1.138 sec. (0.000 rows/sec., 0.000 MiB/sec.)"] [thread_id=440]
[2022/03/28 13:45:00.019 +08:00] [TRACE] [<unknown>] ["MPPTask:<query:432130306537684995,task:1> ParallelAggregating Aggregated. 0 to 0 rows (from 0.000 MiB) in 1.138 sec. (0.000 rows/sec., 0.000 MiB/sec.)"] [thread_id=440]
[2022/03/28 13:45:00.019 +08:00] [TRACE] [<unknown>] ["MPPTask:<query:432130306537684995,task:1> ParallelAggregating Aggregated. 0 to 0 rows (from 0.000 MiB) in 1.138 sec. (0.000 rows/sec., 0.000 MiB/sec.)"] [thread_id=440]
[2022/03/28 13:45:00.019 +08:00] [TRACE] [<unknown>] ["MPPTask:<query:432130306537684995,task:1> ParallelAggregating Aggregated. 0 to 0 rows (from 0.000 MiB) in 1.138 sec. (0.000 rows/sec., 0.000 MiB/sec.)"] [thread_id=440]
[2022/03/28 13:45:00.019 +08:00] [TRACE] [<unknown>] ["MPPTask:<query:432130306537684995,task:1> ParallelAggregating Aggregated. 0 to 0 rows (from 0.000 MiB) in 1.138 sec. (0.000 rows/sec., 0.000 MiB/sec.)"] [thread_id=440]
[2022/03/28 13:45:00.019 +08:00] [TRACE] [<unknown>] ["MPPTask:<query:432130306537684995,task:1> ParallelAggregating Aggregated. 0 to 0 rows (from 0.000 MiB) in 1.138 sec. (0.000 rows/sec., 0.000 MiB/sec.)"] [thread_id=440]
[2022/03/28 13:45:00.019 +08:00] [TRACE] [<unknown>] ["MPPTask:<query:432130306537684995,task:1> ParallelAggregating Aggregated. 0 to 0 rows (from 0.000 MiB) in 1.138 sec. (0.000 rows/sec., 0.000 MiB/sec.)"] [thread_id=440]
[2022/03/28 13:45:00.019 +08:00] [TRACE] [<unknown>] ["MPPTask:<query:432130306537684995,task:1> ParallelAggregating Aggregated. 0 to 0 rows (from 0.000 MiB) in 1.138 sec. (0.000 rows/sec., 0.000 MiB/sec.)"] [thread_id=440]
[2022/03/28 13:45:00.019 +08:00] [TRACE] [<unknown>] ["MPPTask:<query:432130306537684995,task:1> ParallelAggregating Aggregated. 0 to 0 rows (from 0.000 MiB) in 1.138 sec. (0.000 rows/sec., 0.000 MiB/sec.)"] [thread_id=440]
[2022/03/28 13:45:00.019 +08:00] [TRACE] [<unknown>] ["MPPTask:<query:432130306537684995,task:1> ParallelAggregating Aggregated. 0 to 0 rows (from 0.000 MiB) in 1.138 sec. (0.000 rows/sec., 0.000 MiB/sec.)"] [thread_id=440]
[2022/03/28 13:45:00.019 +08:00] [TRACE] [<unknown>] ["MPPTask:<query:432130306537684995,task:1> ParallelAggregating Aggregated. 0 to 0 rows (from 0.000 MiB) in 1.138 sec. (0.000 rows/sec., 0.000 MiB/sec.)"] [thread_id=440]
[2022/03/28 13:45:00.019 +08:00] [TRACE] [<unknown>] ["MPPTask:<query:432130306537684995,task:1> ParallelAggregating Aggregated. 0 to 0 rows (from 0.000 MiB) in 1.138 sec. (0.000 rows/sec., 0.000 MiB/sec.)"] [thread_id=440]
[2022/03/28 13:45:00.019 +08:00] [TRACE] [<unknown>] ["MPPTask:<query:432130306537684995,task:1> ParallelAggregating Aggregated. 0 to 0 rows (from 0.000 MiB) in 1.138 sec. (0.000 rows/sec., 0.000 MiB/sec.)"] [thread_id=440]
[2022/03/28 13:45:00.019 +08:00] [TRACE] [<unknown>] ["MPPTask:<query:432130306537684995,task:1> ParallelAggregating Aggregated. 0 to 0 rows (from 0.000 MiB) in 1.138 sec. (0.000 rows/sec., 0.000 MiB/sec.)"] [thread_id=440]
[2022/03/28 13:45:00.019 +08:00] [TRACE] [<unknown>] ["MPPTask:<query:432130306537684995,task:1> ParallelAggregating Aggregated. 738791 to 68382 rows (from 47.710 MiB) in 1.138 sec. (649215.877 rows/sec., 41.925 MiB/sec.)"] [thread_id=440]
[2022/03/28 13:45:00.019 +08:00] [TRACE] [<unknown>] ["MPPTask:<query:432130306537684995,task:1> ParallelAggregating Aggregated. 0 to 0 rows (from 0.000 MiB) in 1.138 sec. (0.000 rows/sec., 0.000 MiB/sec.)"] [thread_id=440]
[2022/03/28 13:45:00.019 +08:00] [TRACE] [<unknown>] ["MPPTask:<query:432130306537684995,task:1> ParallelAggregating Aggregated. 0 to 0 rows (from 0.000 MiB) in 1.138 sec. (0.000 rows/sec., 0.000 MiB/sec.)"] [thread_id=440]
[2022/03/28 13:45:00.019 +08:00] [TRACE] [<unknown>] ["MPPTask:<query:432130306537684995,task:1> ParallelAggregating Aggregated. 0 to 0 rows (from 0.000 MiB) in 1.138 sec. (0.000 rows/sec., 0.000 MiB/sec.)"] [thread_id=440]
[2022/03/28 13:45:00.019 +08:00] [TRACE] [<unknown>] ["MPPTask:<query:432130306537684995,task:1> ParallelAggregating Aggregated. 0 to 0 rows (from 0.000 MiB) in 1.138 sec. (0.000 rows/sec., 0.000 MiB/sec.)"] [thread_id=440]
[2022/03/28 13:45:00.019 +08:00] [TRACE] [<unknown>] ["MPPTask:<query:432130306537684995,task:1> ParallelAggregating Aggregated. 0 to 0 rows (from 0.000 MiB) in 1.138 sec. (0.000 rows/sec., 0.000 MiB/sec.)"] [thread_id=440]
[2022/03/28 13:45:00.019 +08:00] [TRACE] [<unknown>] ["MPPTask:<query:432130306537684995,task:1> ParallelAggregating Aggregated. 0 to 0 rows (from 0.000 MiB) in 1.138 sec. (0.000 rows/sec., 0.000 MiB/sec.)"] [thread_id=440]
[2022/03/28 13:45:00.019 +08:00] [TRACE] [<unknown>] ["MPPTask:<query:432130306537684995,task:1> ParallelAggregating Aggregated. 31862 to 23385 rows (from 2.057 MiB) in 1.138 sec. (27998.874 rows/sec., 1.808 MiB/sec.)"] [thread_id=440]
[2022/03/28 13:45:00.019 +08:00] [TRACE] [<unknown>] ["MPPTask:<query:432130306537684995,task:1> ParallelAggregating Aggregated. 0 to 0 rows (from 0.000 MiB) in 1.138 sec. (0.000 rows/sec., 0.000 MiB/sec.)"] [thread_id=440]
[2022/03/28 13:45:00.019 +08:00] [TRACE] [<unknown>] ["MPPTask:<query:432130306537684995,task:1> ParallelAggregating Aggregated. 0 to 0 rows (from 0.000 MiB) in 1.138 sec. (0.000 rows/sec., 0.000 MiB/sec.)"] [thread_id=440]
[2022/03/28 13:45:00.019 +08:00] [TRACE] [<unknown>] ["MPPTask:<query:432130306537684995,task:1> ParallelAggregating Aggregated. 0 to 0 rows (from 0.000 MiB) in 1.138 sec. (0.000 rows/sec., 0.000 MiB/sec.)"] [thread_id=440]
[2022/03/28 13:45:00.019 +08:00] [TRACE] [<unknown>] ["MPPTask:<query:432130306537684995,task:1> ParallelAggregating Aggregated. 0 to 0 rows (from 0.000 MiB) in 1.138 sec. (0.000 rows/sec., 0.000 MiB/sec.)"] [thread_id=440]
[2022/03/28 13:45:00.019 +08:00] [TRACE] [<unknown>] ["MPPTask:<query:432130306537684995,task:1> ParallelAggregating Aggregated. 26031 to 20123 rows (from 1.681 MiB) in 1.138 sec. (22874.857 rows/sec., 1.477 MiB/sec.)"] [thread_id=440]
[2022/03/28 13:45:00.019 +08:00] [TRACE] [<unknown>] ["MPPTask:<query:432130306537684995,task:1> ParallelAggregating Aggregated. 0 to 0 rows (from 0.000 MiB) in 1.138 sec. (0.000 rows/sec., 0.000 MiB/sec.)"] [thread_id=440]
[2022/03/28 13:45:00.019 +08:00] [TRACE] [<unknown>] ["MPPTask:<query:432130306537684995,task:1> ParallelAggregating Aggregated. 0 to 0 rows (from 0.000 MiB) in 1.138 sec. (0.000 rows/sec., 0.000 MiB/sec.)"] [thread_id=440]
[2022/03/28 13:45:00.019 +08:00] [TRACE] [<unknown>] ["MPPTask:<query:432130306537684995,task:1> ParallelAggregating Aggregated. 32304 to 23623 rows (from 2.086 MiB) in 1.138 sec. (28387.284 rows/sec., 1.833 MiB/sec.)"] [thread_id=440]
[2022/03/28 13:45:00.019 +08:00] [TRACE] [<unknown>] ["MPPTask:<query:432130306537684995,task:1> ParallelAggregating Aggregated. 1397174 to 85606 rows (from 90.228 MiB) in 1.138 sec. (1227772.866 rows/sec., 79.288 MiB/sec.)"] [thread_id=440]
[2022/03/28 13:45:00.019 +08:00] [TRACE] [<unknown>] ["MPPTask:<query:432130306537684995,task:1> ParallelAggregating Aggregated. 25474 to 19872 rows (from 1.645 MiB) in 1.138 sec. (22385.391 rows/sec., 1.445 MiB/sec.)"] [thread_id=440]
[2022/03/28 13:45:00.019 +08:00] [TRACE] [<unknown>] ["MPPTask:<query:432130306537684995,task:1> ParallelAggregating Aggregated. 26486 to 20378 rows (from 1.710 MiB) in 1.138 sec. (23274.690 rows/sec., 1.503 MiB/sec.)"] [thread_id=440]
[2022/03/28 13:45:00.019 +08:00] [TRACE] [<unknown>] ["MPPTask:<query:432130306537684995,task:1> ParallelAggregating Aggregated. 0 to 0 rows (from 0.000 MiB) in 1.138 sec. (0.000 rows/sec., 0.000 MiB/sec.)"] [thread_id=440]
[2022/03/28 13:45:00.019 +08:00] [TRACE] [<unknown>] ["MPPTask:<query:432130306537684995,task:1> ParallelAggregating Aggregated. 0 to 0 rows (from 0.000 MiB) in 1.138 sec. (0.000 rows/sec., 0.000 MiB/sec.)"] [thread_id=440]

Only 8 pipeline has data.

We need try to improve the performance of partition table in such extreme case
@windtalker windtalker added the type/enhancement The issue or PR belongs to an enhancement. label Mar 28, 2022
@windtalker windtalker self-assigned this Mar 28, 2022
@windtalker windtalker assigned wshwsh12 and bestwoody and unassigned windtalker and wshwsh12 Apr 19, 2022
@bestwoody bestwoody mentioned this issue May 24, 2022
Merged
12 tasks
ti-chi-bot pushed a commit that referenced this issue Jun 8, 2022
@bestwoody
Improve the performance of partition table in extreme case (#4988)
5b61ae7 
close #4474
@SeaRise SeaRise mentioned this issue May 16, 2023
Merged
12 tasks
@SeaRise SeaRise mentioned this issue Jul 2, 2023
Merged
12 tasks
ti-chi-bot bot pushed a commit that referenced this issue Jul 7, 2023
@SeaRise
Pipeline: optimize part scan (#7723)
e98f5d6 
ref #4474, ref #6518
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@bestwoody bestwoody

Labels
type/enhancement The issue or PR belongs to an enhancement.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Improve the performance of partition table in extreme case bestwoody/tiflash
3 participants
@windtalker @wshwsh12 @bestwoody