planner: disable `tidb_prefer_broadcast_join_by_exchange_data_size` by default; set scale factor to optimize estimating broadcast join; #42915

solotzg · 2023-04-11T05:54:00Z

What problem does this PR solve?

Issue Number: ref #40494 #42961

Problem

In ch-benchmark, tidb will choose the unexpected join method. The estimate row count of HashJoin_56 is far more bigger than the real one. It may cause the chosen join algorithm(Broadcast Join) is not the best one because the size of hash table when using broadcast join is bigger than the shuffle way, which means it may bring worse performance.

	id                                                              	task        	estRows         	operator info                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 	actRows  	execution info                                                                                                                                                                                                                                                                                                                                                                                       	memory 	disk
	TableReader_271                                                 	root        	5               	MppVersion: 1, data:ExchangeSender_270                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        	5        	time:1m6.2s, loops:2, cop_task: {num: 4, max: 0s, min: 0s, avg: 0s, p95: 0s, copr_cache_hit_ratio: 0.00}                                                                                                                                                                                                                                                                                             	2.16 KB	N/A
	└─ExchangeSender_270                                            	cop[tiflash]	5               	ExchangeType: PassThrough                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     	5        	tiflash_task:{proc max:1m6.2s, min:1m6.2s, avg: 1m6.2s, p80:1m6.2s, p95:1m6.2s, iters:2, tasks:2, threads:8}                                                                                                                                                                                                                                                                                         	N/A    	N/A
	  └─Projection_28                                               	cop[tiflash]	5               	tpcc.nation.n_name, Column#75                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 	5        	tiflash_task:{proc max:1m6.2s, min:1m6.2s, avg: 1m6.2s, p80:1m6.2s, p95:1m6.2s, iters:2, tasks:2, threads:8}                                                                                                                                                                                                                                                                                         	N/A    	N/A
	    └─Projection_266                                            	cop[tiflash]	5               	Column#75, tpcc.nation.n_name                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 	5        	tiflash_task:{proc max:1m6.2s, min:1m6.2s, avg: 1m6.2s, p80:1m6.2s, p95:1m6.2s, iters:2, tasks:2, threads:8}                                                                                                                                                                                                                                                                                         	N/A    	N/A
	      └─HashAgg_267                                             	cop[tiflash]	5               	group by:tpcc.nation.n_name, funcs:sum(Column#142)->Column#75, funcs:firstrow(tpcc.nation.n_name)->tpcc.nation.n_name, stream_count: 16                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       	5        	tiflash_task:{proc max:1m6.2s, min:1m6.2s, avg: 1m6.2s, p80:1m6.2s, p95:1m6.2s, iters:2, tasks:2, threads:8}                                                                                                                                                                                                                                                                                         	N/A    	N/A
	        └─ExchangeReceiver_269                                  	cop[tiflash]	5               	stream_count: 16                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              	10       	tiflash_task:{proc max:1m6.2s, min:1m6.2s, avg: 1m6.2s, p80:1m6.2s, p95:1m6.2s, iters:2, tasks:2, threads:8}                                                                                                                                                                                                                                                                                         	N/A    	N/A
	          └─ExchangeSender_268                                  	cop[tiflash]	5               	ExchangeType: HashPartition, Compression: FAST, Hash Cols: [name: tpcc.nation.n_name, collate: N/A], stream_count: 16                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         	10       	tiflash_task:{proc max:1m13.6s, min:0s, avg: 36.8s, p80:1m13.6s, p95:1m13.6s, iters:2, tasks:2, threads:8}                                                                                                                                                                                                                                                                                           	N/A    	N/A
	            └─HashAgg_264                                       	cop[tiflash]	5               	group by:tpcc.nation.n_name, funcs:sum(tpcc.order_line.ol_amount)->Column#142                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 	10       	tiflash_task:{proc max:1m13.6s, min:0s, avg: 36.8s, p80:1m13.6s, p95:1m13.6s, iters:2, tasks:2, threads:8}                                                                                                                                                                                                                                                                                           	N/A    	N/A
	              └─Projection_253                                  	cop[tiflash]	2307121986.87   	tpcc.order_line.ol_amount, tpcc.nation.n_name                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 	2484662  	tiflash_task:{proc max:1m13.4s, min:0s, avg: 36.7s, p80:1m13.4s, p95:1m13.4s, iters:14870, tasks:2, threads:11}                                                                                                                                                                                                                                                                                      	N/A    	N/A
	                └─HashJoin_243                                  	cop[tiflash]	2307121986.87   	inner join, equal:[eq(tpcc.nation.n_nationkey, tpcc.supplier.s_nationkey) eq(tpcc.nation.n_nationkey, Column#94)]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             	2484662  	tiflash_task:{proc max:1m13.4s, min:0s, avg: 36.7s, p80:1m13.4s, p95:1m13.4s, iters:14870, tasks:2, threads:11}                                                                                                                                                                                                                                                                                      	N/A    	N/A
	                  ├─ExchangeReceiver_47(Build)                  	cop[tiflash]	5               	                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              	10       	tiflash_task:{proc max:60.6ms, min:0s, avg: 30.3ms, p80:60.6ms, p95:60.6ms, iters:2, tasks:2, threads:8}                                                                                                                                                                                                                                                                                             	N/A    	N/A
	                  │ └─ExchangeSender_46                         	cop[tiflash]	5               	ExchangeType: Broadcast, Compression: FAST                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    	5        	tiflash_task:{proc max:29.2ms, min:0s, avg: 14.6ms, p80:29.2ms, p95:29.2ms, iters:1, tasks:2, threads:4}                                                                                                                                                                                                                                                                                             	N/A    	N/A
	                  │   └─HashJoin_40                             	cop[tiflash]	5               	inner join, equal:[eq(tpcc.region.r_regionkey, tpcc.nation.n_regionkey)]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      	5        	tiflash_task:{proc max:29.2ms, min:0s, avg: 14.6ms, p80:29.2ms, p95:29.2ms, iters:1, tasks:2, threads:4}                                                                                                                                                                                                                                                                                             	N/A    	N/A
	                  │     ├─ExchangeReceiver_44(Build)            	cop[tiflash]	1               	                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              	1        	tiflash_task:{proc max:29.2ms, min:0s, avg: 14.6ms, p80:29.2ms, p95:29.2ms, iters:1, tasks:2, threads:4}                                                                                                                                                                                                                                                                                             	N/A    	N/A
	                  │     │ └─ExchangeSender_43                   	cop[tiflash]	1               	ExchangeType: Broadcast, Compression: FAST                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    	1        	tiflash_task:{proc max:13.9ms, min:0s, avg: 6.93ms, p80:13.9ms, p95:13.9ms, iters:1, tasks:2, threads:1}                                                                                                                                                                                                                                                                                             	N/A    	N/A
	                  │     │   └─Selection_42                      	cop[tiflash]	1               	eq(tpcc.region.r_name, "EUROPE")                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              	1        	tiflash_task:{proc max:13.9ms, min:0s, avg: 6.93ms, p80:13.9ms, p95:13.9ms, iters:1, tasks:2, threads:1}                                                                                                                                                                                                                                                                                             	N/A    	N/A
	                  │     │     └─TableFullScan_41                	cop[tiflash]	5               	table:region, keep order:false                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                	5        	tiflash_task:{proc max:13.9ms, min:0s, avg: 6.93ms, p80:13.9ms, p95:13.9ms, iters:1, tasks:2, threads:1}, tiflash_scan:{dtfile:{total_scanned_packs:1, total_skipped_packs:0, total_scanned_rows:5, total_skipped_rows:0, total_rs_index_load_time: 0ms, total_read_time: 0ms}, total_create_snapshot_time: 0ms, total_local_region_num: 0, total_remote_region_num: 0}                              	N/A    	N/A
	                  │     └─TableFullScan_45(Probe)               	cop[tiflash]	25              	table:nation, keep order:false                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                	25       	tiflash_task:{proc max:29.2ms, min:0s, avg: 14.6ms, p80:29.2ms, p95:29.2ms, iters:1, tasks:2, threads:1}, tiflash_scan:{dtfile:{total_scanned_packs:1, total_skipped_packs:0, total_scanned_rows:25, total_skipped_rows:0, total_rs_index_load_time: 0ms, total_read_time: 0ms}, total_create_snapshot_time: 0ms, total_local_region_num: 0, total_remote_region_num: 0}                             	N/A    	N/A
	                  └─Projection_48(Probe)                        	cop[tiflash]	311922903022.71 	tpcc.order_line.ol_amount, tpcc.supplier.s_nationkey, minus(ascii(substr(tpcc.customer.c_state, 1, 1)), 65)->Column#94                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        	12373201 	tiflash_task:{proc max:1m13.3s, min:0s, avg: 36.6s, p80:1m13.3s, p95:1m13.3s, iters:14870, tasks:2, threads:11}                                                                                                                                                                                                                                                                                      	N/A    	N/A
	                    └─Projection_49                             	cop[tiflash]	311922903022.71 	tpcc.customer.c_state, tpcc.order_line.ol_amount, tpcc.supplier.s_nationkey                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   	12373201 	tiflash_task:{proc max:1m13.2s, min:0s, avg: 36.6s, p80:1m13.2s, p95:1m13.2s, iters:14870, tasks:2, threads:11}                                                                                                                                                                                                                                                                                      	N/A    	N/A
	                      └─HashJoin_50                             	cop[tiflash]	311922903022.71 	inner join, equal:[eq(tpcc.supplier.s_suppkey, Column#76) eq(tpcc.supplier.s_suppkey, Column#77) eq(tpcc.supplier.s_suppkey, Column#78) eq(tpcc.supplier.s_suppkey, Column#79) eq(tpcc.supplier.s_suppkey, Column#80) eq(tpcc.supplier.s_suppkey, Column#81) eq(tpcc.supplier.s_suppkey, Column#82) eq(tpcc.supplier.s_nationkey, Column#83) eq(tpcc.supplier.s_suppkey, Column#84)]                                                                                                                                                                                                                                                                                                                          	12373201 	tiflash_task:{proc max:1m13.2s, min:0s, avg: 36.6s, p80:1m13.2s, p95:1m13.2s, iters:14870, tasks:2, threads:11}                                                                                                                                                                                                                                                                                      	N/A    	N/A
	                        ├─ExchangeReceiver_53(Build)            	cop[tiflash]	10000           	                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              	20000    	tiflash_task:{proc max:60.6ms, min:0s, avg: 30.3ms, p80:60.6ms, p95:60.6ms, iters:2, tasks:2, threads:8}                                                                                                                                                                                                                                                                                             	N/A    	N/A
	                        │ └─ExchangeSender_52                   	cop[tiflash]	10000           	ExchangeType: Broadcast, Compression: FAST                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    	10000    	tiflash_task:{proc max:30.3ms, min:0s, avg: 15.2ms, p80:30.3ms, p95:30.3ms, iters:1, tasks:2, threads:1}                                                                                                                                                                                                                                                                                             	N/A    	N/A
	                        │   └─TableFullScan_51                  	cop[tiflash]	10000           	table:supplier, keep order:false                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              	10000    	tiflash_task:{proc max:30.3ms, min:0s, avg: 15.2ms, p80:30.3ms, p95:30.3ms, iters:1, tasks:2, threads:1}, tiflash_scan:{dtfile:{total_scanned_packs:2, total_skipped_packs:0, total_scanned_rows:10000, total_skipped_rows:0, total_rs_index_load_time: 0ms, total_read_time: 0ms}, total_create_snapshot_time: 0ms, total_local_region_num: 0, total_remote_region_num: 0}                          	N/A    	N/A
	                        └─Projection_54(Probe)                  	cop[tiflash]	3320661750495.92	tpcc.customer.c_state, tpcc.order_line.ol_amount, mod(mul(tpcc.order_line.ol_w_id, tpcc.order_line.ol_i_id), 10000)->Column#76, mod(mul(tpcc.orders.o_w_id, tpcc.order_line.ol_i_id), 10000)->Column#77, mod(mul(tpcc.customer.c_w_id, tpcc.order_line.ol_i_id), 10000)->Column#78, mod(mul(tpcc.stock.s_w_id, tpcc.order_line.ol_i_id), 10000)->Column#79, mod(mul(tpcc.order_line.ol_w_id, tpcc.stock.s_i_id), 10000)->Column#80, mod(mul(tpcc.orders.o_w_id, tpcc.stock.s_i_id), 10000)->Column#81, mod(mul(tpcc.customer.c_w_id, tpcc.stock.s_i_id), 10000)->Column#82, minus(ascii(substr(tpcc.customer.c_state, 1, 1)), 65)->Column#83, mod(mul(tpcc.stock.s_w_id, tpcc.stock.s_i_id), 10000)->Column#84	322010529	tiflash_task:{proc max:1m1.4s, min:0s, avg: 30.7s, p80:1m1.4s, p95:1m1.4s, iters:14870, tasks:2, threads:11}                                                                                                                                                                                                                                                                                         	N/A    	N/A
	                          └─HashJoin_55                         	cop[tiflash]	3320661750495.92	inner join, equal:[eq(tpcc.order_line.ol_w_id, tpcc.stock.s_w_id) eq(tpcc.order_line.ol_i_id, tpcc.stock.s_i_id)]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             	322010529	tiflash_task:{proc max:58.2s, min:0s, avg: 29.1s, p80:58.2s, p95:58.2s, iters:14870, tasks:2, threads:11}                                                                                                                                                                                                                                                                                            	N/A    	N/A
	                            ├─ExchangeReceiver_68(Build)        	cop[tiflash]	100000000       	                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              	200000000	tiflash_task:{proc max:2.87s, min:0s, avg: 1.44s, p80:2.87s, p95:2.87s, iters:4068, tasks:2, threads:8}                                                                                                                                                                                                                                                                                              	N/A    	N/A
	                            │ └─ExchangeSender_67               	cop[tiflash]	100000000       	ExchangeType: Broadcast, Compression: FAST                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    	100000000	tiflash_task:{proc max:13.7s, min:0s, avg: 6.87s, p80:13.7s, p95:13.7s, iters:7877, tasks:2, threads:16}                                                                                                                                                                                                                                                                                             	N/A    	N/A
	                            │   └─TableFullScan_66              	cop[tiflash]	100000000       	table:stock, keep order:false                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 	100000000	tiflash_task:{proc max:2.78s, min:0s, avg: 1.39s, p80:2.78s, p95:2.78s, iters:7877, tasks:2, threads:16}, tiflash_scan:{dtfile:{total_scanned_packs:77338, total_skipped_packs:24815, total_scanned_rows:617235697, total_skipped_rows:198069923, total_rs_index_load_time: 6ms, total_read_time: 6953ms}, total_create_snapshot_time: 0ms, total_local_region_num: 0, total_remote_region_num: 13}  	N/A    	N/A
	                            └─HashJoin_56(Probe)                	cop[tiflash]	3214290062856.99	inner join, equal:[eq(tpcc.orders.o_id, tpcc.order_line.ol_o_id) eq(tpcc.orders.o_w_id, tpcc.order_line.ol_w_id) eq(tpcc.orders.o_d_id, tpcc.order_line.ol_d_id)]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             	322010529	tiflash_task:{proc max:21.6s, min:0s, avg: 10.8s, p80:21.6s, p95:21.6s, iters:14870, tasks:2, threads:11}                                                                                                                                                                                                                                                                                            	N/A    	N/A
	                              ├─ExchangeReceiver_64(Build)      	cop[tiflash]	32143056        	                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              	64402506 	tiflash_task:{proc max:5.23s, min:0s, avg: 2.62s, p80:5.23s, p95:5.23s, iters:1343, tasks:2, threads:8}                                                                                                                                                                                                                                                                                              	N/A    	N/A
	                              │ └─ExchangeSender_63             	cop[tiflash]	32143056        	ExchangeType: Broadcast, Compression: FAST                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    	32201253 	tiflash_task:{proc max:7.08s, min:0s, avg: 3.54s, p80:7.08s, p95:7.08s, iters:1441, tasks:2, threads:13}                                                                                                                                                                                                                                                                                             	N/A    	N/A
	                              │   └─HashJoin_57                 	cop[tiflash]	32143056        	inner join, equal:[eq(tpcc.customer.c_id, tpcc.orders.o_c_id) eq(tpcc.customer.c_w_id, tpcc.orders.o_w_id) eq(tpcc.customer.c_d_id, tpcc.orders.o_d_id)]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      	32201253 	tiflash_task:{proc max:6.31s, min:0s, avg: 3.16s, p80:6.31s, p95:6.31s, iters:1441, tasks:2, threads:13}                                                                                                                                                                                                                                                                                             	N/A    	N/A
	                              │     ├─ExchangeReceiver_60(Build)	cop[tiflash]	30000000        	                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              	60000000 	tiflash_task:{proc max:1.21s, min:0s, avg: 605.2ms, p80:1.21s, p95:1.21s, iters:1308, tasks:2, threads:8}                                                                                                                                                                                                                                                                                            	N/A    	N/A
	                              │     │ └─ExchangeSender_59       	cop[tiflash]	30000000        	ExchangeType: Broadcast, Compression: FAST                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    	30000000 	tiflash_task:{proc max:3.24s, min:0s, avg: 1.62s, p80:3.24s, p95:3.24s, iters:2954, tasks:2, threads:16}                                                                                                                                                                                                                                                                                             	N/A    	N/A
	                              │     │   └─TableFullScan_58      	cop[tiflash]	30000000        	table:customer, keep order:false                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              	30000000 	tiflash_task:{proc max:1.22s, min:0s, avg: 610.3ms, p80:1.22s, p95:1.22s, iters:2954, tasks:2, threads:16}, tiflash_scan:{dtfile:{total_scanned_packs:21576, total_skipped_packs:7755, total_scanned_rows:172361588, total_skipped_rows:61820394, total_rs_index_load_time: 2ms, total_read_time: 2608ms}, total_create_snapshot_time: 0ms, total_local_region_num: 0, total_remote_region_num: 18}  	N/A    	N/A
	                              │     └─Selection_62(Probe)       	cop[tiflash]	32143056        	ge(tpcc.orders.o_entry_d, 2007-01-02 00:00:00.000000), not(isnull(tpcc.orders.o_c_id))                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        	32201253 	tiflash_task:{proc max:650.5ms, min:0s, avg: 325.3ms, p80:650.5ms, p95:650.5ms, iters:1441, tasks:2, threads:13}                                                                                                                                                                                                                                                                                     	N/A    	N/A
	                              │       └─TableFullScan_61        	cop[tiflash]	32143056        	table:orders, keep order:false                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                	32201253 	tiflash_task:{proc max:550.5ms, min:0s, avg: 275.3ms, p80:550.5ms, p95:550.5ms, iters:1441, tasks:2, threads:13}, tiflash_scan:{dtfile:{total_scanned_packs:11775, total_skipped_packs:282, total_scanned_rows:95837584, total_skipped_rows:2287305, total_rs_index_load_time: 0ms, total_read_time: 1346ms}, total_create_snapshot_time: 0ms, total_local_region_num: 0, total_remote_region_num: 5}	N/A    	N/A
	                              └─TableFullScan_65(Probe)         	cop[tiflash]	321429017       	table:order_line, keep order:false                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            	322010529	tiflash_task:{proc max:3.55s, min:0s, avg: 1.78s, p80:3.55s, p95:3.55s, iters:14870, tasks:2, threads:11}, tiflash_scan:{dtfile:{total_scanned_packs:123328, total_skipped_packs:6835, total_scanned_rows:1006583553, total_skipped_rows:55787408, total_rs_index_load_time: 8ms, total_read_time: 11812ms}, total_create_snapshot_time: 2ms, total_local_region_num: 0, total_remote_region_num: 3} 	N/A    	N/A

What is changed and how it works?

Add scale factor to indicate the factor used to estimate the approximate cost of broadcast join
The size of hash table when using broadcast join is bigger than the shuffle way, which means it may bring worse performance. Set a relatively conservative value by default.
Disable tidb_prefer_broadcast_join_by_exchange_data_size by default

Benchmark

ENV

TPCH-100
TiFlash x 3
TIFLASH REPLICA x 3
Original TiDB: 88174d2
set tidb_prefer_broadcast_join_by_exchange_data_size=ON

TPCH-100 Q9

original

MySQL [tpch100]> explain analyze select     nation,     o_year,     sum(amount) as sum_profit from     (         select             n_name as nation,             extract(year from o_orderdate) as o_year,             l_extendedprice * (1 - l_discount) - ps_supplycost * l_quantity as amount         from             part,             supplier,             lineitem,             partsupp,             orders,             nation         where             s_suppkey = l_suppkey             and ps_suppkey = l_suppkey             and ps_partkey = l_partkey             and p_partkey = l_partkey             and o_orderkey = l_orderkey             and s_nationkey = n_nationkey             and p_name like '%dim%'     ) as profit group by     nation,     o_year order by     nation,     o_year desc;
+----------------------------------------------------------------------+--------------+-----------+--------------+----------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+---------+
| id                                                                   | estRows      | actRows   | task         | access object  | execution info                                                                                                                                                                                                                                                                                                                                                                                   | operator info                                                                                                                                                                                                                                 | memory  | disk    |
+----------------------------------------------------------------------+--------------+-----------+--------------+----------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+---------+
| Sort_26                                                              | 2406.00      | 175       | root         |                | time:15.1s, loops:2                                                                                                                                                                                                                                                                                                                                                                              | tpch100.nation.n_name, Column#53:desc                                                                                                                                                                                                         | 16.1 KB | 0 Bytes |
| └─TableReader_201                                                    | 2406.00      | 175       | root         |                | time:15.1s, loops:2, cop_task: {num: 23, max: 0s, min: 0s, avg: 0s, p95: 0s, copr_cache_hit_ratio: 0.00}                                                                                                                                                                                                                                                                                         | MppVersion: 1, data:ExchangeSender_200                                                                                                                                                                                                        | 2.22 KB | N/A     |
|   └─ExchangeSender_200                                               | 2406.00      | 175       | mpp[tiflash] |                | tiflash_task:{proc max:15.1s, min:15.1s, avg: 15.1s, p80:15.1s, p95:15.1s, iters:20, tasks:3, threads:60}                                                                                                                                                                                                                                                                                        | ExchangeType: PassThrough                                                                                                                                                                                                                     | N/A     | N/A     |
|     └─Projection_29                                                  | 2406.00      | 175       | mpp[tiflash] |                | tiflash_task:{proc max:15.1s, min:15.1s, avg: 15.1s, p80:15.1s, p95:15.1s, iters:20, tasks:3, threads:60}                                                                                                                                                                                                                                                                                        | tpch100.nation.n_name, Column#53, Column#55                                                                                                                                                                                                   | N/A     | N/A     |
|       └─Projection_196                                               | 2406.00      | 175       | mpp[tiflash] |                | tiflash_task:{proc max:15.1s, min:15.1s, avg: 15.1s, p80:15.1s, p95:15.1s, iters:20, tasks:3, threads:60}                                                                                                                                                                                                                                                                                        | Column#55, tpch100.nation.n_name, Column#53                                                                                                                                                                                                   | N/A     | N/A     |
|         └─HashAgg_197                                                | 2406.00      | 175       | mpp[tiflash] |                | tiflash_task:{proc max:15.1s, min:15.1s, avg: 15.1s, p80:15.1s, p95:15.1s, iters:20, tasks:3, threads:60}                                                                                                                                                                                                                                                                                        | group by:Column#72, tpch100.nation.n_name, funcs:sum(Column#73)->Column#55, funcs:firstrow(tpch100.nation.n_name)->tpch100.nation.n_name, funcs:firstrow(Column#72)->Column#53, stream_count: 20                                              | N/A     | N/A     |
|           └─ExchangeReceiver_199                                     | 2406.00      | 525       | mpp[tiflash] |                | tiflash_task:{proc max:15.1s, min:15.1s, avg: 15.1s, p80:15.1s, p95:15.1s, iters:20, tasks:3, threads:60}                                                                                                                                                                                                                                                                                        | stream_count: 20                                                                                                                                                                                                                              | N/A     | N/A     |
|             └─ExchangeSender_198                                     | 2406.00      | 525       | mpp[tiflash] |                | tiflash_task:{proc max:21.2s, min:0s, avg: 7.05s, p80:21.2s, p95:21.2s, iters:3, tasks:3, threads:60}                                                                                                                                                                                                                                                                                            | ExchangeType: HashPartition, Compression: FAST, Hash Cols: [name: tpch100.nation.n_name, collate: utf8mb4_bin], stream_count: 20                                                                                                              | N/A     | N/A     |
|               └─HashAgg_194                                          | 2406.00      | 525       | mpp[tiflash] |                | tiflash_task:{proc max:21.2s, min:0s, avg: 7.05s, p80:21.2s, p95:21.2s, iters:3, tasks:3, threads:60}                                                                                                                                                                                                                                                                                            | group by:Column#77, Column#78, funcs:sum(Column#76)->Column#73                                                                                                                                                                                | N/A     | N/A     |
|                 └─Projection_204                                     | 471441799.77 | 32571554  | mpp[tiflash] |                | tiflash_task:{proc max:21.1s, min:0s, avg: 7.04s, p80:21.1s, p95:21.1s, iters:10155, tasks:3, threads:60}                                                                                                                                                                                                                                                                                        | minus(mul(tpch100.lineitem.l_extendedprice, minus(1, tpch100.lineitem.l_discount)), mul(tpch100.partsupp.ps_supplycost, tpch100.lineitem.l_quantity))->Column#76, tpch100.nation.n_name, extract(YEAR, tpch100.orders.o_orderdate)->Column#78 | N/A     | N/A     |
|                   └─Projection_183                                   | 471441799.77 | 32571554  | mpp[tiflash] |                | tiflash_task:{proc max:21.1s, min:0s, avg: 7.02s, p80:21.1s, p95:21.1s, iters:10155, tasks:3, threads:60}                                                                                                                                                                                                                                                                                        | tpch100.lineitem.l_quantity, tpch100.lineitem.l_extendedprice, tpch100.lineitem.l_discount, tpch100.partsupp.ps_supplycost, tpch100.orders.o_orderdate, tpch100.nation.n_name                                                                 | N/A     | N/A     |
|                     └─HashJoin_173                                   | 471441799.77 | 32571554  | mpp[tiflash] |                | tiflash_task:{proc max:21s, min:0s, avg: 7.02s, p80:21s, p95:21s, iters:10155, tasks:3, threads:60}                                                                                                                                                                                                                                                                                              | inner join, equal:[eq(tpch100.lineitem.l_orderkey, tpch100.orders.o_orderkey)]                                                                                                                                                                | N/A     | N/A     |
|                       ├─ExchangeReceiver_71(Build)                   | 150000000.00 | 450000000 | mpp[tiflash] |                | tiflash_task:{proc max:604.6ms, min:0s, avg: 201.5ms, p80:604.6ms, p95:604.6ms, iters:7062, tasks:3, threads:60}                                                                                                                                                                                                                                                                                 |                                                                                                                                                                                                                                               | N/A     | N/A     |
|                       │ └─ExchangeSender_70                          | 150000000.00 | 150000000 | mpp[tiflash] |                | tiflash_task:{proc max:5.68s, min:0s, avg: 1.89s, p80:5.68s, p95:5.68s, iters:2354, tasks:3, threads:60}                                                                                                                                                                                                                                                                                         | ExchangeType: Broadcast, Compression: FAST                                                                                                                                                                                                    | N/A     | N/A     |
|                       │   └─TableFullScan_69                         | 150000000.00 | 150000000 | mpp[tiflash] | table:orders   | tiflash_task:{proc max:120.1ms, min:0s, avg: 40ms, p80:120.1ms, p95:120.1ms, iters:2354, tasks:3, threads:60}, tiflash_scan:{dtfile:{total_scanned_packs:18456, total_skipped_packs:0, total_scanned_rows:150000000, total_skipped_rows:0, total_rs_index_load_time: 197ms, total_read_time: 8503ms}, total_create_snapshot_time: 1ms, total_local_region_num: 214, total_remote_region_num: 0}  | keep order:false                                                                                                                                                                                                                              | N/A     | N/A     |
|                       └─HashJoin_51(Probe)                           | 474208396.26 | 32571554  | mpp[tiflash] |                | tiflash_task:{proc max:14.8s, min:0s, avg: 4.94s, p80:14.8s, p95:14.8s, iters:10155, tasks:3, threads:60}                                                                                                                                                                                                                                                                                        | inner join, equal:[eq(tpch100.lineitem.l_suppkey, tpch100.partsupp.ps_suppkey) eq(tpch100.lineitem.l_partkey, tpch100.partsupp.ps_partkey)]                                                                                                   | N/A     | N/A     |
|                         ├─ExchangeReceiver_68(Build)                 | 80000000.00  | 240000000 | mpp[tiflash] |                | tiflash_task:{proc max:258.6ms, min:0s, avg: 86.2ms, p80:258.6ms, p95:258.6ms, iters:3879, tasks:3, threads:60}                                                                                                                                                                                                                                                                                  |                                                                                                                                                                                                                                               | N/A     | N/A     |
|                         │ └─ExchangeSender_67                        | 80000000.00  | 80000000  | mpp[tiflash] |                | tiflash_task:{proc max:5.05s, min:0s, avg: 1.68s, p80:5.05s, p95:5.05s, iters:1293, tasks:3, threads:60}                                                                                                                                                                                                                                                                                         | ExchangeType: Broadcast, Compression: FAST                                                                                                                                                                                                    | N/A     | N/A     |
|                         │   └─TableFullScan_66                       | 80000000.00  | 80000000  | mpp[tiflash] | table:partsupp | tiflash_task:{proc max:66.6ms, min:0s, avg: 22.2ms, p80:66.6ms, p95:66.6ms, iters:1293, tasks:3, threads:60}, tiflash_scan:{dtfile:{total_scanned_packs:9844, total_skipped_packs:0, total_scanned_rows:80000000, total_skipped_rows:0, total_rs_index_load_time: 30ms, total_read_time: 6493ms}, total_create_snapshot_time: 0ms, total_local_region_num: 144, total_remote_region_num: 0}      | keep order:false                                                                                                                                                                                                                              | N/A     | N/A     |
|                         └─HashJoin_52(Probe)                         | 474208396.26 | 32571554  | mpp[tiflash] |                | tiflash_task:{proc max:8.68s, min:0s, avg: 2.89s, p80:8.68s, p95:8.68s, iters:10155, tasks:3, threads:60}                                                                                                                                                                                                                                                                                        | inner join, equal:[eq(tpch100.lineitem.l_partkey, tpch100.part.p_partkey)]                                                                                                                                                                    | N/A     | N/A     |
|                           ├─ExchangeReceiver_65(Build)               | 16000000.00  | 3256917   | mpp[tiflash] |                | tiflash_task:{proc max:209.6ms, min:0s, avg: 69.9ms, p80:209.6ms, p95:209.6ms, iters:300, tasks:3, threads:60}                                                                                                                                                                                                                                                                                   |                                                                                                                                                                                                                                               | N/A     | N/A     |
|                           │ └─ExchangeSender_64                      | 16000000.00  | 1085639   | mpp[tiflash] |                | tiflash_task:{proc max:260.6ms, min:0s, avg: 86.9ms, p80:260.6ms, p95:260.6ms, iters:313, tasks:3, threads:31}                                                                                                                                                                                                                                                                                   | ExchangeType: Broadcast, Compression: FAST                                                                                                                                                                                                    | N/A     | N/A     |
|                           │   └─Selection_63                         | 16000000.00  | 1085639   | mpp[tiflash] |                | tiflash_task:{proc max:249.6ms, min:0s, avg: 83.2ms, p80:249.6ms, p95:249.6ms, iters:313, tasks:3, threads:31}                                                                                                                                                                                                                                                                                   | like(tpch100.part.p_name, "%dim%", 92)                                                                                                                                                                                                        | N/A     | N/A     |
|                           │     └─TableFullScan_62                   | 20000000.00  | 20000000  | mpp[tiflash] | table:part     | tiflash_task:{proc max:209.6ms, min:0s, avg: 69.9ms, p80:209.6ms, p95:209.6ms, iters:313, tasks:3, threads:31}, tiflash_scan:{dtfile:{total_scanned_packs:2465, total_skipped_packs:0, total_scanned_rows:20000000, total_skipped_rows:0, total_rs_index_load_time: 8ms, total_read_time: 4819ms}, total_create_snapshot_time: 0ms, total_local_region_num: 31, total_remote_region_num: 0}      | keep order:false                                                                                                                                                                                                                              | N/A     | N/A     |
|                           └─HashJoin_53(Probe)                       | 595696793.71 | 600037902 | mpp[tiflash] |                | tiflash_task:{proc max:6.69s, min:0s, avg: 2.23s, p80:6.69s, p95:6.69s, iters:10155, tasks:3, threads:60}                                                                                                                                                                                                                                                                                        | inner join, equal:[eq(tpch100.supplier.s_suppkey, tpch100.lineitem.l_suppkey)]                                                                                                                                                                | N/A     | N/A     |
|                             ├─ExchangeReceiver_60(Build)             | 991431.00    | 3000000   | mpp[tiflash] |                | tiflash_task:{proc max:229.6ms, min:0s, avg: 76.5ms, p80:229.6ms, p95:229.6ms, iters:48, tasks:3, threads:60}                                                                                                                                                                                                                                                                                    |                                                                                                                                                                                                                                               | N/A     | N/A     |
|                             │ └─ExchangeSender_59                    | 991431.00    | 1000000   | mpp[tiflash] |                | tiflash_task:{proc max:220.3ms, min:0s, avg: 73.4ms, p80:220.3ms, p95:220.3ms, iters:16, tasks:3, threads:40}                                                                                                                                                                                                                                                                                    | ExchangeType: Broadcast, Compression: FAST                                                                                                                                                                                                    | N/A     | N/A     |
|                             │   └─HashJoin_54                        | 991431.00    | 1000000   | mpp[tiflash] |                | tiflash_task:{proc max:152.3ms, min:0s, avg: 50.8ms, p80:152.3ms, p95:152.3ms, iters:16, tasks:3, threads:40}                                                                                                                                                                                                                                                                                    | inner join, equal:[eq(tpch100.nation.n_nationkey, tpch100.supplier.s_nationkey)]                                                                                                                                                              | N/A     | N/A     |
|                             │     ├─ExchangeReceiver_57(Build)       | 25.00        | 50        | mpp[tiflash] |                | tiflash_task:{proc max:49.3ms, min:0s, avg: 16.4ms, p80:49.3ms, p95:49.3ms, iters:2, tasks:3, threads:40}                                                                                                                                                                                                                                                                                        |                                                                                                                                                                                                                                               | N/A     | N/A     |
|                             │     │ └─ExchangeSender_56              | 25.00        | 25        | mpp[tiflash] |                | tiflash_task:{proc max:7.35ms, min:0s, avg: 2.45ms, p80:7.35ms, p95:7.35ms, iters:1, tasks:3, threads:1}                                                                                                                                                                                                                                                                                         | ExchangeType: Broadcast, Compression: FAST                                                                                                                                                                                                    | N/A     | N/A     |
|                             │     │   └─TableFullScan_55             | 25.00        | 25        | mpp[tiflash] | table:nation   | tiflash_task:{proc max:7.35ms, min:0s, avg: 2.45ms, p80:7.35ms, p95:7.35ms, iters:1, tasks:3, threads:1}, tiflash_scan:{dtfile:{total_scanned_packs:1, total_skipped_packs:0, total_scanned_rows:25, total_skipped_rows:0, total_rs_index_load_time: 0ms, total_read_time: 0ms}, total_create_snapshot_time: 0ms, total_local_region_num: 1, total_remote_region_num: 0}                         | keep order:false                                                                                                                                                                                                                              | N/A     | N/A     |
|                             │     └─TableFullScan_58(Probe)          | 1000000.00   | 1000000   | mpp[tiflash] | table:supplier | tiflash_task:{proc max:108.3ms, min:0s, avg: 36.1ms, p80:108.3ms, p95:108.3ms, iters:16, tasks:3, threads:2}, tiflash_scan:{dtfile:{total_scanned_packs:123, total_skipped_packs:0, total_scanned_rows:1000000, total_skipped_rows:0, total_rs_index_load_time: 0ms, total_read_time: 66ms}, total_create_snapshot_time: 0ms, total_local_region_num: 2, total_remote_region_num: 0}             | keep order:false                                                                                                                                                                                                                              | N/A     | N/A     |
|                             └─TableFullScan_61(Probe)                | 600037902.00 | 600037902 | mpp[tiflash] | table:lineitem | tiflash_task:{proc max:92.6ms, min:0s, avg: 30.9ms, p80:92.6ms, p95:92.6ms, iters:10155, tasks:3, threads:60}, tiflash_scan:{dtfile:{total_scanned_packs:73859, total_skipped_packs:0, total_scanned_rows:600037902, total_skipped_rows:0, total_rs_index_load_time: 46ms, total_read_time: 48134ms}, total_create_snapshot_time: 4ms, total_local_region_num: 1129, total_remote_region_num: 0} | keep order:false                                                                                                                                                                                                                              | N/A     | N/A     |
+----------------------------------------------------------------------+--------------+-----------+--------------+----------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+---------+
33 rows in set (15.17 sec)

optimized

MySQL [tpch100]> explain analyze select     nation,     o_year,     sum(amount) as sum_profit from     (         select             n_name as nation,             extract(year from o_orderdate) as o_year,             l_extendedprice * (1 - l_discount) - ps_supplycost * l_quantity as amount         from             part,             supplier,             lineitem,             partsupp,             orders,             nation         where             s_suppkey = l_suppkey             and ps_suppkey = l_suppkey             and ps_partkey = l_partkey             and p_partkey = l_partkey             and o_orderkey = l_orderkey             and s_nationkey = n_nationkey             and p_name like '%dim%'     ) as profit group by     nation,     o_year order by     nation,     o_year desc;
+------------------------------------------------------------------------------+--------------+-----------+--------------+----------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+---------+
| id                                                                           | estRows      | actRows   | task         | access object  | execution info                                                                                                                                                                                                                                                                                                                                                                                        | operator info                                                                                                                                                                                                                                 | memory  | disk    |
+------------------------------------------------------------------------------+--------------+-----------+--------------+----------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+---------+
| Sort_26                                                                      | 2406.00      | 175       | root         |                | time:10.3s, loops:2                                                                                                                                                                                                                                                                                                                                                                                   | tpch100.nation.n_name, Column#53:desc                                                                                                                                                                                                         | 16.1 KB | 0 Bytes |
| └─TableReader_205                                                            | 2406.00      | 175       | root         |                | time:10.3s, loops:2, cop_task: {num: 23, max: 0s, min: 0s, avg: 0s, p95: 0s, copr_cache_hit_ratio: 0.00}                                                                                                                                                                                                                                                                                              | MppVersion: 1, data:ExchangeSender_204                                                                                                                                                                                                        | 2.01 KB | N/A     |
|   └─ExchangeSender_204                                                       | 2406.00      | 175       | mpp[tiflash] |                | tiflash_task:{proc max:10.3s, min:10.3s, avg: 10.3s, p80:10.3s, p95:10.3s, iters:20, tasks:3, threads:60}                                                                                                                                                                                                                                                                                             | ExchangeType: PassThrough                                                                                                                                                                                                                     | N/A     | N/A     |
|     └─Projection_29                                                          | 2406.00      | 175       | mpp[tiflash] |                | tiflash_task:{proc max:10.3s, min:10.3s, avg: 10.3s, p80:10.3s, p95:10.3s, iters:20, tasks:3, threads:60}                                                                                                                                                                                                                                                                                             | tpch100.nation.n_name, Column#53, Column#55                                                                                                                                                                                                   | N/A     | N/A     |
|       └─Projection_200                                                       | 2406.00      | 175       | mpp[tiflash] |                | tiflash_task:{proc max:10.3s, min:10.3s, avg: 10.3s, p80:10.3s, p95:10.3s, iters:20, tasks:3, threads:60}                                                                                                                                                                                                                                                                                             | Column#55, tpch100.nation.n_name, Column#53                                                                                                                                                                                                   | N/A     | N/A     |
|         └─HashAgg_201                                                        | 2406.00      | 175       | mpp[tiflash] |                | tiflash_task:{proc max:10.3s, min:10.3s, avg: 10.3s, p80:10.3s, p95:10.3s, iters:20, tasks:3, threads:60}                                                                                                                                                                                                                                                                                             | group by:Column#72, tpch100.nation.n_name, funcs:sum(Column#73)->Column#55, funcs:firstrow(tpch100.nation.n_name)->tpch100.nation.n_name, funcs:firstrow(Column#72)->Column#53, stream_count: 20                                              | N/A     | N/A     |
|           └─ExchangeReceiver_203                                             | 2406.00      | 525       | mpp[tiflash] |                | tiflash_task:{proc max:10.3s, min:10.3s, avg: 10.3s, p80:10.3s, p95:10.3s, iters:20, tasks:3, threads:60}                                                                                                                                                                                                                                                                                             | stream_count: 20                                                                                                                                                                                                                              | N/A     | N/A     |
|             └─ExchangeSender_202                                             | 2406.00      | 525       | mpp[tiflash] |                | tiflash_task:{proc max:10.3s, min:0s, avg: 3.43s, p80:10.3s, p95:10.3s, iters:3, tasks:3, threads:60}                                                                                                                                                                                                                                                                                                 | ExchangeType: HashPartition, Compression: FAST, Hash Cols: [name: tpch100.nation.n_name, collate: utf8mb4_bin], stream_count: 20                                                                                                              | N/A     | N/A     |
|               └─HashAgg_198                                                  | 2406.00      | 525       | mpp[tiflash] |                | tiflash_task:{proc max:10.3s, min:0s, avg: 3.43s, p80:10.3s, p95:10.3s, iters:3, tasks:3, threads:60}                                                                                                                                                                                                                                                                                                 | group by:Column#77, Column#78, funcs:sum(Column#76)->Column#73                                                                                                                                                                                | N/A     | N/A     |
|                 └─Projection_208                                             | 471441799.77 | 32571554  | mpp[tiflash] |                | tiflash_task:{proc max:10.3s, min:0s, avg: 3.42s, p80:10.3s, p95:10.3s, iters:3666, tasks:3, threads:60}                                                                                                                                                                                                                                                                                              | minus(mul(tpch100.lineitem.l_extendedprice, minus(1, tpch100.lineitem.l_discount)), mul(tpch100.partsupp.ps_supplycost, tpch100.lineitem.l_quantity))->Column#76, tpch100.nation.n_name, extract(YEAR, tpch100.orders.o_orderdate)->Column#78 | N/A     | N/A     |
|                   └─Projection_187                                           | 471441799.77 | 32571554  | mpp[tiflash] |                | tiflash_task:{proc max:10.2s, min:0s, avg: 3.4s, p80:10.2s, p95:10.2s, iters:3666, tasks:3, threads:60}                                                                                                                                                                                                                                                                                               | tpch100.lineitem.l_quantity, tpch100.lineitem.l_extendedprice, tpch100.lineitem.l_discount, tpch100.partsupp.ps_supplycost, tpch100.orders.o_orderdate, tpch100.nation.n_name                                                                 | N/A     | N/A     |
|                     └─HashJoin_177                                           | 471441799.77 | 32571554  | mpp[tiflash] |                | tiflash_task:{proc max:10.2s, min:0s, avg: 3.39s, p80:10.2s, p95:10.2s, iters:3666, tasks:3, threads:60}                                                                                                                                                                                                                                                                                              | inner join, equal:[eq(tpch100.lineitem.l_orderkey, tpch100.orders.o_orderkey)], stream_count: 20                                                                                                                                              | N/A     | N/A     |
|                       ├─ExchangeReceiver_75(Build)                           | 150000000.00 | 150000000 | mpp[tiflash] |                | tiflash_task:{proc max:1.84s, min:0s, avg: 613.5ms, p80:1.84s, p95:1.84s, iters:15540, tasks:3, threads:60}                                                                                                                                                                                                                                                                                           | stream_count: 20                                                                                                                                                                                                                              | N/A     | N/A     |
|                       │ └─ExchangeSender_74                                  | 150000000.00 | 150000000 | mpp[tiflash] |                | tiflash_task:{proc max:2.09s, min:0s, avg: 695.7ms, p80:2.09s, p95:2.09s, iters:2354, tasks:3, threads:60}                                                                                                                                                                                                                                                                                            | ExchangeType: HashPartition, Compression: FAST, Hash Cols: [name: tpch100.orders.o_orderkey, collate: binary], stream_count: 20                                                                                                               | N/A     | N/A     |
|                       │   └─TableFullScan_73                                 | 150000000.00 | 150000000 | mpp[tiflash] | table:orders   | tiflash_task:{proc max:108ms, min:0s, avg: 36ms, p80:108ms, p95:108ms, iters:2354, tasks:3, threads:60}, tiflash_scan:{dtfile:{total_scanned_packs:18456, total_skipped_packs:0, total_scanned_rows:150000000, total_skipped_rows:0, total_rs_index_load_time: 129ms, total_read_time: 7697ms}, total_create_snapshot_time: 0ms, total_local_region_num: 214, total_remote_region_num: 0}             | keep order:false                                                                                                                                                                                                                              | N/A     | N/A     |
|                       └─ExchangeReceiver_72(Probe)                           | 474208396.26 | 32571554  | mpp[tiflash] |                | tiflash_task:{proc max:6.27s, min:0s, avg: 2.09s, p80:6.27s, p95:6.27s, iters:3666, tasks:3, threads:60}                                                                                                                                                                                                                                                                                              |                                                                                                                                                                                                                                               | N/A     | N/A     |
|                         └─ExchangeSender_71                                  | 474208396.26 | 32571554  | mpp[tiflash] |                | tiflash_task:{proc max:10.3s, min:0s, avg: 3.42s, p80:10.3s, p95:10.3s, iters:3677, tasks:3, threads:60}                                                                                                                                                                                                                                                                                              | ExchangeType: HashPartition, Compression: FAST, Hash Cols: [name: tpch100.lineitem.l_orderkey, collate: binary]                                                                                                                               | N/A     | N/A     |
|                           └─HashJoin_51                                      | 474208396.26 | 32571554  | mpp[tiflash] |                | tiflash_task:{proc max:9.8s, min:0s, avg: 3.27s, p80:9.8s, p95:9.8s, iters:3677, tasks:3, threads:60}                                                                                                                                                                                                                                                                                                 | inner join, equal:[eq(tpch100.lineitem.l_suppkey, tpch100.partsupp.ps_suppkey) eq(tpch100.lineitem.l_partkey, tpch100.partsupp.ps_partkey)], stream_count: 20                                                                                 | N/A     | N/A     |
|                             ├─ExchangeReceiver_70(Build)                     | 80000000.00  | 80000000  | mpp[tiflash] |                | tiflash_task:{proc max:1.21s, min:0s, avg: 403.9ms, p80:1.21s, p95:1.21s, iters:8100, tasks:3, threads:60}                                                                                                                                                                                                                                                                                            | stream_count: 20                                                                                                                                                                                                                              | N/A     | N/A     |
|                             │ └─ExchangeSender_69                            | 80000000.00  | 80000000  | mpp[tiflash] |                | tiflash_task:{proc max:713ms, min:0s, avg: 237.7ms, p80:713ms, p95:713ms, iters:1295, tasks:3, threads:60}                                                                                                                                                                                                                                                                                            | ExchangeType: HashPartition, Compression: FAST, Hash Cols: [name: tpch100.partsupp.ps_suppkey, collate: binary], [name: tpch100.partsupp.ps_partkey, collate: binary], stream_count: 20                                                       | N/A     | N/A     |
|                             │   └─TableFullScan_68                           | 80000000.00  | 80000000  | mpp[tiflash] | table:partsupp | tiflash_task:{proc max:160.4ms, min:0s, avg: 53.5ms, p80:160.4ms, p95:160.4ms, iters:1295, tasks:3, threads:60}, tiflash_scan:{dtfile:{total_scanned_packs:9847, total_skipped_packs:348, total_scanned_rows:80024576, total_skipped_rows:2850387, total_rs_index_load_time: 4ms, total_read_time: 6361ms}, total_create_snapshot_time: 0ms, total_local_region_num: 144, total_remote_region_num: 0} | keep order:false                                                                                                                                                                                                                              | N/A     | N/A     |
|                             └─ExchangeReceiver_67(Probe)                     | 474208396.26 | 32571554  | mpp[tiflash] |                | tiflash_task:{proc max:7.37s, min:0s, avg: 2.46s, p80:7.37s, p95:7.37s, iters:3677, tasks:3, threads:60}                                                                                                                                                                                                                                                                                              |                                                                                                                                                                                                                                               | N/A     | N/A     |
|                               └─ExchangeSender_66                            | 474208396.26 | 32571554  | mpp[tiflash] |                | tiflash_task:{proc max:10.6s, min:0s, avg: 3.53s, p80:10.6s, p95:10.6s, iters:10155, tasks:3, threads:60}                                                                                                                                                                                                                                                                                             | ExchangeType: HashPartition, Compression: FAST, Hash Cols: [name: tpch100.lineitem.l_suppkey, collate: binary], [name: tpch100.lineitem.l_partkey, collate: binary]                                                                           | N/A     | N/A     |
|                                 └─HashJoin_52                                | 474208396.26 | 32571554  | mpp[tiflash] |                | tiflash_task:{proc max:10.3s, min:0s, avg: 3.42s, p80:10.3s, p95:10.3s, iters:10155, tasks:3, threads:60}                                                                                                                                                                                                                                                                                             | inner join, equal:[eq(tpch100.lineitem.l_partkey, tpch100.part.p_partkey)]                                                                                                                                                                    | N/A     | N/A     |
|                                   ├─ExchangeReceiver_65(Build)               | 16000000.00  | 3256917   | mpp[tiflash] |                | tiflash_task:{proc max:343.1ms, min:0s, avg: 114.4ms, p80:343.1ms, p95:343.1ms, iters:309, tasks:3, threads:60}                                                                                                                                                                                                                                                                                       |                                                                                                                                                                                                                                               | N/A     | N/A     |
|                                   │ └─ExchangeSender_64                      | 16000000.00  | 1085639   | mpp[tiflash] |                | tiflash_task:{proc max:423.2ms, min:0s, avg: 141.1ms, p80:423.2ms, p95:423.2ms, iters:313, tasks:3, threads:31}                                                                                                                                                                                                                                                                                       | ExchangeType: Broadcast, Compression: FAST                                                                                                                                                                                                    | N/A     | N/A     |
|                                   │   └─Selection_63                         | 16000000.00  | 1085639   | mpp[tiflash] |                | tiflash_task:{proc max:414.2ms, min:0s, avg: 138.1ms, p80:414.2ms, p95:414.2ms, iters:313, tasks:3, threads:31}                                                                                                                                                                                                                                                                                       | like(tpch100.part.p_name, "%dim%", 92)                                                                                                                                                                                                        | N/A     | N/A     |
|                                   │     └─TableFullScan_62                   | 20000000.00  | 20000000  | mpp[tiflash] | table:part     | tiflash_task:{proc max:372.2ms, min:0s, avg: 124.1ms, p80:372.2ms, p95:372.2ms, iters:313, tasks:3, threads:31}, tiflash_scan:{dtfile:{total_scanned_packs:2465, total_skipped_packs:0, total_scanned_rows:20000000, total_skipped_rows:0, total_rs_index_load_time: 0ms, total_read_time: 4553ms}, total_create_snapshot_time: 0ms, total_local_region_num: 31, total_remote_region_num: 0}          | keep order:false                                                                                                                                                                                                                              | N/A     | N/A     |
|                                   └─HashJoin_53(Probe)                       | 595696793.71 | 600037902 | mpp[tiflash] |                | tiflash_task:{proc max:7.79s, min:0s, avg: 2.6s, p80:7.79s, p95:7.79s, iters:10155, tasks:3, threads:60}                                                                                                                                                                                                                                                                                              | inner join, equal:[eq(tpch100.supplier.s_suppkey, tpch100.lineitem.l_suppkey)]                                                                                                                                                                | N/A     | N/A     |
|                                     ├─ExchangeReceiver_60(Build)             | 991431.00    | 3000000   | mpp[tiflash] |                | tiflash_task:{proc max:125.3ms, min:0s, avg: 41.8ms, p80:125.3ms, p95:125.3ms, iters:48, tasks:3, threads:60}                                                                                                                                                                                                                                                                                         |                                                                                                                                                                                                                                               | N/A     | N/A     |
|                                     │ └─ExchangeSender_59                    | 991431.00    | 1000000   | mpp[tiflash] |                | tiflash_task:{proc max:78.4ms, min:0s, avg: 26.1ms, p80:78.4ms, p95:78.4ms, iters:16, tasks:3, threads:40}                                                                                                                                                                                                                                                                                            | ExchangeType: Broadcast, Compression: FAST                                                                                                                                                                                                    | N/A     | N/A     |
|                                     │   └─HashJoin_54                        | 991431.00    | 1000000   | mpp[tiflash] |                | tiflash_task:{proc max:47.4ms, min:0s, avg: 15.8ms, p80:47.4ms, p95:47.4ms, iters:16, tasks:3, threads:40}                                                                                                                                                                                                                                                                                            | inner join, equal:[eq(tpch100.nation.n_nationkey, tpch100.supplier.s_nationkey)]                                                                                                                                                              | N/A     | N/A     |
|                                     │     ├─ExchangeReceiver_57(Build)       | 25.00        | 50        | mpp[tiflash] |                | tiflash_task:{proc max:10.4ms, min:0s, avg: 3.47ms, p80:10.4ms, p95:10.4ms, iters:2, tasks:3, threads:40}                                                                                                                                                                                                                                                                                             |                                                                                                                                                                                                                                               | N/A     | N/A     |
|                                     │     │ └─ExchangeSender_56              | 25.00        | 25        | mpp[tiflash] |                | tiflash_task:{proc max:9.21ms, min:0s, avg: 3.07ms, p80:9.21ms, p95:9.21ms, iters:1, tasks:3, threads:1}                                                                                                                                                                                                                                                                                              | ExchangeType: Broadcast, Compression: FAST                                                                                                                                                                                                    | N/A     | N/A     |
|                                     │     │   └─TableFullScan_55             | 25.00        | 25        | mpp[tiflash] | table:nation   | tiflash_task:{proc max:9.21ms, min:0s, avg: 3.07ms, p80:9.21ms, p95:9.21ms, iters:1, tasks:3, threads:1}, tiflash_scan:{dtfile:{total_scanned_packs:1, total_skipped_packs:0, total_scanned_rows:25, total_skipped_rows:0, total_rs_index_load_time: 0ms, total_read_time: 0ms}, total_create_snapshot_time: 0ms, total_local_region_num: 1, total_remote_region_num: 0}                              | keep order:false                                                                                                                                                                                                                              | N/A     | N/A     |
|                                     │     └─TableFullScan_58(Probe)          | 1000000.00   | 1000000   | mpp[tiflash] | table:supplier | tiflash_task:{proc max:32.4ms, min:0s, avg: 10.8ms, p80:32.4ms, p95:32.4ms, iters:16, tasks:3, threads:2}, tiflash_scan:{dtfile:{total_scanned_packs:123, total_skipped_packs:0, total_scanned_rows:1000000, total_skipped_rows:0, total_rs_index_load_time: 0ms, total_read_time: 35ms}, total_create_snapshot_time: 0ms, total_local_region_num: 2, total_remote_region_num: 0}                     | keep order:false                                                                                                                                                                                                                              | N/A     | N/A     |
|                                     └─TableFullScan_61(Probe)                | 600037902.00 | 600037902 | mpp[tiflash] | table:lineitem | tiflash_task:{proc max:136.9ms, min:0s, avg: 45.6ms, p80:136.9ms, p95:136.9ms, iters:10155, tasks:3, threads:60}, tiflash_scan:{dtfile:{total_scanned_packs:73859, total_skipped_packs:0, total_scanned_rows:600037902, total_skipped_rows:0, total_rs_index_load_time: 28ms, total_read_time: 46823ms}, total_create_snapshot_time: 16ms, total_local_region_num: 1129, total_remote_region_num: 0}  | keep order:false                                                                                                                                                                                                                              | N/A     | N/A     |
+------------------------------------------------------------------------------+--------------+-----------+--------------+----------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+---------+
37 rows in set (10.33 sec)

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

Signed-off-by: Zhigao Tong <tongzhigao@pingcap.com>

ti-chi-bot · 2023-04-11T05:54:02Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

windtalker
winoros

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

Signed-off-by: Zhigao Tong <tongzhigao@pingcap.com>

planner/core/exhaust_physical_plans.go

windtalker

LGTM

…d-cast-join-ratio

solotzg · 2023-04-11T13:31:03Z

/retest

solotzg · 2023-04-11T14:25:04Z

/test unit-test

solotzg · 2023-04-12T00:45:52Z

/merge

ti-chi-bot · 2023-04-12T00:45:55Z

@solotzg: /merge is only allowed for the committers, you can assign this pull request to the committer in list by filling /assign @committer in the comment to help merge this pull request.

In response to this:

/merge

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

solotzg · 2023-04-12T01:16:38Z

/assign @qw4990

hawkingrei · 2023-04-12T02:39:18Z

/merge

ti-chi-bot · 2023-04-12T02:39:22Z

This pull request has been accepted and is ready to merge.

Commit hash: e92e91f

ti-chi-bot · 2023-04-14T18:19:28Z

@solotzg: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
idc-jenkins-ci-tidb/unit-test	`eaf6f23`	link	true	`/test unit-test`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

1

a1f41f9

Signed-off-by: Zhigao Tong <tongzhigao@pingcap.com>

solotzg changed the title ~~[WIP]~~ planner: set default scale factor when preferring broadcast join Apr 11, 2023

ti-chi-bot removed do-not-merge/invalid-title do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. labels Apr 11, 2023

solotzg added 5 commits April 11, 2023 14:11

2

4c4fd87

Signed-off-by: Zhigao Tong <tongzhigao@pingcap.com>

3

831ce6f

Signed-off-by: Zhigao Tong <tongzhigao@pingcap.com>

4

9c93665

Signed-off-by: Zhigao Tong <tongzhigao@pingcap.com>

5

d046580

Signed-off-by: Zhigao Tong <tongzhigao@pingcap.com>

6

1288c3d

Signed-off-by: Zhigao Tong <tongzhigao@pingcap.com>

ti-chi-bot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Apr 11, 2023

solotzg added 3 commits April 11, 2023 16:20

7

eaf6f23

Signed-off-by: Zhigao Tong <tongzhigao@pingcap.com>

fix

229bf06

Signed-off-by: Zhigao Tong <tongzhigao@pingcap.com>

disable tidb_prefer_broadcast_join_by_exchange_data_size by default

d3a9773

Signed-off-by: Zhigao Tong <tongzhigao@pingcap.com>

solotzg requested a review from a team as a code owner April 11, 2023 09:49

ti-chi-bot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Apr 11, 2023

solotzg changed the title ~~planner: set default scale factor when preferring broadcast join~~ planner: disable tidb_prefer_broadcast_join_by_exchange_data_size by default; set scale factor for estimating costs of broadcast join; Apr 11, 2023

solotzg added 3 commits April 11, 2023 18:07

8

6c66b82

Signed-off-by: Zhigao Tong <tongzhigao@pingcap.com>

Merge remote-tracking branch 'pingcap/master' into broad-cast-join-ratio

678968b

9

d077085

Signed-off-by: Zhigao Tong <tongzhigao@pingcap.com>

winoros approved these changes Apr 11, 2023

View reviewed changes

ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Apr 11, 2023

windtalker reviewed Apr 11, 2023

View reviewed changes

planner/core/exhaust_physical_plans.go Show resolved Hide resolved

solotzg commented Apr 11, 2023

View reviewed changes

planner/core/exhaust_physical_plans.go Outdated Show resolved Hide resolved

Update planner/core/exhaust_physical_plans.go

5f2bc1a

windtalker approved these changes Apr 11, 2023

View reviewed changes

ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Apr 11, 2023

solotzg added 2 commits April 11, 2023 21:26

add more tests

edcbd9d

Merge remote-tracking branch 'origin/broad-cast-join-ratio' into broa…

eebf870

…d-cast-join-ratio

ti-chi-bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Apr 11, 2023

qw4990 approved these changes Apr 11, 2023

View reviewed changes

solotzg added 2 commits April 12, 2023 08:49

10

8f6e010

Merge remote-tracking branch 'pingcap/master' into broad-cast-join-ratio

e92e91f

This was referenced Apr 12, 2023

mpp: optimize data exchange operator #40494

Closed

Optimize MPP Broadcast Join pingcap/tiflash#7084

Closed

ti-chi-bot assigned qw4990 Apr 12, 2023

ti-chi-bot added do-not-merge/needs-triage-completed and removed do-not-merge/needs-triage-completed labels Apr 12, 2023

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Apr 12, 2023

ti-chi-bot merged commit 65e7457 into pingcap:master Apr 12, 2023

solotzg deleted the broad-cast-join-ratio branch April 12, 2023 03:35

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

planner: disable `tidb_prefer_broadcast_join_by_exchange_data_size` by default; set scale factor to optimize estimating broadcast join; #42915

planner: disable `tidb_prefer_broadcast_join_by_exchange_data_size` by default; set scale factor to optimize estimating broadcast join; #42915

solotzg commented Apr 11, 2023 •

edited

Loading

ti-chi-bot commented Apr 11, 2023 •

edited

Loading

windtalker left a comment

solotzg commented Apr 11, 2023

solotzg commented Apr 11, 2023

solotzg commented Apr 12, 2023

ti-chi-bot commented Apr 12, 2023

solotzg commented Apr 12, 2023

hawkingrei commented Apr 12, 2023

ti-chi-bot commented Apr 12, 2023

ti-chi-bot commented Apr 14, 2023

planner: disable tidb_prefer_broadcast_join_by_exchange_data_size by default; set scale factor to optimize estimating broadcast join; #42915

planner: disable tidb_prefer_broadcast_join_by_exchange_data_size by default; set scale factor to optimize estimating broadcast join; #42915

Conversation

solotzg commented Apr 11, 2023 • edited Loading

What problem does this PR solve?

Problem

What is changed and how it works?

Benchmark

TPCH-100 Q9

Check List

Release note

ti-chi-bot commented Apr 11, 2023 • edited Loading

windtalker left a comment

Choose a reason for hiding this comment

solotzg commented Apr 11, 2023

solotzg commented Apr 11, 2023

solotzg commented Apr 12, 2023

ti-chi-bot commented Apr 12, 2023

solotzg commented Apr 12, 2023

hawkingrei commented Apr 12, 2023

ti-chi-bot commented Apr 12, 2023

ti-chi-bot commented Apr 14, 2023

planner: disable `tidb_prefer_broadcast_join_by_exchange_data_size` by default; set scale factor to optimize estimating broadcast join; #42915

planner: disable `tidb_prefer_broadcast_join_by_exchange_data_size` by default; set scale factor to optimize estimating broadcast join; #42915

solotzg commented Apr 11, 2023 •

edited

Loading

ti-chi-bot commented Apr 11, 2023 •

edited

Loading