[improvement](statistics)Limit the max string length to 1024 while collecting column stats to control BE memory usage. (#32470)

[Jibing-Li]

backport #32470

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

[doris-robot]

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

[Jibing-Li]

run buildall

[doris-robot]

TPC-H: Total hot run time: 49913 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit efbda66bd27eef6e5a142062743a379dfb6118bc, data reload: false

------ Round 1 ----------------------------------
q1	17591	4341	4412	4341
q2	2040	151	151	151
q3	10406	1950	1917	1917
q4	10164	1245	1336	1245
q5	8484	3991	4004	3991
q6	233	121	122	121
q7	2063	1594	1591	1591
q8	9304	2706	2730	2706
q9	10665	10507	10366	10366
q10	8656	3512	3532	3512
q11	422	236	245	236
q12	460	295	302	295
q13	18347	3991	4030	3991
q14	354	316	324	316
q15	502	449	457	449
q16	694	580	590	580
q17	1137	1003	1021	1003
q18	7315	7020	6831	6831
q19	1679	1512	1543	1512
q20	506	298	296	296
q21	4461	4152	4081	4081
q22	484	396	382	382
Total cold run time: 115967 ms
Total hot run time: 49913 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4296	4327	4283	4283
q2	325	221	223	221
q3	4145	4153	4123	4123
q4	2766	2758	2739	2739
q5	7249	7208	7138	7138
q6	231	118	120	118
q7	3239	2946	2810	2810
q8	4379	4482	4476	4476
q9	17101	17066	17005	17005
q10	4223	4305	4292	4292
q11	767	700	671	671
q12	1025	828	836	828
q13	6513	3750	3752	3750
q14	455	422	429	422
q15	494	455	454	454
q16	762	709	718	709
q17	3819	3857	3839	3839
q18	8888	8918	8709	8709
q19	1696	1698	1614	1614
q20	2460	2141	2132	2132
q21	8482	8529	8527	8527
q22	1027	962	910	910
Total cold run time: 84342 ms
Total hot run time: 79770 ms

[doris-robot]

TPC-DS: Total hot run time: 201162 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit efbda66bd27eef6e5a142062743a379dfb6118bc, data reload: false

query1	919	388	380	380
query2	6513	2204	1987	1987
query3	6926	202	202	202
query4	21072	18085	18040	18040
query5	19708	6462	6494	6462
query6	292	222	223	222
query7	4162	302	308	302
query8	266	275	235	235
query9	3189	2769	2684	2684
query10	410	272	302	272
query11	11331	10619	10678	10619
query12	115	74	70	70
query13	5577	671	653	653
query14	17760	13470	13481	13470
query15	382	231	234	231
query16	6466	267	249	249
query17	1733	1447	861	861
query18	2338	404	401	401
query19	217	148	145	145
query20	74	79	75	75
query21	186	91	91	91
query22	5075	5014	5005	5005
query23	32642	31844	31926	31844
query24	7062	6519	6488	6488
query25	533	416	415	415
query26	529	163	157	157
query27	1891	305	290	290
query28	6066	2265	2237	2237
query29	2812	2598	2761	2598
query30	241	160	159	159
query31	894	747	731	731
query32	66	58	59	58
query33	396	251	245	245
query34	846	473	476	473
query35	1142	902	941	902
query36	1299	1309	1267	1267
query37	87	61	59	59
query38	3172	2955	2886	2886
query39	1377	1326	1337	1326
query40	207	95	99	95
query41	37	33	32	32
query42	88	82	82	82
query43	641	723	560	560
query44	1166	724	729	724
query45	239	228	226	226
query46	1235	993	974	974
query47	1769	1606	1794	1606
query48	1023	686	671	671
query49	615	374	385	374
query50	858	597	591	591
query51	4780	4673	4633	4633
query52	88	79	85	79
query53	454	309	312	309
query54	2662	2480	2420	2420
query55	92	75	85	75
query56	217	211	210	210
query57	1324	1100	1090	1090
query58	210	194	196	194
query59	3762	3335	3383	3335
query60	209	200	196	196
query61	84	82	88	82
query62	845	426	448	426
query63	480	339	332	332
query64	2548	1467	1430	1430
query65	3631	3548	3594	3548
query66	817	363	366	363
query67	15481	15753	15195	15195
query68	9512	667	661	661
query69	570	342	343	342
query70	1764	1440	1343	1343
query71	409	300	304	300
query72	6567	3395	3453	3395
query73	732	323	320	320
query74	6367	5888	5813	5813
query75	5347	3659	3716	3659
query76	6119	1142	1194	1142
query77	1010	252	254	252
query78	12483	11981	11708	11708
query79	9839	652	655	652
query80	1424	390	397	390
query81	489	230	230	230
query82	1577	97	92	92
query83	177	132	129	129
query84	264	66	66	66
query85	851	288	279	279
query86	352	309	319	309
query87	3302	2965	3059	2965
query88	4841	2308	2312	2308
query89	476	310	280	280
query90	1916	211	206	206
query91	147	117	116	116
query92	54	48	54	48
query93	6764	607	614	607
query94	682	206	197	197
query95	1120	1041	1072	1041
query96	644	328	326	326
query97	6402	6269	6330	6269
query98	198	178	172	172
query99	2966	1002	902	902
Total cold run time: 316693 ms
Total hot run time: 201162 ms

[doris-robot]

ClickBench: Total hot run time: 31.81 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit efbda66bd27eef6e5a142062743a379dfb6118bc, data reload: false

query1	0.03	0.02	0.02
query2	0.07	0.03	0.02
query3	0.24	0.04	0.05
query4	1.79	0.07	0.06
query5	0.53	0.52	0.52
query6	1.23	0.63	0.62
query7	0.01	0.00	0.01
query8	0.04	0.02	0.02
query9	0.52	0.48	0.47
query10	0.54	0.53	0.52
query11	0.11	0.09	0.09
query12	0.12	0.08	0.08
query13	0.62	0.60	0.62
query14	0.79	0.77	0.80
query15	0.77	0.75	0.78
query16	0.37	0.35	0.36
query17	0.98	1.00	0.99
query18	0.23	0.26	0.24
query19	1.93	1.82	1.86
query20	0.01	0.01	0.01
query21	15.50	0.59	0.57
query22	1.93	2.06	1.89
query23	17.15	0.86	0.97
query24	3.83	3.47	2.00
query25	0.40	0.11	0.04
query26	0.71	0.16	0.16
query27	0.05	0.05	0.04
query28	5.87	0.72	0.78
query29	12.69	2.26	2.30
query30	0.62	0.54	0.53
query31	2.82	0.38	0.36
query32	3.41	0.50	0.49
query33	3.06	3.09	3.06
query34	15.25	4.80	4.77
query35	4.85	4.81	4.84
query36	1.05	1.00	1.01
query37	0.06	0.04	0.05
query38	0.04	0.02	0.02
query39	0.01	0.02	0.02
query40	0.16	0.14	0.14
query41	0.06	0.02	0.02
query42	0.02	0.02	0.01
query43	0.02	0.02	0.01
Total cold run time: 100.49 s
Total hot run time: 31.81 s

[doris-robot]

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit efbda66bd27eef6e5a142062743a379dfb6118bc with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       19.9 seconds inserted 10000000 Rows, about 502K ops/s