Skip to content

[bug](cloud) Fix npe when show cluster load statistic #50866

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 3 commits into from
May 17, 2025
Merged

[bug](cloud) Fix npe when show cluster load statistic #50866

dataroaring merged 3 commits into from
May 17, 2025

Conversation

@xy720
Copy link
Member

@xy720 xy720 commented May 13, 2025 •
edited
Loading

What problem does this PR solve?

In cloud mode:

before

MySQL [(none)]> SHOW PROC '/cluster_balance/cluster_load_stat/location_default/REMOTE_CACHE';
ERROR 1105 (HY000): errCode = 2, detailMessage = Cannot invoke "org.apache.doris.clone.LoadStatisticForTag.getStatistic(org.apache.doris.thrift.TStorageMedium)" because "loadStatisticForTag" is null

after

MySQL [(none)]> SHOW PROC "/cluster_balance/cluster_load_stat/location_default/REMOTE_CACHE";
Empty set (0.00 sec)

And may be we should support show cluster load stat by compute-group later.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented May 13, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@xy720
Copy link
Member Author

xy720 commented May 13, 2025

run buildall

@doris-robot
Copy link

doris-robot commented May 13, 2025

TPC-H: Total hot run time: 33851 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 04785ee765ec7d2627332dbba5b04c2de9da15a3, data reload: false

------ Round 1 ----------------------------------
q1	25687	5062	5012	5012
q2	2070	273	180	180
q3	10417	1206	685	685
q4	10245	979	516	516
q5	7506	2310	2352	2310
q6	189	159	128	128
q7	919	735	606	606
q8	9296	1225	1069	1069
q9	6723	5066	5072	5066
q10	6852	2320	1916	1916
q11	493	287	269	269
q12	349	353	215	215
q13	17785	3623	3034	3034
q14	234	237	205	205
q15	533	494	487	487
q16	420	431	374	374
q17	596	842	351	351
q18	7625	7192	7284	7192
q19	1941	1008	563	563
q20	335	336	223	223
q21	4347	3395	2478	2478
q22	1047	1020	972	972
Total cold run time: 115609 ms
Total hot run time: 33851 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5176	5042	5180	5042
q2	248	320	230	230
q3	2202	2647	2274	2274
q4	1321	1769	1372	1372
q5	4455	4382	4354	4354
q6	213	172	132	132
q7	1999	1967	1721	1721
q8	2608	2612	2490	2490
q9	7187	7195	6927	6927
q10	3062	3136	2733	2733
q11	566	521	508	508
q12	667	759	619	619
q13	3454	3855	3259	3259
q14	285	315	269	269
q15	517	485	483	483
q16	448	490	432	432
q17	1125	1560	1360	1360
q18	7707	7557	7485	7485
q19	807	838	901	838
q20	2063	2002	1867	1867
q21	5011	4748	4799	4748
q22	1090	1060	986	986
Total cold run time: 52211 ms
Total hot run time: 50129 ms

@doris-robot
Copy link

doris-robot commented May 13, 2025

TPC-DS: Total hot run time: 193284 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 04785ee765ec7d2627332dbba5b04c2de9da15a3, data reload: false

query1	1422	1109	1054	1054
query2	6391	1859	1876	1859
query3	10994	4644	4483	4483
query4	52096	24110	23273	23273
query5	5032	602	479	479
query6	343	216	196	196
query7	4874	523	291	291
query8	316	264	257	257
query9	5404	2665	2663	2663
query10	454	335	282	282
query11	15099	14960	14801	14801
query12	173	117	108	108
query13	1047	556	425	425
query14	10178	6261	6215	6215
query15	197	224	199	199
query16	7100	669	489	489
query17	1071	748	600	600
query18	1523	396	333	333
query19	207	198	161	161
query20	126	122	127	122
query21	206	123	111	111
query22	4401	4394	4251	4251
query23	34359	33452	33414	33414
query24	6552	2457	2420	2420
query25	480	472	421	421
query26	693	277	161	161
query27	2221	528	349	349
query28	3072	2160	2139	2139
query29	587	561	420	420
query30	278	222	200	200
query31	870	847	804	804
query32	84	64	67	64
query33	438	354	327	327
query34	781	867	536	536
query35	813	872	757	757
query36	956	964	893	893
query37	118	97	77	77
query38	4244	4200	4076	4076
query39	1498	1430	1458	1430
query40	207	122	119	119
query41	56	53	51	51
query42	123	112	121	112
query43	524	516	505	505
query44	1363	835	823	823
query45	179	178	171	171
query46	860	1048	655	655
query47	1859	1912	1826	1826
query48	401	430	320	320
query49	691	515	433	433
query50	684	699	418	418
query51	4195	4251	4169	4169
query52	114	111	107	107
query53	231	267	193	193
query54	618	620	547	547
query55	86	87	89	87
query56	325	331	289	289
query57	1154	1193	1141	1141
query58	283	276	271	271
query59	2825	2838	2672	2672
query60	353	338	338	338
query61	161	167	124	124
query62	733	789	707	707
query63	229	191	192	191
query64	1506	1031	703	703
query65	4450	4356	4251	4251
query66	710	385	306	306
query67	15757	15495	15531	15495
query68	7059	885	586	586
query69	535	306	265	265
query70	1153	1133	1105	1105
query71	480	319	304	304
query72	5847	4684	4610	4610
query73	1289	583	358	358
query74	8985	9063	9026	9026
query75	4082	3226	2749	2749
query76	4074	1204	824	824
query77	785	372	285	285
query78	10102	10185	9293	9293
query79	2304	846	579	579
query80	584	514	452	452
query81	491	255	225	225
query82	254	132	95	95
query83	256	260	230	230
query84	292	106	96	96
query85	793	344	394	344
query86	365	312	287	287
query87	4418	4382	4406	4382
query88	3934	2302	2296	2296
query89	397	325	291	291
query90	1783	217	220	217
query91	141	146	112	112
query92	71	59	59	59
query93	1837	987	589	589
query94	667	408	308	308
query95	369	301	286	286
query96	510	586	283	283
query97	3137	3217	3105	3105
query98	240	216	203	203
query99	1358	1390	1276	1276
Total cold run time: 295967 ms
Total hot run time: 193284 ms

@doris-robot
Copy link

doris-robot commented May 13, 2025

ClickBench: Total hot run time: 29.71 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 04785ee765ec7d2627332dbba5b04c2de9da15a3, data reload: false

query1	0.04	0.04	0.02
query2	0.12	0.11	0.12
query3	0.24	0.20	0.19
query4	1.59	0.19	0.19
query5	0.61	0.60	0.60
query6	1.18	0.72	0.72
query7	0.02	0.02	0.02
query8	0.04	0.03	0.04
query9	0.57	0.52	0.51
query10	0.56	0.57	0.57
query11	0.15	0.12	0.11
query12	0.15	0.12	0.11
query13	0.61	0.60	0.60
query14	0.77	0.79	0.80
query15	0.89	0.86	0.86
query16	0.38	0.39	0.38
query17	1.04	1.04	1.04
query18	0.22	0.21	0.21
query19	1.93	1.81	1.80
query20	0.01	0.01	0.02
query21	15.42	0.91	0.56
query22	0.73	1.09	0.68
query23	15.04	1.37	0.63
query24	6.88	1.98	1.13
query25	0.49	0.13	0.15
query26	0.56	0.15	0.15
query27	0.06	0.06	0.05
query28	10.12	0.86	0.45
query29	12.56	4.01	3.34
query30	0.26	0.10	0.06
query31	2.82	0.63	0.38
query32	3.23	0.55	0.46
query33	3.04	3.07	3.07
query34	15.90	5.17	4.47
query35	4.52	4.48	4.53
query36	0.65	0.49	0.48
query37	0.09	0.06	0.06
query38	0.06	0.04	0.04
query39	0.03	0.02	0.02
query40	0.17	0.14	0.13
query41	0.07	0.02	0.02
query42	0.03	0.02	0.03
query43	0.04	0.04	0.03
Total cold run time: 103.89 s
Total hot run time: 29.71 s

@xy720
Copy link
Member Author

xy720 commented May 14, 2025

run cloud_p0

@xy720
Copy link
Member Author

xy720 commented May 14, 2025

run feut

@xy720
Copy link
Member Author

xy720 commented May 15, 2025

run buildall

@xy720
Copy link
Member Author

xy720 commented May 15, 2025

run buildall

@doris-robot
Copy link

doris-robot commented May 15, 2025

TPC-H: Total hot run time: 33733 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7971729f647a8202b54d06b2ea732ed8f981b70e, data reload: false

------ Round 1 ----------------------------------
q1	25943	5087	5015	5015
q2	2071	282	184	184
q3	10398	1232	695	695
q4	10245	998	506	506
q5	7539	2417	2458	2417
q6	185	165	132	132
q7	887	761	615	615
q8	9309	1289	1053	1053
q9	6706	5026	5070	5026
q10	6812	2290	1872	1872
q11	475	287	267	267
q12	336	343	206	206
q13	17762	3664	3065	3065
q14	222	228	214	214
q15	523	479	482	479
q16	415	433	380	380
q17	611	876	383	383
q18	7664	7220	7108	7108
q19	1207	954	566	566
q20	344	340	235	235
q21	3763	3206	2324	2324
q22	1063	998	991	991
Total cold run time: 114480 ms
Total hot run time: 33733 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5096	5034	5057	5034
q2	244	325	227	227
q3	2202	2669	2285	2285
q4	1385	1782	1356	1356
q5	4416	4487	4416	4416
q6	208	173	123	123
q7	1978	1933	1747	1747
q8	2517	2528	2510	2510
q9	7197	6997	7154	6997
q10	3009	3186	2707	2707
q11	556	497	488	488
q12	695	736	639	639
q13	3516	3816	3241	3241
q14	276	294	269	269
q15	526	486	469	469
q16	424	484	438	438
q17	1174	1517	1355	1355
q18	7582	7484	7422	7422
q19	786	803	907	803
q20	2014	2003	1937	1937
q21	4729	4335	4128	4128
q22	1077	1035	988	988
Total cold run time: 51607 ms
Total hot run time: 49579 ms

@doris-robot
Copy link

doris-robot commented May 15, 2025

TPC-DS: Total hot run time: 186007 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7971729f647a8202b54d06b2ea732ed8f981b70e, data reload: false

query1	1012	470	486	470
query2	6545	1831	1814	1814
query3	6742	229	218	218
query4	25637	23631	23148	23148
query5	4331	612	495	495
query6	319	201	199	199
query7	4633	478	282	282
query8	294	242	242	242
query9	8616	2611	2575	2575
query10	439	355	271	271
query11	15766	15032	14785	14785
query12	171	124	101	101
query13	1635	511	422	422
query14	9619	6127	6040	6040
query15	201	188	179	179
query16	7438	641	483	483
query17	1218	721	546	546
query18	1978	391	289	289
query19	192	186	149	149
query20	118	117	116	116
query21	208	121	107	107
query22	3996	4212	4024	4024
query23	33814	33012	33208	33012
query24	8381	2362	2373	2362
query25	531	454	411	411
query26	1228	266	156	156
query27	2740	497	336	336
query28	4364	2098	2075	2075
query29	725	559	423	423
query30	284	211	188	188
query31	939	839	783	783
query32	71	66	61	61
query33	539	351	312	312
query34	793	844	511	511
query35	798	827	753	753
query36	925	990	912	912
query37	116	100	80	80
query38	4292	4241	4038	4038
query39	1482	1390	1428	1390
query40	216	120	109	109
query41	56	54	55	54
query42	125	107	117	107
query43	496	487	466	466
query44	1260	797	796	796
query45	178	173	170	170
query46	829	1022	636	636
query47	1763	1784	1718	1718
query48	386	415	307	307
query49	773	497	431	431
query50	632	673	408	408
query51	4190	4197	4137	4137
query52	110	107	128	107
query53	217	252	177	177
query54	568	568	501	501
query55	83	86	82	82
query56	287	298	299	298
query57	1110	1129	1089	1089
query58	265	249	257	249
query59	2535	2564	2566	2564
query60	324	342	302	302
query61	125	123	123	123
query62	817	752	660	660
query63	225	181	186	181
query64	4415	1101	769	769
query65	4310	4261	4292	4261
query66	1128	439	332	332
query67	15801	15502	15391	15391
query68	7861	882	525	525
query69	493	309	283	283
query70	1253	1135	1127	1127
query71	397	356	304	304
query72	6085	4660	4783	4660
query73	680	624	345	345
query74	8807	8847	8829	8829
query75	3202	3222	2731	2731
query76	3274	1187	754	754
query77	464	370	294	294
query78	10077	10144	9283	9283
query79	2367	819	574	574
query80	666	536	449	449
query81	489	277	219	219
query82	449	127	97	97
query83	287	249	235	235
query84	299	111	92	92
query85	791	348	299	299
query86	388	302	308	302
query87	4382	4481	4357	4357
query88	3599	2278	2287	2278
query89	393	313	282	282
query90	1917	209	214	209
query91	141	142	116	116
query92	76	62	55	55
query93	1906	923	578	578
query94	709	397	313	313
query95	375	292	284	284
query96	494	581	284	284
query97	3192	3238	3152	3152
query98	228	215	203	203
query99	1447	1418	1311	1311
Total cold run time: 273542 ms
Total hot run time: 186007 ms

@doris-robot
Copy link

doris-robot commented May 15, 2025

ClickBench: Total hot run time: 29.53 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7971729f647a8202b54d06b2ea732ed8f981b70e, data reload: false

query1	0.03	0.04	0.03
query2	0.13	0.11	0.11
query3	0.25	0.19	0.19
query4	1.59	0.20	0.11
query5	0.57	0.57	0.55
query6	1.21	0.71	0.73
query7	0.02	0.02	0.02
query8	0.04	0.04	0.03
query9	0.59	0.53	0.52
query10	0.57	0.58	0.57
query11	0.16	0.11	0.11
query12	0.14	0.12	0.12
query13	0.61	0.59	0.60
query14	0.79	0.80	0.81
query15	0.88	0.86	0.86
query16	0.40	0.39	0.39
query17	1.04	1.03	1.04
query18	0.23	0.21	0.21
query19	1.88	1.87	1.82
query20	0.01	0.00	0.01
query21	15.40	0.91	0.53
query22	0.77	1.16	0.69
query23	14.88	1.38	0.66
query24	6.62	1.37	0.98
query25	0.47	0.22	0.18
query26	0.56	0.16	0.13
query27	0.06	0.05	0.06
query28	9.33	0.94	0.45
query29	12.62	3.94	3.35
query30	0.25	0.10	0.06
query31	2.83	0.58	0.38
query32	3.24	0.55	0.47
query33	3.02	3.05	3.15
query34	15.76	5.11	4.49
query35	4.47	4.51	4.49
query36	0.67	0.50	0.48
query37	0.08	0.06	0.06
query38	0.06	0.04	0.04
query39	0.03	0.03	0.03
query40	0.18	0.14	0.13
query41	0.09	0.03	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.02
Total cold run time: 102.59 s
Total hot run time: 29.53 s

@github-actions
Copy link
Contributor

github-actions bot commented May 16, 2025

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels May 16, 2025
@github-actions
Copy link
Contributor

github-actions bot commented May 16, 2025

PR approved by anyone and no changes requested.

dataroaring
dataroaring approved these changes May 17, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit 53b7f14 into apache:master May 17, 2025
26 of 27 checks passed
github-actions bot pushed a commit that referenced this pull request May 17, 2025
@xy720
[bug](cloud) Fix npe when show cluster load statistic (#50866)
16d3d69 
### What problem does this PR solve?

In cloud mode:

before
```
MySQL [(none)]> SHOW PROC '/cluster_balance/cluster_load_stat/location_default/REMOTE_CACHE';
ERROR 1105 (HY000): errCode = 2, detailMessage = Cannot invoke "org.apache.doris.clone.LoadStatisticForTag.getStatistic(org.apache.doris.thrift.TStorageMedium)" because "loadStatisticForTag" is null
```

after
```
MySQL [(none)]> SHOW PROC "/cluster_balance/cluster_load_stat/location_default/REMOTE_CACHE";
Empty set (0.00 sec)
```

And may be we should support show cluster load stat by compute-group
later.
@github-actions github-actions bot mentioned this pull request May 17, 2025
Merged
dataroaring pushed a commit that referenced this pull request May 17, 2025
@github-actions @xy720
branch-3.0: [bug](cloud) Fix npe when show cluster load statistic #50866
25d7592 
 (#50992)

Cherry-picked from #50866

Co-authored-by: xy720 <22125576+xy720@users.noreply.github.com>
koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025
@xy720 @koarz
[bug](cloud) Fix npe when show cluster load statistic (apache#50866)
a1723b7 
### What problem does this PR solve?

In cloud mode:

before
```
MySQL [(none)]> SHOW PROC '/cluster_balance/cluster_load_stat/location_default/REMOTE_CACHE';
ERROR 1105 (HY000): errCode = 2, detailMessage = Cannot invoke "org.apache.doris.clone.LoadStatisticForTag.getStatistic(org.apache.doris.thrift.TStorageMedium)" because "loadStatisticForTag" is null
```

after
```
MySQL [(none)]> SHOW PROC "/cluster_balance/cluster_load_stat/location_default/REMOTE_CACHE";
Empty set (0.00 sec)
```

And may be we should support show cluster load stat by compute-group
later.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@cambyzju cambyzju cambyzju approved these changes

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.0.6-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@xy720 @Thearas @doris-robot @cambyzju @dataroaring