Skip to content

branch-3.0: [fix](cache) fix sql cache throw npe in cloud mode #47221 #47266

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 1 commit into from
Feb 5, 2025
Merged

branch-3.0: [fix](cache) fix sql cache throw npe in cloud mode #47221 #47266

dataroaring merged 1 commit into from
Feb 5, 2025

Conversation

@github-actions
Copy link
Contributor

@github-actions github-actions bot commented Jan 21, 2025

Cherry-picked from #47221

@924060929
[fix](cache) fix sql cache throw npe in cloud mode (#47221)
73713b7 
fix sql cache throw npe in cloud mode, when some partitions is dropped:
```
2025-01-15 18:18:54,811 WARN (mysql-nio-pool-101426|288) [ConnectProcessor.handleQueryException():537] Process one query failed because unknown reason:
java.lang.NullPointerException: Cannot invoke "org.apache.doris.cloud.catalog.CloudPartition.getDbId()" because "partition" is null
        at org.apache.doris.cloud.catalog.CloudPartition.getSnapshotVisibleVersion(CloudPartition.java:196) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.catalog.OlapTable.getVersionInBatchForCloudMode(OlapTable.java:1190) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.cache.CacheAnalyzer.buildCacheTableForOlapScanNode(CacheAnalyzer.java:700) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.cache.CacheAnalyzer.buildCacheTableList(CacheAnalyzer.java:512) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.cache.CacheAnalyzer.innerCheckCacheModeForNereids(CacheAnalyzer.java:412) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.cache.CacheAnalyzer.getCacheData(CacheAnalyzer.java:522) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.handleCacheStmt(StmtExecutor.java:1725) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.handleQueryStmt(StmtExecutor.java:1831) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.handleQueryWithRetry(StmtExecutor.java:874) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.executeByNereids(StmtExecutor.java:811) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:607) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.queryRetry(StmtExecutor.java:557) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:547) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.ConnectProcessor.executeQuery(ConnectProcessor.java:397) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:238) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.MysqlConnectProcessor.handleQuery(MysqlConnectProcessor.java:194) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.MysqlConnectProcessor.dispatch(MysqlConnectProcessor.java:222) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.MysqlConnectProcessor.processOnce(MysqlConnectProcessor.java:281) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.mysql.ReadListener.lambda$handleEvent$0(ReadListener.java:52) ~[doris-fe.jar:1.2-SNAPSHOT]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
        at java.lang.Thread.run(Thread.java:840) ~[?:?] 
```

This is an unstable exception, so I can not add test
@github-actions github-actions bot requested a review from dataroaring as a code owner January 21, 2025 07:04
@Thearas
Copy link
Contributor

Thearas commented Jan 21, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring reopened this Jan 21, 2025
@Thearas
Copy link
Contributor

Thearas commented Jan 21, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jan 21, 2025

TPC-H: Total hot run time: 41258 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 73713b7a488e0116268d936a18f6c5b87ca50ab5, data reload: false

------ Round 1 ----------------------------------
q1	18048	8594	7365	7365
q2	2731	219	177	177
q3	11140	1125	1248	1125
q4	11097	750	748	748
q5	8090	2926	2835	2835
q6	233	147	147	147
q7	974	604	595	595
q8	9353	1945	2008	1945
q9	6526	6401	6403	6401
q10	7006	2321	2301	2301
q11	466	270	262	262
q12	407	207	219	207
q13	17817	3013	2998	2998
q14	237	214	204	204
q15	564	530	532	530
q16	688	630	610	610
q17	979	558	622	558
q18	7583	6770	6745	6745
q19	1440	1047	1087	1047
q20	495	207	195	195
q21	4093	3278	3336	3278
q22	1127	985	1010	985
Total cold run time: 111094 ms
Total hot run time: 41258 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7296	7226	7224	7224
q2	346	246	239	239
q3	3159	2946	2990	2946
q4	2097	1847	1747	1747
q5	5679	5714	5673	5673
q6	219	139	137	137
q7	2229	1830	1802	1802
q8	3339	3513	3459	3459
q9	8901	8815	8819	8815
q10	3598	3561	3573	3561
q11	602	496	516	496
q12	782	570	572	570
q13	9196	3187	3149	3149
q14	322	264	286	264
q15	576	514	515	514
q16	703	702	677	677
q17	1844	1586	1592	1586
q18	8261	7627	7472	7472
q19	1638	1507	1375	1375
q20	2055	1803	1794	1794
q21	5413	5171	5227	5171
q22	1127	1042	1028	1028
Total cold run time: 69382 ms
Total hot run time: 59699 ms

@doris-robot
Copy link

doris-robot commented Jan 21, 2025

TPC-DS: Total hot run time: 192320 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 73713b7a488e0116268d936a18f6c5b87ca50ab5, data reload: false

query1	995	363	357	357
query2	6525	2156	2089	2089
query3	6701	213	225	213
query4	34040	23625	23551	23551
query5	4340	467	448	448
query6	269	185	181	181
query7	4704	309	306	306
query8	276	213	212	212
query9	9501	2669	2670	2669
query10	476	256	263	256
query11	18055	15186	15195	15186
query12	156	100	101	100
query13	1652	426	433	426
query14	9683	7011	7345	7011
query15	253	170	180	170
query16	8202	466	491	466
query17	1679	577	556	556
query18	2161	314	321	314
query19	382	150	146	146
query20	112	111	106	106
query21	214	100	102	100
query22	4518	4410	4388	4388
query23	34673	33954	34887	33954
query24	11633	2818	2881	2818
query25	720	389	392	389
query26	1820	164	165	164
query27	2882	339	332	332
query28	7871	2433	2421	2421
query29	1026	433	429	429
query30	335	165	157	157
query31	1009	803	820	803
query32	100	57	62	57
query33	782	307	291	291
query34	942	506	516	506
query35	858	716	701	701
query36	1098	965	939	939
query37	162	76	75	75
query38	4035	3913	3909	3909
query39	1498	1419	1420	1419
query40	283	100	101	100
query41	53	50	49	49
query42	114	101	100	100
query43	540	489	497	489
query44	1285	804	810	804
query45	184	170	168	168
query46	1163	729	708	708
query47	1968	1838	1879	1838
query48	469	367	371	367
query49	1251	384	382	382
query50	824	423	417	417
query51	7248	7173	7186	7173
query52	103	93	86	86
query53	257	183	192	183
query54	1209	485	469	469
query55	84	79	82	79
query56	269	255	256	255
query57	1233	1136	1112	1112
query58	237	215	221	215
query59	3109	2909	2970	2909
query60	308	271	264	264
query61	134	132	132	132
query62	829	685	700	685
query63	220	199	189	189
query64	5337	753	739	739
query65	3318	3178	3212	3178
query66	1213	321	329	321
query67	15906	15824	15780	15780
query68	4327	583	578	578
query69	428	285	271	271
query70	1220	1055	1156	1055
query71	342	265	261	261
query72	6534	4400	4009	4009
query73	748	349	360	349
query74	9997	8907	8927	8907
query75	3359	2632	2647	2632
query76	2805	1074	1097	1074
query77	435	274	272	272
query78	10395	9646	9605	9605
query79	2094	616	615	615
query80	799	435	418	418
query81	552	239	238	238
query82	1087	118	120	118
query83	205	145	143	143
query84	237	81	83	81
query85	1504	313	300	300
query86	475	319	274	274
query87	4448	4240	4269	4240
query88	4118	2387	2358	2358
query89	399	293	301	293
query90	1811	187	186	186
query91	180	145	148	145
query92	72	51	53	51
query93	1722	568	548	548
query94	701	291	302	291
query95	355	252	257	252
query96	607	284	276	276
query97	3306	3178	3223	3178
query98	231	212	201	201
query99	1554	1320	1295	1295
Total cold run time: 302533 ms
Total hot run time: 192320 ms

@doris-robot
Copy link

doris-robot commented Jan 21, 2025

ClickBench: Total hot run time: 32.72 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 73713b7a488e0116268d936a18f6c5b87ca50ab5, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.02	0.03
query3	0.23	0.07	0.06
query4	1.63	0.11	0.10
query5	0.53	0.52	0.51
query6	1.15	0.73	0.74
query7	0.02	0.02	0.02
query8	0.03	0.03	0.03
query9	0.58	0.53	0.51
query10	0.56	0.56	0.56
query11	0.14	0.10	0.12
query12	0.16	0.11	0.13
query13	0.61	0.59	0.59
query14	3.10	2.91	2.93
query15	0.90	0.84	0.83
query16	0.39	0.37	0.37
query17	1.07	0.99	1.06
query18	0.24	0.22	0.21
query19	1.98	1.93	1.95
query20	0.01	0.01	0.01
query21	15.36	0.61	0.60
query22	2.89	2.29	1.08
query23	17.01	1.01	0.78
query24	2.70	1.22	1.49
query25	0.23	0.20	0.10
query26	0.50	0.14	0.13
query27	0.05	0.03	0.04
query28	10.34	1.13	1.07
query29	12.60	3.31	3.31
query30	0.25	0.07	0.06
query31	2.87	0.40	0.38
query32	3.24	0.46	0.47
query33	2.98	2.98	3.04
query34	17.02	4.56	4.53
query35	4.64	4.56	4.59
query36	0.67	0.52	0.50
query37	0.09	0.06	0.06
query38	0.05	0.04	0.05
query39	0.04	0.03	0.02
query40	0.16	0.13	0.12
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 107.27 s
Total hot run time: 32.72 s

@dataroaring dataroaring merged commit b1e6ee2 into branch-3.0 Feb 5, 2025
22 checks passed
@github-actions github-actions bot deleted the auto-pick-47221-branch-3.0 branch February 5, 2025 09:50
@gavinchou gavinchou mentioned this pull request Feb 18, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dataroaring dataroaring Awaiting requested review from dataroaring

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@Thearas @doris-robot @dataroaring @924060929