Skip to content

Conversation

@Yulei-Yang
Copy link
Contributor

@Yulei-Yang Yulei-Yang commented Mar 25, 2024

Proposed changes

pick from #32815

Currently, when reading a hive on cosn table, doris return empty result, but the table has data.
iceberg on cosn is ok.
The reason is misuse of cosn's file sytem. according to cosn's doc, its fs.cosn.impl should be org.apache.hadoop.fs.CosFileSystem

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@Yulei-Yang
Copy link
Contributor Author

run buildall

@Yulei-Yang Yulei-Yang changed the title [fix](multicatalog) fix no data error when read table on cosn [fix](multicatalog) fix no data error when read hive table on cosn Mar 25, 2024
@doris-robot
Copy link

TPC-H: Total hot run time: 50490 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1bfa2049d68836c722d6e6ec6004fc1fb3690bf9, data reload: false

------ Round 1 ----------------------------------
q1	17561	4439	4386	4386
q2	2029	152	139	139
q3	10287	1904	1954	1904
q4	10105	1229	1342	1229
q5	8516	3961	3952	3952
q6	228	120	121	120
q7	2021	1581	1631	1581
q8	9286	2733	2727	2727
q9	10845	11090	10886	10886
q10	8704	3532	3572	3532
q11	426	237	242	237
q12	461	299	296	296
q13	18366	3985	4037	3985
q14	345	325	316	316
q15	510	456	453	453
q16	704	606	598	598
q17	1148	941	994	941
q18	7339	6855	6823	6823
q19	1674	1596	1571	1571
q20	502	297	295	295
q21	4455	4137	4121	4121
q22	514	406	398	398
Total cold run time: 116026 ms
Total hot run time: 50490 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4329	4352	4317	4317
q2	317	222	221	221
q3	4168	4126	4160	4126
q4	2765	2748	2743	2743
q5	7336	7315	7226	7226
q6	235	117	114	114
q7	3251	2885	2851	2851
q8	4402	4456	4470	4456
q9	17233	16955	17045	16955
q10	4265	4265	4306	4265
q11	746	680	714	680
q12	1036	871	864	864
q13	5658	3758	3712	3712
q14	454	430	417	417
q15	492	458	450	450
q16	749	702	708	702
q17	3790	3887	3876	3876
q18	8857	8812	8779	8779
q19	1696	1712	1638	1638
q20	2368	2193	2110	2110
q21	8625	8533	8553	8533
q22	1061	960	972	960
Total cold run time: 83833 ms
Total hot run time: 79995 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 199698 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1bfa2049d68836c722d6e6ec6004fc1fb3690bf9, data reload: false

query1	948	397	378	378
query2	6515	2126	2215	2126
query3	6927	205	203	203
query4	19999	17955	17852	17852
query5	19722	6517	6455	6455
query6	291	211	231	211
query7	4167	297	301	297
query8	261	247	234	234
query9	3089	2657	2578	2578
query10	404	291	282	282
query11	11399	10794	10688	10688
query12	115	82	71	71
query13	5569	652	641	641
query14	17674	13133	13445	13133
query15	384	236	240	236
query16	6463	268	261	261
query17	1573	1470	845	845
query18	2305	398	394	394
query19	216	139	146	139
query20	76	78	76	76
query21	192	96	101	96
query22	5300	5052	4996	4996
query23	32671	31980	31862	31862
query24	7663	6539	6456	6456
query25	513	418	405	405
query26	706	162	156	156
query27	2170	289	292	289
query28	6065	2244	2211	2211
query29	2855	2746	2887	2746
query30	239	159	162	159
query31	901	718	736	718
query32	63	61	62	61
query33	402	248	241	241
query34	848	454	460	454
query35	1106	919	944	919
query36	1414	1130	1161	1130
query37	91	60	61	60
query38	3070	2912	2919	2912
query39	1386	1318	1311	1311
query40	301	93	95	93
query41	35	34	36	34
query42	98	87	88	87
query43	624	555	573	555
query44	1114	720	727	720
query45	232	227	229	227
query46	1244	978	978	978
query47	1873	1837	1654	1654
query48	988	677	668	668
query49	641	353	359	353
query50	861	628	618	618
query51	4771	4624	4685	4624
query52	81	75	76	75
query53	439	315	311	311
query54	2640	2427	2475	2427
query55	82	71	72	71
query56	219	204	204	204
query57	1251	1069	1097	1069
query58	192	211	205	205
query59	3546	3142	3045	3045
query60	205	186	205	186
query61	82	86	81	81
query62	843	457	452	452
query63	469	328	323	323
query64	2789	1510	1406	1406
query65	3654	3584	3515	3515
query66	794	375	361	361
query67	15480	15204	14753	14753
query68	11203	634	675	634
query69	561	352	351	351
query70	2212	1412	1445	1412
query71	411	291	310	291
query72	6482	3353	3385	3353
query73	1012	311	322	311
query74	6281	5859	5782	5782
query75	5494	3672	3707	3672
query76	6918	1185	1231	1185
query77	1177	268	241	241
query78	12711	11505	11481	11481
query79	10935	640	642	640
query80	1324	374	381	374
query81	461	235	223	223
query82	846	96	96	96
query83	167	137	133	133
query84	259	69	66	66
query85	846	280	283	280
query86	331	287	290	287
query87	3243	3024	3019	3019
query88	4540	2337	2349	2337
query89	480	293	295	293
query90	1957	215	203	203
query91	155	131	116	116
query92	57	52	51	51
query93	7023	598	623	598
query94	727	209	209	209
query95	1123	1071	1048	1048
query96	631	329	334	329
query97	6504	6425	6276	6276
query98	191	177	172	172
query99	3156	858	919	858
Total cold run time: 320718 ms
Total hot run time: 199698 ms

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 1bfa2049d68836c722d6e6ec6004fc1fb3690bf9 with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       22.3 seconds inserted 10000000 Rows, about 448K ops/s

Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 26, 2024
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

Copy link
Contributor

@lide-reed lide-reed left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@lide-reed lide-reed merged commit ac7a514 into apache:branch-2.0 Mar 26, 2024
@Yulei-Yang Yulei-Yang deleted the fix_cosn_nodata branch March 26, 2024 04:18
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants