Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[enhancement](exception) catch exception for streamload and validate column #40092

Merged
merged 8 commits into from
Aug 29, 2024
Merged

[enhancement](exception) catch exception for streamload and validate column #40092

merged 8 commits into from
Aug 29, 2024

Conversation

yiguolei
Copy link
Contributor

Proposed changes

Issue Number: close #xxx

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@yiguolei
Copy link
Contributor Author

run buildall

github-actions[bot]
github-actions bot reviewed Aug 29, 2024
View reviewed changes
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/src/vec/sink/vtablet_block_convertor.cpp Show resolved Hide resolved
be/src/vec/sink/vtablet_block_convertor.cpp Show resolved Hide resolved
@doris-robot
Copy link

TPC-H: Total hot run time: 38158 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b275b77fd707e912c0cbb7d6ff7b0d9ced36d8d7, data reload: false

------ Round 1 ----------------------------------
q1	17620	4967	4304	4304
q2	2027	186	199	186
q3	11976	991	1094	991
q4	10509	753	717	717
q5	7771	2866	2840	2840
q6	232	140	142	140
q7	973	612	617	612
q8	9339	2064	2083	2064
q9	7306	6540	6592	6540
q10	7013	2242	2187	2187
q11	474	247	239	239
q12	398	226	236	226
q13	18728	3042	3029	3029
q14	270	251	236	236
q15	521	482	501	482
q16	590	492	521	492
q17	997	714	675	675
q18	7461	6929	6908	6908
q19	1413	1066	1067	1066
q20	673	339	331	331
q21	3987	3095	2877	2877
q22	1141	1021	1016	1016
Total cold run time: 111419 ms
Total hot run time: 38158 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4546	4308	4333	4308
q2	376	284	277	277
q3	2925	2612	2666	2612
q4	1985	1657	1754	1657
q5	5669	5699	5776	5699
q6	238	137	136	136
q7	2231	1828	1803	1803
q8	3303	3413	3519	3413
q9	8919	8859	8854	8854
q10	3632	3456	3350	3350
q11	606	510	502	502
q12	826	663	651	651
q13	14812	3201	3262	3201
q14	331	297	293	293
q15	542	494	514	494
q16	623	601	579	579
q17	1832	1536	1546	1536
q18	8031	7828	7894	7828
q19	1782	1537	1502	1502
q20	2179	1911	1910	1910
q21	5733	5513	5414	5414
q22	1138	1088	1095	1088
Total cold run time: 72259 ms
Total hot run time: 57107 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 192661 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b275b77fd707e912c0cbb7d6ff7b0d9ced36d8d7, data reload: false

query1	1274	878	881	878
query2	6415	1972	1889	1889
query3	10729	4173	4085	4085
query4	60127	24057	23358	23358
query5	5573	523	492	492
query6	435	164	168	164
query7	5864	305	288	288
query8	298	202	201	201
query9	8748	2500	2497	2497
query10	467	270	259	259
query11	17792	15070	15323	15070
query12	158	106	107	106
query13	1496	406	373	373
query14	11070	7377	6613	6613
query15	236	183	175	175
query16	7169	480	522	480
query17	1121	617	575	575
query18	2003	337	294	294
query19	299	155	148	148
query20	121	109	127	109
query21	203	103	101	101
query22	4762	4402	4465	4402
query23	34426	33594	33420	33420
query24	5934	2893	2853	2853
query25	527	383	380	380
query26	679	157	157	157
query27	1785	300	277	277
query28	3741	2124	2112	2112
query29	644	404	407	404
query30	237	158	159	158
query31	917	767	773	767
query32	81	54	59	54
query33	453	298	285	285
query34	859	479	498	479
query35	844	772	724	724
query36	1068	919	989	919
query37	153	96	94	94
query38	4009	3799	3898	3799
query39	1431	1398	1557	1398
query40	199	115	118	115
query41	46	48	44	44
query42	123	97	94	94
query43	526	455	474	455
query44	1098	753	756	753
query45	199	171	171	171
query46	1091	730	730	730
query47	1961	1782	1833	1782
query48	364	299	295	295
query49	747	423	441	423
query50	815	413	421	413
query51	7161	7111	7061	7061
query52	100	84	90	84
query53	251	176	181	176
query54	555	449	463	449
query55	77	75	76	75
query56	274	256	258	256
query57	1181	1112	1120	1112
query58	221	264	234	234
query59	3026	2925	2778	2778
query60	291	264	267	264
query61	101	98	94	94
query62	746	660	672	660
query63	216	188	186	186
query64	2855	679	721	679
query65	3232	3139	3171	3139
query66	679	336	329	329
query67	15409	15215	15443	15215
query68	2917	577	588	577
query69	394	287	274	274
query70	1168	1130	1062	1062
query71	332	275	272	272
query72	5067	4038	3972	3972
query73	754	334	334	334
query74	9251	8815	8867	8815
query75	3358	2659	2710	2659
query76	1373	1000	938	938
query77	547	333	313	313
query78	11235	9803	9103	9103
query79	1055	540	527	527
query80	683	516	502	502
query81	463	235	234	234
query82	242	145	145	145
query83	169	148	152	148
query84	251	79	83	79
query85	747	354	344	344
query86	305	306	281	281
query87	4437	4302	4380	4302
query88	3289	2337	2336	2336
query89	382	289	292	289
query90	1918	198	194	194
query91	138	114	112	112
query92	66	54	54	54
query93	1031	543	539	539
query94	697	321	302	302
query95	317	269	265	265
query96	596	269	270	269
query97	3205	3074	3109	3074
query98	224	217	206	206
query99	1510	1260	1237	1237
Total cold run time: 305813 ms
Total hot run time: 192661 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.26 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b275b77fd707e912c0cbb7d6ff7b0d9ced36d8d7, data reload: false

query1	0.05	0.04	0.04
query2	0.07	0.05	0.04
query3	0.22	0.05	0.05
query4	1.68	0.06	0.07
query5	0.49	0.50	0.49
query6	1.13	0.75	0.73
query7	0.02	0.01	0.01
query8	0.05	0.05	0.04
query9	0.55	0.49	0.50
query10	0.54	0.55	0.55
query11	0.15	0.11	0.11
query12	0.14	0.12	0.12
query13	0.63	0.58	0.58
query14	2.01	2.05	2.13
query15	0.86	0.83	0.83
query16	0.37	0.37	0.37
query17	1.06	1.05	1.07
query18	0.22	0.21	0.21
query19	1.84	1.76	1.82
query20	0.01	0.01	0.01
query21	15.41	0.68	0.67
query22	3.98	7.69	1.83
query23	18.25	1.42	1.26
query24	2.06	0.24	0.21
query25	0.16	0.08	0.08
query26	0.27	0.17	0.18
query27	0.09	0.07	0.08
query28	13.30	1.02	1.00
query29	12.62	3.30	3.27
query30	0.24	0.05	0.06
query31	2.90	0.39	0.40
query32	3.25	0.47	0.47
query33	2.97	3.03	3.10
query34	17.61	4.54	4.55
query35	4.60	4.55	4.62
query36	0.66	0.48	0.48
query37	0.20	0.16	0.16
query38	0.15	0.15	0.15
query39	0.05	0.03	0.04
query40	0.16	0.13	0.13
query41	0.09	0.06	0.05
query42	0.06	0.04	0.04
query43	0.05	0.04	0.04
Total cold run time: 111.22 s
Total hot run time: 32.26 s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 29, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

xinyiZzz
xinyiZzz approved these changes Aug 29, 2024
View reviewed changes
Copy link
Contributor

@xinyiZzz xinyiZzz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@gavinchou gavinchou merged commit 6fae472 into apache:master Aug 29, 2024
27 of 30 checks passed
yiguolei added a commit to yiguolei/incubator-doris that referenced this pull request Sep 9, 2024
@yiguolei @Doris-Extras
[enhancement](exception) catch exception for streamload and validate …
dc7a90c 
…column (apache#40092)
@yiguolei yiguolei mentioned this pull request Sep 9, 2024
Closed
@xinyiZzz xinyiZzz mentioned this pull request Sep 9, 2024
Closed
xinyiZzz pushed a commit to xinyiZzz/incubator-doris that referenced this pull request Sep 14, 2024
@yiguolei @xinyiZzz
[enhancement](exception) catch exception for streamload and validate …
fbb7c85 
…column (apache#40092)
@xinyiZzz xinyiZzz mentioned this pull request Sep 14, 2024
Merged
yiguolei added a commit that referenced this pull request Sep 15, 2024
@xinyiZzz @sollhui @yiguolei
[branch-2.1](memory) When Load ends, check memory tracker value retur…
b52b572 
…ns is equal to 0 (#40850)

pick
#38960
#39908
#40043
#40092
#40016
#40439

---------

Co-authored-by: hui lai <1353307710@qq.com>
Co-authored-by: yiguolei <676222867@qq.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@github-actions GitHub Actions GitHub Actions left review comments

@gavinchou gavinchou gavinchou approved these changes

@xinyiZzz xinyiZzz xinyiZzz approved these changes

@platoneko platoneko Awaiting requested review from platoneko platoneko is a code owner

@dataroaring dataroaring Awaiting requested review from dataroaring dataroaring is a code owner

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.7-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@yiguolei @doris-robot @gavinchou @xinyiZzz @Doris-Extras