Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[pipelineX](bug) Fix core dump if cancelled #27449

Merged
merged 3 commits into from
Nov 24, 2023
Merged

[pipelineX](bug) Fix core dump if cancelled #27449

merged 3 commits into from
Nov 24, 2023

Conversation

Gabriel39
Copy link
Contributor

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@Gabriel39
Copy link
Contributor Author

run buildall

github-actions[bot]
github-actions bot reviewed Nov 23, 2023
View reviewed changes
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/src/pipeline/pipeline_x/dependency.h
}
_release_tracker();
}
Status _destroy_agg_status(vectorized::AggregateDataPtr data) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: method '_destroy_agg_status' can be made static [readability-convert-member-functions-to-static]

Suggested change
Status _destroy_agg_status(vectorized::AggregateDataPtr data) {
static Status _destroy_agg_status(vectorized::AggregateDataPtr data) {

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.56% (8449/23110)
Line Coverage: 28.86% (68673/237955)
Region Coverage: 27.83% (35525/127646)
Branch Coverage: 24.57% (18122/73770)
Coverage Report: http://coverage.selectdb-in.cc/coverage/b19d9b35d718ef4ac75507cdd48b0149f595ca19_b19d9b35d718ef4ac75507cdd48b0149f595ca19/report/index.html

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 43.78 seconds
stream load tsv: 562 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.3 seconds inserted 10000000 Rows, about 353K ops/s
storage size: 17098821396 Bytes

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' 
Tpch sf100 test result on commit b19d9b35d718ef4ac75507cdd48b0149f595ca19, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4874	4686	4673	4673
q2	369	151	160	151
q3	2032	1915	1878	1878
q4	1388	1238	1242	1238
q5	3952	3913	4041	3913
q6	256	132	134	132
q7	1446	877	899	877
q8	2778	2775	2777	2775
q9	9674	9518	9392	9392
q10	3446	3509	3510	3509
q11	389	247	255	247
q12	442	285	294	285
q13	4577	3808	3769	3769
q14	313	292	287	287
q15	580	529	537	529
q16	659	582	584	582
q17	1171	968	919	919
q18	7840	7488	7453	7453
q19	1680	1662	1697	1662
q20	567	324	295	295
q21	4429	3976	3991	3976
q22	483	377	369	369
Total cold run time: 53345 ms
Total hot run time: 48911 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4608	4566	4550	4550
q2	350	227	265	227
q3	4015	4014	3987	3987
q4	2707	2692	2695	2692
q5	9550	9584	9593	9584
q6	248	120	126	120
q7	3030	2481	2430	2430
q8	4438	4430	4435	4430
q9	12905	12874	12832	12832
q10	4087	4149	4184	4149
q11	832	629	664	629
q12	983	822	814	814
q13	4296	3608	3558	3558
q14	367	349	345	345
q15	576	530	516	516
q16	753	672	670	670
q17	3809	3850	3926	3850
q18	9519	9012	9087	9012
q19	1795	1788	1794	1788
q20	2399	2063	2039	2039
q21	8808	8719	8698	8698
q22	908	803	825	803
Total cold run time: 80983 ms
Total hot run time: 77723 ms

@Gabriel39
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.57% (8449/23103)
Line Coverage: 28.86% (68670/237910)
Region Coverage: 27.83% (35515/127621)
Branch Coverage: 24.56% (18117/73764)
Coverage Report: http://coverage.selectdb-in.cc/coverage/aa0ecf3882004503c95cfd560d11fa7ed13faab8_aa0ecf3882004503c95cfd560d11fa7ed13faab8/report/index.html

@Gabriel39
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 43.63 seconds
stream load tsv: 570 seconds loaded 74807831229 Bytes, about 125 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.0 seconds inserted 10000000 Rows, about 357K ops/s
storage size: 17099169799 Bytes

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.57% (8449/23101)
Line Coverage: 28.86% (68668/237914)
Region Coverage: 27.83% (35511/127618)
Branch Coverage: 24.55% (18112/73766)
Coverage Report: http://coverage.selectdb-in.cc/coverage/2d95587f9967dff37241d55582422418be3d3f78_2d95587f9967dff37241d55582422418be3d3f78/report/index.html

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' 
Tpch sf100 test result on commit 2d95587f9967dff37241d55582422418be3d3f78, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4903	4677	4720	4677
q2	364	158	146	146
q3	2041	1917	1930	1917
q4	1399	1295	1252	1252
q5	3989	3945	4014	3945
q6	248	125	127	125
q7	1435	877	884	877
q8	2797	2810	2775	2775
q9	9660	9625	9566	9566
q10	3446	3527	3538	3527
q11	375	248	251	248
q12	439	287	293	287
q13	4571	3839	3790	3790
q14	325	286	283	283
q15	573	532	531	531
q16	667	590	581	581
q17	1148	975	944	944
q18	7895	7397	7456	7397
q19	1701	1674	1672	1672
q20	568	329	315	315
q21	4408	3999	4026	3999
q22	477	377	370	370
Total cold run time: 53429 ms
Total hot run time: 49224 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4576	4605	4596	4596
q2	351	218	235	218
q3	4038	4012	4000	4000
q4	2690	2690	2689	2689
q5	9713	9788	9743	9743
q6	245	121	123	121
q7	3033	2476	2478	2476
q8	4470	4542	4487	4487
q9	13009	12892	12920	12892
q10	4071	4189	4157	4157
q11	788	631	650	631
q12	965	826	816	816
q13	4278	3560	3503	3503
q14	380	356	346	346
q15	572	535	529	529
q16	746	670	684	670
q17	3905	3830	3922	3830
q18	9543	9067	9184	9067
q19	1838	1790	1794	1790
q20	2400	2070	2062	2062
q21	8782	8724	8720	8720
q22	924	783	797	783
Total cold run time: 81317 ms
Total hot run time: 78126 ms

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 23, 2023
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

HappenLee
HappenLee approved these changes Nov 23, 2023
View reviewed changes
Copy link
Contributor

@HappenLee HappenLee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@yiguolei yiguolei merged commit 5a82a57 into apache:master Nov 24, 2023
26 of 28 checks passed
seawinde pushed a commit to seawinde/doris that referenced this pull request Nov 28, 2023
@Gabriel39 @seawinde
[pipelineX](bug) Fix core dump if cancelled (apache#27449)
91ede00
XuJianxu pushed a commit to XuJianxu/doris that referenced this pull request Dec 14, 2023
@Gabriel39
[pipelineX](bug) Fix core dump if cancelled (apache#27449)
085a7b1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@github-actions GitHub Actions GitHub Actions left review comments

@BiteTheDDDDt BiteTheDDDDt BiteTheDDDDt approved these changes

@HappenLee HappenLee HappenLee approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@Gabriel39 @doris-robot @BiteTheDDDDt @HappenLee @yiguolei