Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Revert "[improvement](routine-load) add routine load rows check (#25818)" #27304

Merged
merged 1 commit into from
Nov 21, 2023
Merged

Revert "[improvement](routine-load) add routine load rows check (#25818)" #27304

merged 1 commit into from
Nov 21, 2023

Conversation

sollhui
Copy link
Contributor

@sollhui sollhui commented Nov 20, 2023

Proposed changes

This reverts commit 5e3277e. pr #25818

why revert this pr:

When importing array data composed of JSON, one piece of data will be parsed into multiple pieces, so the actual number of pieces received does not match the imported data.For example:

 [   {"v6": "7395.231067", "k4": "1000", "k3": "100", "k2": "10", "k1": "1", "v3": "111", "v2": "t", "v1": "2010-01-01", "k5": "2011-01-01 00:00:00", "v5": "180.998031", "v4": "38.638843"}, {"v6": "2080.504502", "k4": "2000", "k3": "200", "k2": "20", "k1": "2", "v3": "111", "v2": "f", "v1": "2010-01-02", "k5": "2012-01-01 00:00:00", "v5": "539.922834", "v4": "506.044046"}, {"v6": "4605.253205", "k4": "3000", "k3": "300", "k2": "30", "k1": "3", "v3": "11", "v2": "t", "v1": "2010-01-03", "k5": "2013-01-01 00:00:00", "v5": "577.044148", "v4": "377.793209"}, {"v6": "7291.703724", "k4": "4000", "k3": "400", "k2": "40", "k1": "4", "v3": "111", "v2": "n", "v1": "2010-01-04", "k5": "2014-01-01 00:00:00", "v5": "919.067864", "v4": "871.354536"},

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@sollhui
Copy link
Contributor Author

sollhui commented Nov 20, 2023

run buildall

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.52% (8448/23132)
Line Coverage: 28.87% (68687/237881)
Region Coverage: 27.85% (35526/127572)
Branch Coverage: 24.60% (18114/73622)
Coverage Report: http://coverage.selectdb-in.cc/coverage/f21f374e094386fe39ec7035fa747f6b2f4c4402_f21f374e094386fe39ec7035fa747f6b2f4c4402/report/index.html

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
Tpch sf100 test result on commit f21f374e094386fe39ec7035fa747f6b2f4c4402, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4886	4678	4677	4677
q2	356	166	163	163
q3	2050	1985	1888	1888
q4	1396	1319	1267	1267
q5	3992	3966	4050	3966
q6	250	130	133	130
q7	1443	886	901	886
q8	2769	2805	2798	2798
q9	9892	9901	9659	9659
q10	3498	3556	3549	3549
q11	378	257	247	247
q12	442	300	293	293
q13	4602	3847	3817	3817
q14	329	291	296	291
q15	594	546	521	521
q16	669	590	593	590
q17	1151	966	933	933
q18	7769	7304	7293	7293
q19	1679	1695	1690	1690
q20	583	315	301	301
q21	4400	3967	3972	3967
q22	471	371	378	371
Total cold run time: 53599 ms
Total hot run time: 49297 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4626	4603	4590	4590
q2	341	220	253	220
q3	4015	4032	4021	4021
q4	2714	2699	2704	2699
q5	9773	9796	9765	9765
q6	242	123	125	123
q7	3009	2487	2472	2472
q8	4451	4425	4463	4425
q9	13238	13156	13103	13103
q10	4102	4190	4217	4190
q11	780	640	639	639
q12	984	809	806	806
q13	4331	3596	3555	3555
q14	389	344	358	344
q15	584	526	524	524
q16	746	696	678	678
q17	3877	3906	3950	3906
q18	9516	8824	9057	8824
q19	1819	1786	1777	1777
q20	2425	2083	2056	2056
q21	8788	8535	8673	8535
q22	932	846	759	759
Total cold run time: 81682 ms
Total hot run time: 78011 ms

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.17 seconds
stream load tsv: 577 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.3 seconds inserted 10000000 Rows, about 353K ops/s
storage size: 17099821547 Bytes

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 20, 2023
Copy link
Contributor

PR approved by at least one committer and no changes requested.

Copy link
Contributor

PR approved by anyone and no changes requested.

@dataroaring dataroaring merged commit 3e8177b into apache:master Nov 21, 2023
superdiaodiao pushed a commit to superdiaodiao/doris that referenced this pull request Nov 21, 2023
seawinde pushed a commit to seawinde/doris that referenced this pull request Nov 28, 2023
XuJianxu pushed a commit to XuJianxu/doris that referenced this pull request Dec 14, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants