Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Improvement](hms catalog) support show_create_database for hms catalog #28145

Merged
merged 2 commits into from
Dec 8, 2023
Merged

[Improvement](hms catalog) support show_create_database for hms catalog #28145

merged 2 commits into from
Dec 8, 2023

Conversation

Yulei-Yang
Copy link
Contributor

@Yulei-Yang Yulei-Yang commented Dec 7, 2023 •
edited
Loading

show_create_database is designed for internal database, it's not suitable for hms catalog

For hive database, before:
image

After(same like result in hive):
image
Already tested in hive v2.1.1, v2.3.9, v3.1.3

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@Yulei-Yang
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' 
Tpch sf100 test result on commit c988b6716ba4cdb6b1848d7bc8871a5d5c8360f4, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4687	4472	4461	4461
q2	368	145	158	145
q3	1443	1230	1246	1230
q4	1117	922	907	907
q5	3138	3140	3141	3140
q6	252	132	127	127
q7	995	481	469	469
q8	2187	2226	2166	2166
q9	6664	6717	6678	6678
q10	3231	3276	3270	3270
q11	325	214	207	207
q12	346	201	204	201
q13	4617	3832	3798	3798
q14	247	207	210	207
q15	569	521	519	519
q16	433	383	380	380
q17	1020	603	564	564
q18	7451	6984	8812	6984
q19	1506	1384	1397	1384
q20	545	288	319	288
q21	3087	2609	2690	2609
q22	351	284	279	279
Total cold run time: 44579 ms
Total hot run time: 40013 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4365	4394	4379	4379
q2	263	162	170	162
q3	3527	3506	3497	3497
q4	2372	2362	2362	2362
q5	5733	5735	5734	5734
q6	240	119	120	119
q7	2376	1877	1903	1877
q8	3507	3509	3525	3509
q9	9006	8969	8933	8933
q10	3916	4001	3980	3980
q11	499	381	387	381
q12	780	603	595	595
q13	4296	3589	3565	3565
q14	282	250	258	250
q15	569	512	515	512
q16	496	452	496	452
q17	1872	1848	1833	1833
q18	8684	8122	8228	8122
q19	1722	1749	1746	1746
q20	2257	1932	1942	1932
q21	6508	6165	6154	6154
q22	490	439	419	419
Total cold run time: 63760 ms
Total hot run time: 60513 ms

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.28 seconds
stream load tsv: 589 seconds loaded 74807831229 Bytes, about 121 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 28.9 seconds inserted 10000000 Rows, about 346K ops/s
storage size: 17214715819 Bytes

@Yulei-Yang
Copy link
Contributor Author

run buildall

@Yulei-Yang
Copy link
Contributor Author

run clickbench

nextdreamblue
nextdreamblue approved these changes Dec 8, 2023
View reviewed changes
Copy link
Contributor

@nextdreamblue nextdreamblue left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 8, 2023
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Dec 8, 2023

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Dec 8, 2023

PR approved by anyone and no changes requested.

@Yulei-Yang
Copy link
Contributor Author

run p0

@Yulei-Yang
Copy link
Contributor Author

run clickbench

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.39 seconds
stream load tsv: 579 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 28.6 seconds inserted 10000000 Rows, about 349K ops/s
storage size: 17214803035 Bytes

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' 
Tpch sf100 test result on commit 19e813a0d8a52196017583eee1a65d85a6a38a5b, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4685	4442	4480	4442
q2	371	142	158	142
q3	1466	1271	1239	1239
q4	1110	952	887	887
q5	3117	3135	3159	3135
q6	252	133	127	127
q7	991	485	477	477
q8	2205	2236	2156	2156
q9	6644	6669	6683	6669
q10	3229	3266	3248	3248
q11	329	203	200	200
q12	354	207	210	207
q13	5599	3773	3848	3773
q14	239	208	223	208
q15	565	523	516	516
q16	439	393	393	393
q17	1007	609	585	585
q18	7468	7020	8935	7020
q19	1514	1407	1415	1407
q20	511	309	298	298
q21	3054	2648	2704	2648
q22	348	278	282	278
Total cold run time: 45497 ms
Total hot run time: 40055 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4427	4467	4448	4448
q2	268	161	178	161
q3	3547	3533	3529	3529
q4	2373	2376	2366	2366
q5	5728	5724	5726	5724
q6	241	121	122	121
q7	2374	1857	1889	1857
q8	3524	3518	3538	3518
q9	9027	8978	9025	8978
q10	3888	3991	3994	3991
q11	514	380	381	380
q12	763	606	605	605
q13	4324	3584	3606	3584
q14	291	251	253	251
q15	605	518	515	515
q16	513	442	462	442
q17	1903	1871	1863	1863
q18	8585	8265	8321	8265
q19	1743	1729	1779	1729
q20	2245	1950	1929	1929
q21	6506	6218	6117	6117
q22	508	411	433	411
Total cold run time: 63897 ms
Total hot run time: 60784 ms

yuxuan-luo
yuxuan-luo approved these changes Dec 8, 2023
View reviewed changes
Copy link
Contributor

@yuxuan-luo yuxuan-luo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@yiguolei yiguolei merged commit b6e72d5 into apache:master Dec 8, 2023
XuJianxu pushed a commit to XuJianxu/doris that referenced this pull request Dec 14, 2023
@Yulei-Yang
[Improvement](hms catalog) support show_create_database for hms catal…
56b6978 
…og (apache#28145)

* [Improvement](hms catalog) support show_create_database for hms catalog

* update
@Yulei-Yang Yulei-Yang mentioned this pull request Jun 14, 2024
Merged
@xiaokang xiaokang mentioned this pull request Jun 23, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@nextdreamblue nextdreamblue nextdreamblue approved these changes

@yuxuan-luo yuxuan-luo yuxuan-luo approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.0.12-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@Yulei-Yang @doris-robot @nextdreamblue @yuxuan-luo @xiaokang @yiguolei