Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[enhance](catalog)External partition prune return partitionName instead of partitionId #44415

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

zddr
Copy link
Contributor

@zddr zddr commented Nov 21, 2024

What problem does this PR solve?

The partition ID of external data sources is meaningless, and some data sources only have partition names, so the return result of partition pruning is replaced with name instead of ID
Issue Number: close #xxx

Related PR: #xxx

Problem Summary:
External partition prune return partitionName instead of partitionId

Release note

External partition prune return partitionName instead of partitionId

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@zddr
Copy link
Contributor Author

zddr commented Nov 21, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40189 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5e4e33e50dcbfa259ed7480bd84b5639ce5f1e75, data reload: false

------ Round 1 ----------------------------------
q1	17580	7549	7349	7349
q2	2050	175	186	175
q3	10581	1101	1221	1101
q4	10525	777	770	770
q5	7631	2777	2788	2777
q6	246	152	152	152
q7	978	629	622	622
q8	9234	1877	1907	1877
q9	6682	6489	6453	6453
q10	7001	2310	2356	2310
q11	462	259	258	258
q12	430	218	211	211
q13	17790	3021	3046	3021
q14	253	208	213	208
q15	576	539	526	526
q16	656	579	593	579
q17	984	547	583	547
q18	7308	6681	6707	6681
q19	1341	1025	1069	1025
q20	465	180	182	180
q21	3988	3055	3250	3055
q22	382	318	312	312
Total cold run time: 107143 ms
Total hot run time: 40189 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7349	7345	7354	7345
q2	333	230	229	229
q3	2915	2836	2977	2836
q4	2095	1826	1910	1826
q5	5696	5724	5646	5646
q6	233	148	144	144
q7	2289	1822	1814	1814
q8	3468	3529	3531	3529
q9	8850	8922	8932	8922
q10	3609	3558	3562	3558
q11	606	502	493	493
q12	840	592	635	592
q13	11135	3248	3186	3186
q14	310	270	279	270
q15	573	513	516	513
q16	692	646	661	646
q17	1879	1644	1621	1621
q18	8286	7810	7793	7793
q19	1688	1653	1648	1648
q20	2131	1895	1877	1877
q21	5560	5516	5274	5274
q22	636	568	560	560
Total cold run time: 71173 ms
Total hot run time: 60322 ms

public class OneListPartitionEvaluator
extends DefaultExpressionRewriter<Map<Slot, PartitionSlotInput>> implements OnePartitionEvaluator {
private final long partitionId;
public class OneListPartitionEvaluator<K>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why add this , this seems not used by this pr

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The ID used for partitioning the inner table, the name used for the external table, and the underlying clipping method are compatible with generics

@doris-robot
Copy link

TPC-DS: Total hot run time: 198249 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5e4e33e50dcbfa259ed7480bd84b5639ce5f1e75, data reload: false

query1	1254	931	959	931
query2	6295	2085	2091	2085
query3	10790	3974	3982	3974
query4	67778	28780	23606	23606
query5	4895	467	465	465
query6	401	187	179	179
query7	5552	293	287	287
query8	316	233	228	228
query9	8754	2685	2678	2678
query10	470	284	251	251
query11	17270	15121	15999	15121
query12	159	101	103	101
query13	1463	436	430	430
query14	10326	7715	7697	7697
query15	214	182	187	182
query16	7152	476	489	476
query17	1227	561	560	560
query18	1833	303	310	303
query19	212	154	154	154
query20	120	121	112	112
query21	207	111	104	104
query22	4814	4697	4755	4697
query23	34988	34545	34548	34545
query24	5483	2510	2473	2473
query25	493	406	385	385
query26	666	148	153	148
query27	2027	285	282	282
query28	4194	2478	2454	2454
query29	708	447	420	420
query30	217	153	160	153
query31	1006	840	877	840
query32	67	58	54	54
query33	412	292	293	292
query34	967	532	535	532
query35	854	767	765	765
query36	1105	955	956	955
query37	128	75	75	75
query38	4463	4398	4412	4398
query39	1613	1502	1492	1492
query40	203	97	131	97
query41	44	43	45	43
query42	110	99	100	99
query43	528	515	505	505
query44	1221	866	850	850
query45	178	166	168	166
query46	1161	726	730	726
query47	2065	1951	1932	1932
query48	458	321	322	321
query49	743	408	425	408
query50	841	403	406	403
query51	7397	7238	7095	7095
query52	96	87	93	87
query53	264	181	183	181
query54	536	399	385	385
query55	80	114	76	76
query56	262	228	237	228
query57	1318	1181	1182	1181
query58	225	214	212	212
query59	3267	3012	2982	2982
query60	270	255	244	244
query61	107	106	103	103
query62	825	673	680	673
query63	211	195	186	186
query64	1416	686	625	625
query65	3297	3205	3266	3205
query66	708	310	308	308
query67	16074	15957	15817	15817
query68	3719	589	576	576
query69	438	253	264	253
query70	1201	1100	1162	1100
query71	360	253	258	253
query72	6404	4100	4138	4100
query73	773	365	365	365
query74	10240	9072	9079	9072
query75	3409	2721	2681	2681
query76	1875	1034	1173	1034
query77	470	272	273	272
query78	10495	9482	9430	9430
query79	2158	607	604	604
query80	1373	428	430	428
query81	523	235	232	232
query82	1273	117	121	117
query83	261	152	147	147
query84	278	74	70	70
query85	1037	354	297	297
query86	432	296	291	291
query87	4816	4574	4612	4574
query88	3667	2261	2202	2202
query89	417	301	300	300
query90	1961	196	191	191
query91	140	102	100	100
query92	67	52	47	47
query93	2904	535	543	535
query94	850	314	296	296
query95	354	253	253	253
query96	630	279	279	279
query97	2868	2646	2700	2646
query98	220	199	197	197
query99	1650	1285	1290	1285
Total cold run time: 322061 ms
Total hot run time: 198249 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.16 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5e4e33e50dcbfa259ed7480bd84b5639ce5f1e75, data reload: false

query1	0.04	0.04	0.03
query2	0.07	0.03	0.03
query3	0.25	0.07	0.07
query4	1.62	0.10	0.11
query5	0.43	0.42	0.41
query6	1.14	0.67	0.64
query7	0.02	0.02	0.02
query8	0.04	0.03	0.04
query9	0.58	0.51	0.51
query10	0.56	0.58	0.56
query11	0.14	0.10	0.10
query12	0.14	0.12	0.11
query13	0.60	0.60	0.59
query14	2.71	2.72	2.75
query15	0.89	0.81	0.82
query16	0.39	0.39	0.38
query17	1.01	1.06	1.00
query18	0.23	0.21	0.21
query19	1.97	1.81	1.91
query20	0.02	0.01	0.01
query21	15.37	0.58	0.58
query22	2.47	2.77	1.70
query23	17.23	0.94	0.77
query24	2.95	0.84	0.75
query25	0.15	0.11	0.21
query26	0.36	0.14	0.14
query27	0.04	0.04	0.04
query28	11.09	1.11	1.08
query29	12.53	3.32	3.34
query30	0.24	0.06	0.06
query31	2.87	0.38	0.38
query32	3.26	0.45	0.46
query33	3.10	3.04	3.05
query34	17.13	4.40	4.41
query35	4.48	4.50	4.43
query36	0.68	0.50	0.47
query37	0.09	0.07	0.06
query38	0.04	0.04	0.03
query39	0.04	0.02	0.03
query40	0.16	0.13	0.12
query41	0.07	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 107.27 s
Total hot run time: 32.16 s

@zddr
Copy link
Contributor Author

zddr commented Nov 21, 2024

run p0

@zddr
Copy link
Contributor Author

zddr commented Nov 21, 2024

run cloud_p0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants