Skip to content

branch-3.1: [fix](hudi)fix hudi get current schema for not partitioned table. #51152 #52623

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morrySnow merged 2 commits into from
Jul 2, 2025
Merged

branch-3.1: [fix](hudi)fix hudi get current schema for not partitioned table. #51152 #52623

morrySnow merged 2 commits into from
Jul 2, 2025

Conversation

@hubgeter
Copy link
Contributor

@hubgeter hubgeter commented Jul 2, 2025 •
edited by morrySnow
Loading

bp #51152

@hubgeter
[fix](hudi)fix hudi get current schema for not partitioned table. (ap…
69c2ebd 
…ache#51152)

### What problem does this PR solve?

Related PR: apache#49956

Problem Summary:
In pr apache#49956, the concept of `HudiMvccSnapshot` is introduced to
implement `hudi asynchronous materialized view partition refresh`. This
pr uses the `LastUpdateTimestamp` of `TablePartitionValues` ​​in
`HudiMvccSnapshot` to obtain the hudi schema, which will cause the
`LastUpdateTimestamp` value to be always 0 if the table is not a
partitioned table. This will result in the actual hudischema not being
obtained. This pr refers to `IcebergMvccSnapshot` and adds the concept
of `timestamp` in `HudiMvccSnapshot` to obtain the correct hudi schema.
Correct hudi schema: It contains information such as column unique id
@hubgeter hubgeter requested a review from morrySnow as a code owner July 2, 2025 01:50
@Thearas
Copy link
Contributor

Thearas commented Jul 2, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@hubgeter
Copy link
Contributor Author

hubgeter commented Jul 2, 2025

run buildall

@morrySnow morrySnow changed the title [fix](hudi)fix hudi get current schema for not partitioned table. (#51152) branch-3.1: [fix](hudi)fix hudi get current schema for not partitioned table. #51152 Jul 2, 2025
@hubgeter
Copy link
Contributor Author

hubgeter commented Jul 2, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jul 2, 2025

TPC-H: Total hot run time: 39605 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ae7b8ca5fb61207ee257b6c4e6c470c01d7f55f7, data reload: false

------ Round 1 ----------------------------------
q1	17581	6749	6633	6633
q2	2088	180	179	179
q3	10510	1134	1152	1134
q4	10218	712	682	682
q5	7750	2847	2842	2842
q6	208	134	136	134
q7	990	637	610	610
q8	9355	1947	1996	1947
q9	6596	6393	6375	6375
q10	6972	2304	2333	2304
q11	467	262	273	262
q12	417	207	206	206
q13	17786	2973	2980	2973
q14	233	202	205	202
q15	512	467	457	457
q16	482	378	367	367
q17	988	594	574	574
q18	7120	6541	6474	6474
q19	1314	1010	1002	1002
q20	460	205	199	199
q21	4169	3080	3103	3080
q22	1088	969	985	969
Total cold run time: 107304 ms
Total hot run time: 39605 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6584	6551	6555	6551
q2	326	229	241	229
q3	2889	2823	2901	2823
q4	2052	1793	1779	1779
q5	5686	5714	5710	5710
q6	209	128	127	127
q7	2241	1811	1802	1802
q8	3385	3477	3502	3477
q9	8859	8845	8895	8845
q10	3569	3556	3531	3531
q11	593	482	488	482
q12	843	608	586	586
q13	10289	3183	3142	3142
q14	305	259	262	259
q15	499	471	464	464
q16	478	437	447	437
q17	1840	1626	1607	1607
q18	8160	7853	7761	7761
q19	1685	1427	1598	1427
q20	2082	1806	1858	1806
q21	5172	5054	4918	4918
q22	1120	1022	1045	1022
Total cold run time: 68866 ms
Total hot run time: 58785 ms

@doris-robot
Copy link

doris-robot commented Jul 2, 2025

TPC-DS: Total hot run time: 195923 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ae7b8ca5fb61207ee257b6c4e6c470c01d7f55f7, data reload: false

query1	1282	904	891	891
query2	6227	1906	1922	1906
query3	10822	4271	4264	4264
query4	33221	23536	23579	23536
query5	3601	460	452	452
query6	268	193	186	186
query7	3994	314	315	314
query8	305	227	233	227
query9	9699	2550	2534	2534
query10	489	272	251	251
query11	17899	15690	15095	15095
query12	160	101	104	101
query13	1558	436	427	427
query14	10296	6578	7054	6578
query15	239	182	201	182
query16	7891	488	472	472
query17	1583	599	606	599
query18	2171	329	325	325
query19	217	162	166	162
query20	127	126	118	118
query21	214	111	107	107
query22	4765	4459	4364	4364
query23	34765	34110	33855	33855
query24	10839	2875	2905	2875
query25	628	429	426	426
query26	739	179	181	179
query27	2262	365	359	359
query28	6160	2119	2138	2119
query29	809	457	450	450
query30	274	167	165	165
query31	1032	815	836	815
query32	100	55	55	55
query33	660	294	307	294
query34	915	511	507	507
query35	895	735	744	735
query36	1123	946	921	921
query37	104	66	70	66
query38	4102	3930	3984	3930
query39	1524	1460	1486	1460
query40	210	100	101	100
query41	47	55	45	45
query42	121	100	104	100
query43	532	487	502	487
query44	1174	827	802	802
query45	188	173	172	172
query46	1166	742	737	737
query47	1983	1932	1911	1911
query48	442	343	350	343
query49	893	398	384	384
query50	836	448	415	415
query51	7445	7317	7246	7246
query52	107	90	86	86
query53	255	183	183	183
query54	1215	469	464	464
query55	78	75	77	75
query56	281	248	251	248
query57	1319	1210	1192	1192
query58	230	219	208	208
query59	3255	3041	2995	2995
query60	290	263	272	263
query61	109	109	105	105
query62	838	699	712	699
query63	230	186	188	186
query64	3890	683	648	648
query65	3350	3302	3293	3293
query66	829	317	299	299
query67	15950	15540	15657	15540
query68	4393	591	582	582
query69	422	263	265	263
query70	1142	1098	1056	1056
query71	342	254	251	251
query72	6358	4003	3987	3987
query73	757	353	355	353
query74	10293	9214	9076	9076
query75	3408	2657	2636	2636
query76	2719	1071	1068	1068
query77	385	263	260	260
query78	10537	9646	9605	9605
query79	2436	600	609	600
query80	1137	424	420	420
query81	552	222	226	222
query82	911	88	85	85
query83	232	147	150	147
query84	235	79	70	70
query85	1436	306	291	291
query86	484	275	291	275
query87	4407	4235	4234	4234
query88	4168	2341	2325	2325
query89	414	288	286	286
query90	1943	185	182	182
query91	141	105	107	105
query92	64	49	49	49
query93	2220	552	570	552
query94	780	299	314	299
query95	366	253	252	252
query96	609	282	276	276
query97	3254	3107	3200	3107
query98	216	203	206	203
query99	1542	1283	1278	1278
Total cold run time: 299525 ms
Total hot run time: 195923 ms

@doris-robot
Copy link

doris-robot commented Jul 2, 2025

ClickBench: Total hot run time: 29.19 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ae7b8ca5fb61207ee257b6c4e6c470c01d7f55f7, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.04	0.03
query3	0.23	0.07	0.06
query4	1.64	0.11	0.11
query5	0.51	0.50	0.51
query6	1.17	0.73	0.74
query7	0.02	0.02	0.01
query8	0.04	0.04	0.03
query9	0.55	0.51	0.51
query10	0.55	0.55	0.56
query11	0.14	0.10	0.10
query12	0.14	0.11	0.11
query13	0.61	0.59	0.59
query14	0.80	0.81	0.81
query15	0.84	0.82	0.85
query16	0.39	0.40	0.39
query17	1.06	1.09	1.06
query18	0.24	0.22	0.21
query19	1.89	1.73	1.81
query20	0.02	0.01	0.01
query21	15.41	0.59	0.56
query22	2.43	2.45	1.31
query23	16.84	1.10	0.84
query24	3.30	0.54	0.58
query25	0.11	0.09	0.08
query26	0.37	0.14	0.13
query27	0.06	0.05	0.05
query28	11.06	0.54	0.45
query29	12.60	3.26	3.22
query30	0.24	0.06	0.06
query31	2.87	0.40	0.40
query32	3.23	0.47	0.46
query33	2.94	2.97	3.03
query34	17.01	4.46	4.50
query35	4.52	4.54	4.50
query36	0.67	0.47	0.47
query37	0.09	0.06	0.06
query38	0.05	0.03	0.03
query39	0.03	0.02	0.02
query40	0.16	0.12	0.12
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 105.08 s
Total hot run time: 29.19 s

@morrySnow morrySnow merged commit 7e145bb into apache:branch-3.1 Jul 2, 2025
21 checks passed
etah000 pushed a commit to etah000/doris that referenced this pull request Jul 7, 2025
@hubgeter @etah000
branch-3.1: [fix](hudi)fix hudi get current schema for not partitione…
7aef7a1 
…d table. apache#51152 (apache#52623)

bp apache#51152
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@hubgeter @Thearas @doris-robot @morrySnow