Skip to content

[fix](hive) fix invalid edit log after inserting hive partition table #58748

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morningman merged 1 commit into from
Dec 6, 2025
Merged

[fix](hive) fix invalid edit log after inserting hive partition table #58748

morningman merged 1 commit into from
Dec 6, 2025

Conversation

@morningman
Copy link
Contributor

@morningman morningman commented Dec 5, 2025 •
edited
Loading

What problem does this PR solve?

Followup #58166
In #58166, the edit log need record "modified partitions" and "new partitions" separately,
so that non-master FE can correctly update the partition cache.
Otherwise, some new partitions can not be queried in non-master FE after inserting.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

hello-stephen commented Dec 5, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@morningman morningman mentioned this pull request Dec 5, 2025
Merged
@morningman morningman changed the title [fix](hive) fix invalid edit after inserting hive partition table [fix](hive) fix invalid edit log after inserting hive partition table Dec 5, 2025
@morningman
Copy link
Contributor Author

morningman commented Dec 5, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Dec 5, 2025

TPC-H: Total hot run time: 33972 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1f3396a51fb706a24efec2ca835b5a18986414ca, data reload: false

------ Round 1 ----------------------------------
q1	17613	5051	4898	4898
q2	2001	312	192	192
q3	10277	1289	735	735
q4	10203	823	313	313
q5	7535	2398	2066	2066
q6	195	175	139	139
q7	947	778	626	626
q8	9349	1363	1047	1047
q9	6930	5283	5303	5283
q10	6842	2185	1772	1772
q11	520	309	287	287
q12	341	370	212	212
q13	17789	3661	3009	3009
q14	230	231	208	208
q15	590	528	519	519
q16	901	888	821	821
q17	704	837	458	458
q18	7648	7322	7095	7095
q19	1161	964	596	596
q20	391	334	228	228
q21	3922	3115	2522	2522
q22	1042	1020	946	946
Total cold run time: 107131 ms
Total hot run time: 33972 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4966	4924	4916	4916
q2	321	386	323	323
q3	2170	2649	2244	2244
q4	1311	1728	1336	1336
q5	4195	4361	4569	4361
q6	220	172	130	130
q7	2051	1935	1866	1866
q8	2703	2464	2487	2464
q9	7615	7531	7502	7502
q10	3024	3315	2801	2801
q11	604	512	491	491
q12	689	792	840	792
q13	3420	3902	3355	3355
q14	282	323	282	282
q15	563	524	513	513
q16	876	909	881	881
q17	1186	1481	1414	1414
q18	7856	7750	7518	7518
q19	1006	867	876	867
q20	1935	2112	1813	1813
q21	4793	4240	4109	4109
q22	1128	1041	983	983
Total cold run time: 52914 ms
Total hot run time: 50961 ms

@doris-robot
Copy link

doris-robot commented Dec 5, 2025

TPC-DS: Total hot run time: 179833 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1f3396a51fb706a24efec2ca835b5a18986414ca, data reload: false

query5	5056	664	507	507
query6	349	234	219	219
query7	4666	472	282	282
query8	314	246	238	238
query9	8723	2665	2644	2644
query10	557	351	275	275
query11	14948	14714	14630	14630
query12	185	119	122	119
query13	1700	487	377	377
query14	6048	3311	3015	3015
query14_1	2956	3037	2927	2927
query15	207	197	188	188
query16	7735	476	458	458
query17	1218	726	626	626
query18	2049	435	370	370
query19	219	194	173	173
query20	135	120	122	120
query21	223	140	118	118
query22	3835	4131	3839	3839
query23	16441	16158	16196	16158
query23_1	16084	16271	15972	15972
query24	7240	1643	1277	1277
query24_1	1210	1188	1240	1188
query25	591	475	415	415
query26	1263	284	173	173
query27	2876	467	307	307
query28	4432	2166	2159	2159
query29	775	564	445	445
query30	307	250	221	221
query31	793	703	628	628
query32	80	69	72	69
query33	697	354	300	300
query34	855	874	547	547
query35	805	849	749	749
query36	902	906	816	816
query37	126	91	79	79
query38	3900	3790	3797	3790
query39	755	739	717	717
query39_1	707	698	865	698
query40	229	127	117	117
query41	67	63	97	63
query42	122	97	98	97
query43	455	421	392	392
query44	1286	771	759	759
query45	202	192	183	183
query46	905	974	592	592
query47	1679	1722	1606	1606
query48	403	327	233	233
query49	789	438	372	372
query50	685	307	239	239
query51	3850	3815	3770	3770
query52	114	96	90	90
query53	246	238	181	181
query54	339	264	230	230
query55	97	80	78	78
query56	343	302	308	302
query57	1141	1157	1102	1102
query58	306	261	248	248
query59	2263	2427	2244	2244
query60	357	322	305	305
query61	161	162	164	162
query62	781	693	626	626
query63	238	176	184	176
query64	4427	1172	872	872
query65	4123	3936	4005	3936
query66	1191	456	330	330
query67	15194	14992	14863	14863
query68	5449	954	669	669
query69	524	303	270	270
query70	1125	987	994	987
query71	458	298	281	281
query72	6031	4928	4850	4850
query73	686	538	302	302
query74	8552	8736	8594	8594
query75	3042	3035	2530	2530
query76	3301	1148	748	748
query77	514	408	325	325
query78	9499	9763	8974	8974
query79	1012	860	608	608
query80	1273	558	482	482
query81	562	270	240	240
query82	429	129	111	111
query83	379	273	263	263
query84	263	124	104	104
query85	931	495	448	448
query86	382	293	297	293
query87	4105	4075	3939	3939
query88	2915	2128	2135	2128
query89	400	315	284	284
query90	1838	161	162	161
query91	174	178	140	140
query92	67	71	63	63
query93	1149	1038	684	684
query94	786	322	279	279
query95	570	329	335	329
query96	542	481	217	217
query97	2616	2680	2555	2555
query98	242	205	196	196
query99	1317	1329	1202	1202
Total cold run time: 265321 ms
Total hot run time: 179833 ms

@doris-robot
Copy link

doris-robot commented Dec 5, 2025

ClickBench: Total hot run time: 27.56 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1f3396a51fb706a24efec2ca835b5a18986414ca, data reload: false

query1	0.05	0.05	0.05
query2	0.09	0.04	0.05
query3	0.26	0.09	0.10
query4	1.61	0.11	0.11
query5	0.27	0.24	0.25
query6	1.20	0.64	0.64
query7	0.03	0.02	0.03
query8	0.05	0.04	0.05
query9	0.56	0.50	0.50
query10	0.56	0.59	0.57
query11	0.15	0.11	0.11
query12	0.15	0.12	0.12
query13	0.62	0.63	0.61
query14	0.97	0.99	0.98
query15	0.81	0.80	0.79
query16	0.39	0.40	0.40
query17	1.08	1.03	1.02
query18	0.23	0.21	0.21
query19	1.94	1.89	1.86
query20	0.02	0.01	0.01
query21	15.43	0.28	0.14
query22	4.70	0.05	0.06
query23	16.04	0.28	0.11
query24	1.78	0.33	0.37
query25	0.09	0.05	0.05
query26	0.14	0.14	0.13
query27	0.06	0.06	0.06
query28	3.57	1.24	1.03
query29	12.57	4.02	3.23
query30	0.28	0.13	0.12
query31	2.82	0.62	0.38
query32	3.23	0.55	0.47
query33	3.06	3.07	3.05
query34	16.79	5.20	4.57
query35	4.55	4.61	4.61
query36	0.66	0.49	0.48
query37	0.10	0.07	0.06
query38	0.08	0.04	0.05
query39	0.04	0.02	0.03
query40	0.17	0.14	0.13
query41	0.09	0.03	0.03
query42	0.04	0.03	0.02
query43	0.04	0.04	0.04
Total cold run time: 97.37 s
Total hot run time: 27.56 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Dec 5, 2025

FE Regression Coverage Report

Increment line coverage 60.00% (18/30) 🎉
Increment coverage report
Complete coverage report

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 5, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Dec 5, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Dec 5, 2025

PR approved by anyone and no changes requested.

morningman added a commit that referenced this pull request Dec 5, 2025
@morningman
branch-4.0: [fix](hive) fix invalid edit log after inserting hive par…
23bc28f 
…tition table (#58747)

bp #58748
@morningman morningman merged commit 3e804d1 into apache:master Dec 6, 2025
37 of 38 checks passed
morningman added a commit to morningman/doris that referenced this pull request Dec 10, 2025
@morningman
[fix](hive) fix invalid edit log after inserting hive partition table (…
0411864 
…apache#58748)

### What problem does this PR solve?

Followup apache#58166
In apache#58166, the edit log need record "modified partitions" and "new
partitions" separately,
so that non-master FE can correctly update the partition cache.
Otherwise, some new partitions can not be queried in non-master FE after
inserting.
@morningman morningman mentioned this pull request Dec 10, 2025
Merged
nagisa-kunhah pushed a commit to nagisa-kunhah/doris that referenced this pull request Dec 14, 2025
@morningman @nagisa-kunhah
[fix](hive) fix invalid edit log after inserting hive partition table (…
158587f 
…apache#58748)

### What problem does this PR solve?

Followup apache#58166
In apache#58166, the edit log need record "modified partitions" and "new
partitions" separately,
so that non-master FE can correctly update the partition cache.
Otherwise, some new partitions can not be queried in non-master FE after
inserting.
morrySnow pushed a commit that referenced this pull request Dec 15, 2025
@morningman @zy-kkk @CalvinKirs
branch-3.1: [opt](hive) Speed up Hive insert on partition tables usin…
a5ce97a 
…g cache #58166 #58606 #58748 (#58886)

picked from #58166 #58606 #58748

---------

Co-authored-by: zy-kkk <zhongyk10@gmail.com>
Co-authored-by: Calvin Kirs <guoqiang@selectdb.com>
zy-kkk added a commit to zy-kkk/doris that referenced this pull request Dec 25, 2025
@zy-kkk
Revert "branch-3.1: [opt](hive) Speed up Hive insert on partition tab…
3819549 
…les using cache apache#58166 apache#58606 apache#58748 (apache#58886)"

This reverts commit a5ce97a.
morrySnow pushed a commit that referenced this pull request Dec 25, 2025
@zy-kkk
branh-3.1: Revert "branch-3.1: [opt](hive) Speed up Hive insert on pa…
d3177ba 
…rtition tables using cache #58166 #58606 #58748 (#58886)" (#59348)

revert #58932
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@CalvinKirs CalvinKirs CalvinKirs approved these changes

@zy-kkk zy-kkk zy-kkk approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.1.4-merged dev/4.0.2-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@morningman @hello-stephen @doris-robot @CalvinKirs @zy-kkk @yiguolei @morrySnow