Skip to content

[opt](iceberg) able to ignore iceberg dangling delete to do count push down #58889

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morningman merged 3 commits into from
Dec 15, 2025
Merged

[opt](iceberg) able to ignore iceberg dangling delete to do count push down #58889

morningman merged 3 commits into from
Dec 15, 2025

Conversation

@morningman
Copy link
Contributor

@morningman morningman commented Dec 10, 2025 •
edited
Loading

What problem does this PR solve?

Related PR: #44039

Problem Summary:

In #44039, we disable the count push down to iceberg table if it has position delete file,
because there may be dangling delete file that will cause the statistics info incorrect.

This PR add a new session variable ignore_iceberg_dangling_delete, default is false,
when set to true, Doris will ignore the dangling delete issue and do count push down for iceberg table,
add user should take care of the result.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Dec 10, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@morningman
Copy link
Contributor Author

morningman commented Dec 10, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Dec 10, 2025

TPC-H: Total hot run time: 35634 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a6bb8511e5260b043c23444ca19867b87f91951e, data reload: false

------ Round 1 ----------------------------------
q1	17615	4256	4070	4070
q2	2027	366	240	240
q3	10156	1329	760	760
q4	10265	798	312	312
q5	7561	2158	1945	1945
q6	186	172	142	142
q7	1016	850	715	715
q8	9353	1455	1120	1120
q9	7031	5326	5352	5326
q10	6777	2392	1997	1997
q11	527	321	294	294
q12	684	742	576	576
q13	17768	3739	3072	3072
q14	294	302	270	270
q15	586	523	508	508
q16	941	931	866	866
q17	735	774	561	561
q18	7914	7258	7139	7139
q19	1123	971	620	620
q20	413	367	252	252
q21	4313	3951	3862	3862
q22	1037	992	987	987
Total cold run time: 108322 ms
Total hot run time: 35634 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4112	4046	4060	4046
q2	329	426	307	307
q3	2148	2715	2323	2323
q4	1315	1722	1299	1299
q5	4249	4849	4739	4739
q6	229	180	130	130
q7	2054	1972	1822	1822
q8	2648	2584	2514	2514
q9	7943	7499	7671	7499
q10	3026	3245	2819	2819
q11	607	507	493	493
q12	703	734	623	623
q13	3584	3898	3331	3331
q14	288	293	295	293
q15	581	514	509	509
q16	892	915	936	915
q17	1323	1415	1498	1415
q18	7894	7583	7632	7583
q19	870	855	925	855
q20	2051	2122	1955	1955
q21	4678	4191	4139	4139
q22	1091	1047	1011	1011
Total cold run time: 52615 ms
Total hot run time: 50620 ms

@doris-robot
Copy link

doris-robot commented Dec 10, 2025

TPC-DS: Total hot run time: 181181 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a6bb8511e5260b043c23444ca19867b87f91951e, data reload: false

query5	5340	651	482	482
query6	324	226	210	210
query7	4232	465	274	274
query8	303	252	239	239
query9	8778	2572	2591	2572
query10	540	373	340	340
query11	15466	15259	14605	14605
query12	179	120	116	116
query13	1277	488	399	399
query14	6239	3234	2912	2912
query14_1	2877	2856	2829	2829
query15	214	208	184	184
query16	805	450	473	450
query17	1094	730	572	572
query18	2542	435	340	340
query19	236	232	215	215
query20	123	114	114	114
query21	224	141	115	115
query22	4179	4239	4065	4065
query23	16569	15950	15964	15950
query23_1	15977	16046	16020	16020
query24	7423	1681	1232	1232
query24_1	1221	1251	1215	1215
query25	558	468	414	414
query26	1263	269	157	157
query27	2782	453	316	316
query28	4486	2159	2150	2150
query29	818	545	434	434
query30	309	239	217	217
query31	812	686	638	638
query32	77	68	67	67
query33	539	322	291	291
query34	915	913	543	543
query35	802	804	732	732
query36	863	908	831	831
query37	128	90	79	79
query38	3926	3809	3754	3754
query39	749	750	708	708
query39_1	704	697	705	697
query40	229	144	126	126
query41	73	70	68	68
query42	108	106	108	106
query43	430	429	404	404
query44	1321	762	778	762
query45	194	192	187	187
query46	879	994	633	633
query47	1732	1716	1634	1634
query48	336	336	268	268
query49	659	456	370	370
query50	656	310	226	226
query51	3807	3871	3786	3786
query52	110	115	103	103
query53	321	362	300	300
query54	312	286	282	282
query55	77	77	71	71
query56	311	313	302	302
query57	1163	1152	1077	1077
query58	292	272	259	259
query59	2417	2582	2390	2390
query60	345	327	317	317
query61	188	190	189	189
query62	695	663	616	616
query63	331	298	300	298
query64	5133	1442	1128	1128
query65	4018	3925	3957	3925
query66	1444	468	334	334
query67	15369	15179	14944	14944
query68	7888	1028	752	752
query69	499	354	318	318
query70	1078	1052	1017	1017
query71	374	310	293	293
query72	6067	4972	5143	4972
query73	654	584	304	304
query74	8845	8708	8599	8599
query75	3549	3501	3132	3132
query76	3887	1144	753	753
query77	520	413	290	290
query78	9539	9861	8832	8832
query79	1417	881	600	600
query80	735	657	569	569
query81	524	276	236	236
query82	208	128	103	103
query83	270	262	241	241
query84	257	120	97	97
query85	905	517	472	472
query86	376	295	278	278
query87	4093	3984	3918	3918
query88	3177	2289	2300	2289
query89	477	441	396	396
query90	2241	162	148	148
query91	174	168	146	146
query92	86	70	63	63
query93	1224	925	574	574
query94	488	308	278	278
query95	588	339	370	339
query96	605	468	206	206
query97	2600	2647	2605	2605
query98	218	195	198	195
query99	1271	1288	1245	1245
Total cold run time: 263486 ms
Total hot run time: 181181 ms

@doris-robot
Copy link

doris-robot commented Dec 10, 2025

ClickBench: Total hot run time: 27.47 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a6bb8511e5260b043c23444ca19867b87f91951e, data reload: false

query1	0.05	0.04	0.04
query2	0.10	0.05	0.05
query3	0.26	0.09	0.08
query4	1.62	0.11	0.11
query5	0.28	0.26	0.26
query6	1.16	0.65	0.64
query7	0.03	0.03	0.03
query8	0.05	0.04	0.05
query9	0.57	0.51	0.52
query10	0.55	0.55	0.55
query11	0.16	0.10	0.12
query12	0.15	0.11	0.13
query13	0.62	0.61	0.61
query14	1.00	0.99	0.98
query15	0.81	0.81	0.82
query16	0.42	0.40	0.41
query17	1.09	1.07	1.02
query18	0.23	0.21	0.21
query19	1.86	1.90	1.82
query20	0.02	0.01	0.02
query21	15.43	0.29	0.14
query22	4.67	0.05	0.05
query23	16.03	0.28	0.10
query24	2.26	0.59	0.27
query25	0.08	0.08	0.08
query26	0.14	0.13	0.14
query27	0.07	0.06	0.05
query28	3.78	1.23	1.02
query29	12.65	4.06	3.34
query30	0.28	0.15	0.16
query31	2.82	0.63	0.39
query32	3.24	0.56	0.46
query33	3.02	2.98	2.98
query34	16.86	5.18	4.50
query35	4.61	4.61	4.58
query36	0.65	0.51	0.50
query37	0.10	0.07	0.06
query38	0.07	0.04	0.04
query39	0.05	0.03	0.03
query40	0.17	0.14	0.14
query41	0.09	0.03	0.03
query42	0.04	0.02	0.02
query43	0.04	0.04	0.03
Total cold run time: 98.18 s
Total hot run time: 27.47 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Dec 10, 2025

FE UT Coverage Report

Increment line coverage 12.50% (1/8) 🎉
Increment coverage report
Complete coverage report

@hello-stephen
Copy link
Contributor

hello-stephen commented Dec 10, 2025

FE Regression Coverage Report

Increment line coverage 100.00% (8/8) 🎉
Increment coverage report
Complete coverage report

1 similar comment
@hello-stephen
Copy link
Contributor

hello-stephen commented Dec 10, 2025

FE Regression Coverage Report

Increment line coverage 100.00% (8/8) 🎉
Increment coverage report
Complete coverage report

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 15, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Dec 15, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Dec 15, 2025

PR approved by anyone and no changes requested.

@morningman morningman merged commit 37ec44d into apache:master Dec 15, 2025
29 of 31 checks passed
morningman added a commit to morningman/doris that referenced this pull request Dec 16, 2025
@morningman
[opt](iceberg) able to ignore iceberg dangling delete to do count pus…
d1b45db 
…h down (apache#58889)

Related PR: apache#44039

Problem Summary:

In apache#44039, we disable the count push down to iceberg table if it has
position delete file,
because there may be dangling delete file that will cause the statistics
info incorrect.

This PR add a new session variable `ignore_iceberg_dangling_delete`,
default is false,
when set to true, Doris will ignore the dangling delete issue and do
count push down for iceberg table,
add user should take care of the result.
@morningman morningman mentioned this pull request Dec 16, 2025
Merged
morningman added a commit to morningman/doris that referenced this pull request Dec 16, 2025
@morningman
[opt](iceberg) able to ignore iceberg dangling delete to do count pus…
1ee73b7 
…h down (apache#58889)

Related PR: apache#44039

Problem Summary:

In apache#44039, we disable the count push down to iceberg table if it has
position delete file,
because there may be dangling delete file that will cause the statistics
info incorrect.

This PR add a new session variable `ignore_iceberg_dangling_delete`,
default is false,
when set to true, Doris will ignore the dangling delete issue and do
count push down for iceberg table,
add user should take care of the result.
@morningman morningman mentioned this pull request Dec 16, 2025
Merged
morrySnow pushed a commit that referenced this pull request Dec 17, 2025
@morningman
branch-3.1: [opt](iceberg) able to ignore iceberg dangling delete to …
5ac3928 
…do count push down #58889 (#59069)

bp #58889
morningman added a commit to morningman/doris that referenced this pull request Dec 22, 2025
@morningman
[opt](iceberg) able to ignore iceberg dangling delete to do count pus…
44aab14 
…h down (apache#58889)

Related PR: apache#44039

Problem Summary:

In apache#44039, we disable the count push down to iceberg table if it has
position delete file,
because there may be dangling delete file that will cause the statistics
info incorrect.

This PR add a new session variable `ignore_iceberg_dangling_delete`,
default is false,
when set to true, Doris will ignore the dangling delete issue and do
count push down for iceberg table,
add user should take care of the result.
morningman added a commit that referenced this pull request Dec 23, 2025
@morningman
branch-4.0: [opt](iceberg) able to ignore iceberg dangling delete to …
97fa3b1 
…do count push down (#58889) (#59068)

bp #58889
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@CalvinKirs CalvinKirs CalvinKirs approved these changes

+1 more reviewer

@suxiaogang223 suxiaogang223 suxiaogang223 approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.0.x dev/3.1.4-merged dev/4.0.3-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@morningman @Thearas @doris-robot @hello-stephen @CalvinKirs @suxiaogang223 @morrySnow