Skip to content

[fix](load): fix incorrect progress on finished loads #55509

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
liaoxin01 merged 1 commit into from
Sep 1, 2025
Merged

[fix](load): fix incorrect progress on finished loads #55509

liaoxin01 merged 1 commit into from
Sep 1, 2025

Conversation

@kaijchen
Copy link
Member

@kaijchen kaijchen commented Aug 31, 2025

What problem does this PR solve?

Issue Number: CIR-17562

Problem Summary:

Finished load jobs could show less than 100% progress because the status
was not reported correctly when any file scanner had
num_rows_load_total == 0 and num_rows_load_filtered == 0.

This fix ensures the status is reported correctly and finished loads
now display 100% progress.

Before:

| 1756660232981 | load_data_4_objects_large_high_column | FINISHED | 0.00% (0/1000)            | BROKER | ...

After:

| 1756660232984 | load_data_5_objects_large_high_column | FINISHED | 100.00% (1000/1000)       | BROKER | ...

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Aug 31, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@kaijchen
Copy link
Member Author

kaijchen commented Aug 31, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Aug 31, 2025

TPC-H: Total hot run time: 33779 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit da9139d1147ecde2acd618d5a4606745f0634e28, data reload: false

------ Round 1 ----------------------------------
q1	17597	5233	5064	5064
q2	1999	323	210	210
q3	10267	1278	714	714
q4	10230	1001	516	516
q5	7530	2365	2365	2365
q6	187	178	141	141
q7	919	776	636	636
q8	9356	1343	1107	1107
q9	6903	5053	5111	5053
q10	6945	2376	1961	1961
q11	507	301	276	276
q12	359	361	233	233
q13	17783	3684	2990	2990
q14	247	249	220	220
q15	558	508	481	481
q16	417	439	380	380
q17	593	846	381	381
q18	7499	7043	6948	6948
q19	1454	969	598	598
q20	351	354	231	231
q21	3735	2610	2295	2295
q22	1065	1043	979	979
Total cold run time: 106501 ms
Total hot run time: 33779 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5382	5095	5105	5095
q2	255	335	234	234
q3	2149	2686	2284	2284
q4	1356	1760	1416	1416
q5	4217	4197	4641	4197
q6	216	176	131	131
q7	2067	1989	1868	1868
q8	2664	2625	2524	2524
q9	7300	7379	7248	7248
q10	3157	3410	2828	2828
q11	588	517	525	517
q12	675	769	641	641
q13	3574	3894	3267	3267
q14	295	297	296	296
q15	521	477	501	477
q16	457	486	449	449
q17	1171	1556	1424	1424
q18	7741	7923	7497	7497
q19	856	850	981	850
q20	1928	1939	1840	1840
q21	4859	4276	4326	4276
q22	1125	1058	1036	1036
Total cold run time: 52553 ms
Total hot run time: 50395 ms

@doris-robot
Copy link

doris-robot commented Aug 31, 2025

TPC-DS: Total hot run time: 187082 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit da9139d1147ecde2acd618d5a4606745f0634e28, data reload: false

query1	1043	437	440	437
query2	6586	1755	1803	1755
query3	6753	232	236	232
query4	26282	24232	23365	23365
query5	4822	683	519	519
query6	332	271	242	242
query7	5129	541	305	305
query8	423	272	253	253
query9	8821	3025	3024	3024
query10	498	359	311	311
query11	16003	15071	14803	14803
query12	172	125	116	116
query13	1684	585	445	445
query14	9562	5864	5842	5842
query15	213	184	175	175
query16	7672	680	460	460
query17	1195	746	628	628
query18	2047	488	328	328
query19	208	228	175	175
query20	134	126	118	118
query21	217	134	116	116
query22	4223	4227	4051	4051
query23	33767	32907	33011	32907
query24	8135	2400	2364	2364
query25	578	518	440	440
query26	1238	278	167	167
query27	2732	515	365	365
query28	4361	2274	2266	2266
query29	784	614	498	498
query30	281	220	199	199
query31	913	798	734	734
query32	122	81	79	79
query33	586	383	356	356
query34	800	841	520	520
query35	819	848	766	766
query36	986	997	909	909
query37	127	112	94	94
query38	4039	4076	4028	4028
query39	1561	1427	1464	1427
query40	239	138	125	125
query41	67	62	60	60
query42	127	112	121	112
query43	532	502	481	481
query44	1352	866	864	864
query45	185	172	175	172
query46	870	1013	642	642
query47	1763	1874	1766	1766
query48	399	428	325	325
query49	770	516	404	404
query50	645	684	412	412
query51	4133	4097	4072	4072
query52	121	115	108	108
query53	245	283	200	200
query54	609	612	537	537
query55	101	94	92	92
query56	326	323	325	323
query57	1194	1207	1121	1121
query58	292	275	285	275
query59	2653	2718	2725	2718
query60	351	346	345	345
query61	168	193	188	188
query62	811	753	672	672
query63	237	200	199	199
query64	4566	1246	968	968
query65	4309	4250	4195	4195
query66	1104	439	346	346
query67	15594	15501	15103	15103
query68	9309	939	597	597
query69	469	339	298	298
query70	1290	1155	1181	1155
query71	494	343	332	332
query72	5880	5041	4876	4876
query73	729	609	367	367
query74	8914	9199	8831	8831
query75	4373	3075	2635	2635
query76	3786	1133	752	752
query77	945	409	344	344
query78	9514	9674	8829	8829
query79	5871	865	579	579
query80	734	581	519	519
query81	483	258	223	223
query82	490	142	110	110
query83	291	276	252	252
query84	302	117	93	93
query85	866	469	425	425
query86	350	335	302	302
query87	4332	4232	4284	4232
query88	2823	2220	2251	2220
query89	442	330	299	299
query90	2092	229	223	223
query91	156	161	132	132
query92	93	76	72	72
query93	3415	980	648	648
query94	684	431	312	312
query95	410	330	322	322
query96	483	598	281	281
query97	2641	2666	2625	2625
query98	244	214	215	214
query99	1440	1433	1339	1339
Total cold run time: 283956 ms
Total hot run time: 187082 ms

@doris-robot
Copy link

doris-robot commented Aug 31, 2025

ClickBench: Total hot run time: 32.37 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit da9139d1147ecde2acd618d5a4606745f0634e28, data reload: false

query1	0.05	0.05	0.05
query2	0.09	0.05	0.06
query3	0.26	0.08	0.08
query4	1.61	0.12	0.11
query5	0.46	0.44	0.41
query6	1.17	0.64	0.66
query7	0.03	0.03	0.02
query8	0.05	0.04	0.04
query9	0.61	0.53	0.52
query10	0.59	0.57	0.56
query11	0.16	0.11	0.12
query12	0.15	0.12	0.13
query13	0.63	0.62	0.62
query14	0.81	0.85	0.84
query15	0.89	0.86	0.86
query16	0.39	0.40	0.38
query17	1.06	1.05	1.03
query18	0.22	0.20	0.21
query19	1.94	1.78	1.74
query20	0.04	0.01	0.01
query21	15.42	0.97	0.59
query22	0.77	1.05	0.80
query23	14.91	1.38	0.63
query24	6.50	2.03	0.41
query25	0.43	0.21	0.13
query26	0.63	0.16	0.13
query27	0.06	0.06	0.05
query28	9.47	0.92	0.43
query29	12.57	4.00	3.21
query30	3.07	3.05	3.01
query31	2.83	0.60	0.39
query32	3.24	0.56	0.48
query33	3.08	3.11	3.12
query34	16.04	5.47	4.89
query35	4.94	4.85	4.97
query36	0.69	0.54	0.50
query37	0.10	0.08	0.07
query38	0.06	0.05	0.04
query39	0.03	0.03	0.03
query40	0.18	0.16	0.15
query41	0.08	0.03	0.03
query42	0.04	0.02	0.02
query43	0.05	0.03	0.03
Total cold run time: 106.4 s
Total hot run time: 32.37 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Aug 31, 2025

BE UT Coverage Report

Increment line coverage 0.00% (0/1) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 51.65% (17121/33148)
Line Coverage 37.17% (156374/420710)
Region Coverage 31.85% (119286/374542)
Branch Coverage 33.18% (52415/157959)

@hello-stephen
Copy link
Contributor

hello-stephen commented Aug 31, 2025

BE Regression && UT Coverage Report

Increment line coverage 100.00% (1/1) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 70.12% (22834/32564)
Line Coverage 56.36% (237027/420582)
Region Coverage 51.81% (196924/380059)
Branch Coverage 53.51% (85046/158940)

dataroaring
dataroaring approved these changes Sep 1, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring added dev/3.0.x dev/3.1.x usercase Important user case type label labels Sep 1, 2025
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 1, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Sep 1, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Sep 1, 2025

PR approved by anyone and no changes requested.

liaoxin01
liaoxin01 reviewed Sep 1, 2025
View reviewed changes
Copy link
Contributor

@liaoxin01 liaoxin01 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@hello-stephen
Copy link
Contributor

hello-stephen commented Sep 1, 2025

BE Regression && UT Coverage Report

Increment line coverage 100.00% (1/1) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 70.50% (22959/32564)
Line Coverage 56.79% (238866/420582)
Region Coverage 52.27% (198673/380059)
Branch Coverage 53.92% (85694/158940)

@liaoxin01 liaoxin01 merged commit 0190c5e into apache:master Sep 1, 2025
30 of 34 checks passed
github-actions bot pushed a commit that referenced this pull request Sep 1, 2025
@kaijchen
[fix](load): fix incorrect progress on finished loads (#55509)
b1ed101 
Finished load jobs could show less than 100% progress because the status
was not reported correctly when any file scanner had
`num_rows_load_total == 0` and `num_rows_load_filtered == 0`.

This fix ensures the status is reported correctly and finished loads
now display 100% progress.

Before:
```
| 1756660232981 | load_data_4_objects_large_high_column | FINISHED | 0.00% (0/1000)            | BROKER | ...
```

After:
```
| 1756660232984 | load_data_5_objects_large_high_column | FINISHED | 100.00% (1000/1000)       | BROKER | ...
```
@github-actions github-actions bot mentioned this pull request Sep 1, 2025
Merged
github-actions bot pushed a commit that referenced this pull request Sep 1, 2025
@kaijchen
[fix](load): fix incorrect progress on finished loads (#55509)
cf2c873 
Finished load jobs could show less than 100% progress because the status
was not reported correctly when any file scanner had
`num_rows_load_total == 0` and `num_rows_load_filtered == 0`.

This fix ensures the status is reported correctly and finished loads
now display 100% progress.

Before:
```
| 1756660232981 | load_data_4_objects_large_high_column | FINISHED | 0.00% (0/1000)            | BROKER | ...
```

After:
```
| 1756660232984 | load_data_5_objects_large_high_column | FINISHED | 100.00% (1000/1000)       | BROKER | ...
```
@github-actions github-actions bot mentioned this pull request Sep 1, 2025
Merged
dataroaring pushed a commit that referenced this pull request Sep 1, 2025
@github-actions @kaijchen
branch-3.0: [fix](load): fix incorrect progress on finished loads #55509
5052f90 
 (#55529)

Cherry-picked from #55509

Co-authored-by: Kaijie Chen <chenkaijie@selectdb.com>
morrySnow pushed a commit that referenced this pull request Sep 4, 2025
@github-actions @kaijchen
branch-3.1: [fix](load): fix incorrect progress on finished loads #55509
e751dc8 
 (#55530)

Cherry-picked from #55509

Co-authored-by: Kaijie Chen <chenkaijie@selectdb.com>
@morrySnow morrySnow mentioned this pull request Sep 22, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@liaoxin01 liaoxin01 liaoxin01 approved these changes

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.0.9-merged dev/3.1.1-merged reviewed usercase Important user case type label

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@kaijchen @Thearas @doris-robot @hello-stephen @liaoxin01 @dataroaring @morrySnow