Skip to content

branch-3.0: [fix](load): fix incorrect progress on finished loads #55509 #55529

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 1 commit into from
Sep 1, 2025
Merged

branch-3.0: [fix](load): fix incorrect progress on finished loads #55509 #55529

dataroaring merged 1 commit into from
Sep 1, 2025

Conversation

@github-actions
Copy link
Contributor

@github-actions github-actions bot commented Sep 1, 2025

Cherry-picked from #55509

@kaijchen
[fix](load): fix incorrect progress on finished loads (#55509)
b1ed101 
Finished load jobs could show less than 100% progress because the status
was not reported correctly when any file scanner had
`num_rows_load_total == 0` and `num_rows_load_filtered == 0`.

This fix ensures the status is reported correctly and finished loads
now display 100% progress.

Before:
```
| 1756660232981 | load_data_4_objects_large_high_column | FINISHED | 0.00% (0/1000)            | BROKER | ...
```

After:
```
| 1756660232984 | load_data_5_objects_large_high_column | FINISHED | 100.00% (1000/1000)       | BROKER | ...
```
@Thearas
Copy link
Contributor

Thearas commented Sep 1, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring closed this Sep 1, 2025
@dataroaring dataroaring reopened this Sep 1, 2025
@Thearas
Copy link
Contributor

Thearas commented Sep 1, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Sep 1, 2025

TPC-H: Total hot run time: 40023 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b1ed101e59eff4c2cfba60e802cadae7071a6fa6, data reload: false

------ Round 1 ----------------------------------
q1	17590	6775	6663	6663
q2	2030	188	169	169
q3	10523	1154	1179	1154
q4	10217	776	770	770
q5	7725	2953	2850	2850
q6	213	130	135	130
q7	981	620	602	602
q8	9366	1995	2025	1995
q9	6773	6354	6358	6354
q10	7002	2284	2241	2241
q11	455	266	262	262
q12	403	215	224	215
q13	17798	2970	2998	2970
q14	231	201	206	201
q15	514	455	459	455
q16	472	374	372	372
q17	996	623	626	623
q18	7277	6634	6701	6634
q19	1399	1067	1139	1067
q20	463	205	209	205
q21	3902	3110	3326	3110
q22	1121	996	981	981
Total cold run time: 107451 ms
Total hot run time: 40023 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6621	6548	6557	6548
q2	341	231	230	230
q3	2904	2967	2909	2909
q4	2039	1843	1866	1843
q5	5764	5733	5681	5681
q6	211	131	130	130
q7	2218	1805	1794	1794
q8	3375	3574	3613	3574
q9	8822	8863	8827	8827
q10	3573	3514	3524	3514
q11	583	498	495	495
q12	799	595	654	595
q13	8209	3241	3124	3124
q14	305	266	269	266
q15	504	465	479	465
q16	506	447	444	444
q17	1874	1626	1623	1623
q18	8358	7867	7745	7745
q19	1667	1515	1540	1515
q20	2067	1845	1869	1845
q21	5328	5131	5072	5072
q22	1139	1056	1036	1036
Total cold run time: 67207 ms
Total hot run time: 59275 ms

@doris-robot
Copy link

doris-robot commented Sep 1, 2025

TPC-DS: Total hot run time: 193723 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b1ed101e59eff4c2cfba60e802cadae7071a6fa6, data reload: false

query1	942	421	396	396
query2	6236	1957	1907	1907
query3	8687	217	205	205
query4	33067	23658	23843	23658
query5	3670	470	446	446
query6	297	181	171	171
query7	4220	323	327	323
query8	297	229	224	224
query9	9489	2604	2596	2596
query10	491	264	259	259
query11	18153	15243	15295	15243
query12	158	105	103	103
query13	1550	445	421	421
query14	9390	7358	7297	7297
query15	259	175	185	175
query16	8042	516	543	516
query17	1611	599	598	598
query18	2177	322	338	322
query19	288	162	163	162
query20	125	117	116	116
query21	204	113	105	105
query22	4596	4447	4337	4337
query23	34830	34378	34909	34378
query24	12024	2873	2923	2873
query25	541	429	401	401
query26	1078	173	181	173
query27	2778	366	373	366
query28	7887	2170	2161	2161
query29	694	442	439	439
query30	263	158	159	158
query31	1061	871	820	820
query32	99	55	58	55
query33	769	318	316	316
query34	978	527	526	526
query35	872	740	745	740
query36	1097	991	973	973
query37	190	74	74	74
query38	4164	4033	3974	3974
query39	1550	1453	1499	1453
query40	259	101	98	98
query41	53	46	50	46
query42	116	106	103	103
query43	532	498	494	494
query44	1272	811	819	811
query45	185	173	169	169
query46	1141	745	758	745
query47	2041	1948	1936	1936
query48	501	421	391	391
query49	1144	422	388	388
query50	830	444	445	444
query51	7443	7265	7229	7229
query52	108	95	98	95
query53	274	195	194	194
query54	1102	481	478	478
query55	82	77	81	77
query56	267	266	269	266
query57	1306	1204	1212	1204
query58	223	212	218	212
query59	3227	3019	2970	2970
query60	308	270	261	261
query61	109	128	146	128
query62	881	717	677	677
query63	231	199	202	199
query64	5105	774	749	749
query65	3405	3357	3362	3357
query66	1231	312	310	310
query67	16403	15660	15602	15602
query68	4782	586	611	586
query69	449	274	295	274
query70	1163	1141	1123	1123
query71	355	262	271	262
query72	6009	4070	3972	3972
query73	771	351	359	351
query74	10688	9264	8943	8943
query75	3357	2619	2651	2619
query76	2855	1151	1156	1151
query77	370	273	271	271
query78	10428	9498	9506	9498
query79	2654	627	627	627
query80	917	427	434	427
query81	544	223	220	220
query82	601	100	93	93
query83	250	148	147	147
query84	237	82	79	79
query85	1691	303	296	296
query86	478	292	290	290
query87	4389	4283	4281	4281
query88	4352	2451	2410	2410
query89	415	301	299	299
query90	1996	188	188	188
query91	185	150	152	150
query92	63	51	62	51
query93	2428	565	569	565
query94	829	310	278	278
query95	363	264	265	264
query96	606	287	280	280
query97	3285	3146	3136	3136
query98	231	200	198	198
query99	1545	1351	1301	1301
Total cold run time: 303889 ms
Total hot run time: 193723 ms

@doris-robot
Copy link

doris-robot commented Sep 1, 2025

BE UT Coverage Report

Increment line coverage 0.00% (0/1) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 42.08% (11242/26713)
Line Coverage 32.60% (96230/295152)
Region Coverage 30.54% (55248/180921)
Branch Coverage 26.86% (27348/101808)

@doris-robot
Copy link

doris-robot commented Sep 1, 2025

ClickBench: Total hot run time: 31.13 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b1ed101e59eff4c2cfba60e802cadae7071a6fa6, data reload: false

query1	0.03	0.03	0.03
query2	0.06	0.03	0.03
query3	0.23	0.06	0.06
query4	1.64	0.10	0.10
query5	0.52	0.51	0.49
query6	1.13	0.72	0.72
query7	0.02	0.01	0.01
query8	0.05	0.03	0.03
query9	0.58	0.51	0.50
query10	0.56	0.55	0.58
query11	0.14	0.10	0.10
query12	0.14	0.11	0.10
query13	0.61	0.59	0.59
query14	0.78	0.80	0.81
query15	0.85	0.82	0.82
query16	0.41	0.37	0.39
query17	0.95	1.04	1.06
query18	0.23	0.22	0.23
query19	1.95	1.74	1.79
query20	0.02	0.01	0.01
query21	15.41	0.58	0.58
query22	2.25	2.08	1.88
query23	16.90	0.86	0.64
query24	3.25	2.24	2.09
query25	0.13	0.05	0.07
query26	0.78	0.15	0.14
query27	0.04	0.05	0.04
query28	8.13	0.50	0.52
query29	12.58	3.21	3.20
query30	0.25	0.06	0.06
query31	2.87	0.40	0.39
query32	3.22	0.46	0.45
query33	3.03	2.99	3.03
query34	17.26	4.53	4.51
query35	4.56	4.53	4.61
query36	0.68	0.50	0.48
query37	0.09	0.06	0.05
query38	0.04	0.03	0.04
query39	0.03	0.02	0.03
query40	0.16	0.12	0.13
query41	0.08	0.02	0.03
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 102.72 s
Total hot run time: 31.13 s

dataroaring
dataroaring approved these changes Sep 1, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit 5052f90 into branch-3.0 Sep 1, 2025
23 of 25 checks passed
@github-actions github-actions bot deleted the auto-pick-55509-branch-3.0 branch September 1, 2025 11:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@Thearas @doris-robot @dataroaring @kaijchen