[fix](warmup) fix warm up jobs missing last batch #53860

kaijchen · 2025-07-24T11:54:16Z

What problem does this PR solve?

Issue Number: DORIS-21591

Problem Summary:

_pending_job_metas may be cleared by a CLEAR_JOB request, so we need to check it again.
We can not call pop_front before the job is finished, because GET_CURRENT_JOB_STATE_AND_LEASE is relying on the pending job size.

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

Thearas · 2025-07-24T11:54:23Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

kaijchen · 2025-07-24T11:54:25Z

run buildall

kaijchen · 2025-07-24T11:58:51Z

run buildall

freemandealer

LGTM

github-actions · 2025-07-24T12:03:13Z

PR approved by anyone and no changes requested.

github-actions · 2025-07-24T12:09:27Z

PR approved by at least one committer and no changes requested.

hello-stephen · 2025-07-24T13:50:36Z

BE UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	57.56% (15937/27689)
Line Coverage	46.35% (143280/309116)
Region Coverage	35.78% (107941/301714)
Branch Coverage	38.31% (47645/124380)

hello-stephen · 2025-07-24T14:14:58Z

BE Regression && UT Coverage Report

Increment line coverage 33.33% (2/6) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	81.14% (22055/27183)
Line Coverage	73.77% (227721/308696)
Region Coverage	61.39% (189997/309470)
Branch Coverage	65.17% (81984/125792)

### What problem does this PR solve? Issue Number: DORIS-21591 Related PR: #53860 Problem Summary: 1. `_pending_job_metas` may be cleared by a CLEAR_JOB request, so we need to check it again. 2. We can not call pop_front before the job is finished, because GET_CURRENT_JOB_STATE_AND_LEASE is relying on the pending job size. ### Release note None ### Check List (For Author) - Test  - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason  - Behavior changed: - [ ] No. - [ ] Yes.  - Does this need documentation? - [ ] No. - [ ] Yes.  ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label

hello-stephen · 2025-07-25T02:05:36Z

BE Regression && UT Coverage Report

Increment line coverage 33.33% (2/6) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	81.14% (22055/27183)
Line Coverage	73.77% (227721/308696)
Region Coverage	61.39% (189997/309470)
Branch Coverage	65.17% (81984/125792)

doris-robot · 2025-07-25T02:22:15Z

TPC-H: Total hot run time: 34104 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d1bdac9a454f7b5ee9ba611dfe880fee95d3d00e, data reload: false

------ Round 1 ----------------------------------
q1	17581	5239	5179	5179
q2	1921	274	173	173
q3	10341	1265	717	717
q4	10213	1017	516	516
q5	7519	2404	2333	2333
q6	171	163	132	132
q7	902	756	599	599
q8	9308	1306	1136	1136
q9	7024	5083	5137	5083
q10	6905	2389	1969	1969
q11	481	285	282	282
q12	346	355	223	223
q13	17764	3672	3090	3090
q14	242	229	207	207
q15	533	488	492	488
q16	429	423	373	373
q17	603	867	372	372
q18	7485	7093	7029	7029
q19	1241	941	586	586
q20	333	332	226	226
q21	3754	3197	2396	2396
q22	1071	1044	995	995
Total cold run time: 106167 ms
Total hot run time: 34104 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5155	5121	5244	5121
q2	235	320	213	213
q3	2153	2631	2309	2309
q4	1413	1783	1304	1304
q5	4222	4496	4441	4441
q6	214	168	133	133
q7	2070	1947	1804	1804
q8	2617	2576	2587	2576
q9	7312	7185	7374	7185
q10	3033	3343	2824	2824
q11	582	568	497	497
q12	701	796	641	641
q13	3579	3965	3438	3438
q14	279	291	280	280
q15	520	486	483	483
q16	456	507	449	449
q17	1227	1861	1315	1315
q18	7803	7957	7496	7496
q19	828	856	1003	856
q20	2019	2040	1928	1928
q21	4692	4372	4236	4236
q22	1078	1044	1009	1009
Total cold run time: 52188 ms
Total hot run time: 50538 ms

doris-robot · 2025-07-25T02:34:01Z

TPC-DS: Total hot run time: 188178 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d1bdac9a454f7b5ee9ba611dfe880fee95d3d00e, data reload: false

query1	976	409	390	390
query2	6536	1660	1678	1660
query3	6758	226	222	222
query4	26241	23558	23299	23299
query5	4310	637	473	473
query6	290	209	194	194
query7	4648	496	289	289
query8	264	220	203	203
query9	8623	2930	2941	2930
query10	470	330	301	301
query11	15644	14931	14917	14917
query12	171	118	122	118
query13	1653	570	444	444
query14	8660	6010	5926	5926
query15	209	186	173	173
query16	7356	647	453	453
query17	1220	737	610	610
query18	2007	405	325	325
query19	193	194	174	174
query20	130	123	114	114
query21	219	130	111	111
query22	4291	4369	4273	4273
query23	33927	32998	33090	32998
query24	8287	2425	2417	2417
query25	552	469	400	400
query26	1237	277	159	159
query27	2746	503	364	364
query28	4324	2234	2236	2234
query29	821	580	460	460
query30	285	222	194	194
query31	939	827	792	792
query32	82	74	75	74
query33	553	387	341	341
query34	809	835	526	526
query35	855	861	739	739
query36	990	1040	914	914
query37	130	106	86	86
query38	4209	4077	4013	4013
query39	1459	1420	1402	1402
query40	214	127	113	113
query41	61	56	54	54
query42	128	111	114	111
query43	496	484	476	476
query44	1406	858	854	854
query45	182	169	164	164
query46	845	1006	657	657
query47	1786	1793	1730	1730
query48	413	455	319	319
query49	740	490	415	415
query50	644	684	430	430
query51	5509	5556	5454	5454
query52	119	108	100	100
query53	232	284	197	197
query54	610	611	536	536
query55	90	85	82	82
query56	321	317	322	317
query57	1172	1183	1125	1125
query58	280	272	283	272
query59	2588	2593	2486	2486
query60	348	350	340	340
query61	134	127	160	127
query62	789	727	670	670
query63	232	193	198	193
query64	4338	1019	683	683
query65	4300	4232	4174	4174
query66	1171	416	349	349
query67	15740	15618	15366	15366
query68	8429	911	593	593
query69	476	329	297	297
query70	1281	1160	1181	1160
query71	452	340	321	321
query72	5532	4756	4831	4756
query73	743	604	359	359
query74	8938	9030	8948	8948
query75	3822	3155	2674	2674
query76	3611	1149	758	758
query77	798	402	321	321
query78	10003	10082	9210	9210
query79	2110	828	605	605
query80	635	532	493	493
query81	482	260	220	220
query82	450	138	108	108
query83	269	322	247	247
query84	245	109	82	82
query85	782	369	330	330
query86	396	311	295	295
query87	4387	4498	4303	4303
query88	3351	2263	2258	2258
query89	391	306	294	294
query90	1897	225	214	214
query91	134	142	110	110
query92	86	71	67	67
query93	1561	985	639	639
query94	678	392	274	274
query95	400	320	302	302
query96	487	578	280	280
query97	2703	2701	2645	2645
query98	236	219	208	208
query99	1417	1421	1301	1301
Total cold run time: 275144 ms
Total hot run time: 188178 ms

doris-robot · 2025-07-25T02:39:28Z

ClickBench: Total hot run time: 32.69 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d1bdac9a454f7b5ee9ba611dfe880fee95d3d00e, data reload: false

query1	0.05	0.03	0.03
query2	0.09	0.04	0.04
query3	0.24	0.08	0.07
query4	1.63	0.11	0.12
query5	0.43	0.40	0.40
query6	1.17	0.67	0.66
query7	0.03	0.02	0.02
query8	0.05	0.03	0.04
query9	0.61	0.52	0.51
query10	0.58	0.57	0.56
query11	0.15	0.11	0.11
query12	0.15	0.11	0.11
query13	0.62	0.61	0.61
query14	0.81	0.84	0.82
query15	0.89	0.85	0.88
query16	0.39	0.39	0.38
query17	1.08	1.04	1.06
query18	0.21	0.21	0.20
query19	1.94	1.82	1.84
query20	0.01	0.02	0.01
query21	15.44	0.90	0.52
query22	0.77	1.20	0.73
query23	14.86	1.35	0.58
query24	6.80	0.94	0.89
query25	0.51	0.11	0.11
query26	0.67	0.17	0.13
query27	0.07	0.06	0.06
query28	9.84	0.89	0.43
query29	12.58	4.11	3.44
query30	3.04	3.00	3.01
query31	2.80	0.57	0.38
query32	3.24	0.55	0.48
query33	3.06	3.05	3.06
query34	15.98	5.41	4.82
query35	4.86	4.85	4.88
query36	0.68	0.50	0.50
query37	0.09	0.07	0.07
query38	0.05	0.05	0.04
query39	0.03	0.03	0.03
query40	0.17	0.14	0.14
query41	0.08	0.03	0.02
query42	0.04	0.02	0.03
query43	0.04	0.03	0.02
Total cold run time: 106.83 s
Total hot run time: 32.69 s

…53864) picked from #53860

dataroaring · 2025-07-26T07:50:20Z

run check_coverage

hello-stephen · 2025-07-26T07:57:49Z

BE Regression && UT Coverage Report

Increment line coverage 33.33% (2/6) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	81.14% (22055/27183)
Line Coverage	73.77% (227721/308696)
Region Coverage	61.39% (189997/309470)
Branch Coverage	65.17% (81984/125792)

[fix](warmup) fix warm up jobs missing last batch

d1bdac9

kaijchen requested review from dataroaring and gavinchou as code owners July 24, 2025 11:54

This was referenced Jul 24, 2025

[fix](warmup) fix warm up jobs missing last batch (#53860) #53861

Merged

branch-3.1: [fix](warmup) fix warm up jobs missing last batch #53860 #53864

Merged

freemandealer approved these changes Jul 24, 2025

View reviewed changes

github-actions bot added the reviewed label Jul 24, 2025

gavinchou approved these changes Jul 24, 2025

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 24, 2025

dataroaring added the dev/3.0.7-merged label Jul 25, 2025

morrySnow pushed a commit that referenced this pull request Jul 25, 2025

branch-3.1: [fix](warmup) fix warm up jobs missing last batch #53860 (#…

7e7230c

…53864) picked from #53860

morrySnow added the dev/3.1.0-merged label Jul 25, 2025

dataroaring merged commit 4b817c6 into apache:master Jul 29, 2025
30 of 31 checks passed

gavinchou mentioned this pull request Aug 15, 2025

Release Note 3.0.7 #54868

Open

[fix](warmup) fix warm up jobs missing last batch #53860

[fix](warmup) fix warm up jobs missing last batch #53860

Uh oh!

Conversation

kaijchen commented Jul 24, 2025

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

Thearas commented Jul 24, 2025

Uh oh!

kaijchen commented Jul 24, 2025

Uh oh!

kaijchen commented Jul 24, 2025

Uh oh!

freemandealer left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Jul 24, 2025

Uh oh!

github-actions bot commented Jul 24, 2025

Uh oh!

hello-stephen commented Jul 24, 2025

BE UT Coverage Report

Uh oh!

hello-stephen commented Jul 24, 2025

BE Regression && UT Coverage Report

Uh oh!

hello-stephen commented Jul 25, 2025

BE Regression && UT Coverage Report

Uh oh!

doris-robot commented Jul 25, 2025

Uh oh!

doris-robot commented Jul 25, 2025

Uh oh!

doris-robot commented Jul 25, 2025

Uh oh!

dataroaring commented Jul 26, 2025

Uh oh!

hello-stephen commented Jul 26, 2025

BE Regression && UT Coverage Report

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants