Skip to content

[fix](memtable) add check when memtable init aggregate function to avoid core dump #57934

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 1 commit into from
Nov 12, 2025
Merged

[fix](memtable) add check when memtable init aggregate function to avoid core dump #57934

dataroaring merged 1 commit into from
Nov 12, 2025

Conversation

@sollhui
Copy link
Contributor

@sollhui sollhui commented Nov 12, 2025

What problem does this PR solve?

Sometimes planner will plan error and cause slot_desc less than _num_columns and cause be core dump, this pr add check when memtable init aggregate function to avoid core dump.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

hello-stephen commented Nov 12, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@sollhui
Copy link
Contributor Author

sollhui commented Nov 12, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Nov 12, 2025

TPC-H: Total hot run time: 34416 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c2f24cfa8c157b70809701ec3c86c45301c5acf0, data reload: false

------ Round 1 ----------------------------------
q1	17618	5247	5087	5087
q2	2034	329	215	215
q3	10216	1296	738	738
q4	10232	861	373	373
q5	7679	2382	2375	2375
q6	221	172	136	136
q7	927	803	633	633
q8	9377	1383	1121	1121
q9	7178	5175	5134	5134
q10	6922	2317	1836	1836
q11	500	300	280	280
q12	383	366	233	233
q13	17791	3710	3019	3019
q14	229	244	224	224
q15	595	510	503	503
q16	1022	1011	946	946
q17	598	883	357	357
q18	7809	7263	7132	7132
q19	1475	952	552	552
q20	360	347	228	228
q21	3877	2583	2320	2320
q22	1061	1064	974	974
Total cold run time: 108104 ms
Total hot run time: 34416 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5292	5143	5145	5143
q2	258	328	234	234
q3	2210	2672	2305	2305
q4	1328	1811	1296	1296
q5	4230	4516	4455	4455
q6	215	170	132	132
q7	2031	1997	1890	1890
q8	2631	2534	2507	2507
q9	7504	7412	7343	7343
q10	3064	3318	2832	2832
q11	602	529	505	505
q12	683	774	619	619
q13	3477	3923	3349	3349
q14	293	295	276	276
q15	563	495	528	495
q16	1096	1115	1102	1102
q17	1179	1601	1634	1601
q18	7683	7741	7804	7741
q19	796	840	906	840
q20	2046	2090	1909	1909
q21	4705	4290	4366	4290
q22	1081	1045	988	988
Total cold run time: 52967 ms
Total hot run time: 51852 ms

@doris-robot
Copy link

doris-robot commented Nov 12, 2025

TPC-DS: Total hot run time: 188309 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c2f24cfa8c157b70809701ec3c86c45301c5acf0, data reload: false

query1	1049	410	388	388
query2	6549	1718	1744	1718
query3	6765	230	231	230
query4	26888	23769	23261	23261
query5	4390	630	487	487
query6	340	245	240	240
query7	4653	495	300	300
query8	308	263	251	251
query9	8740	2631	2647	2631
query10	488	362	298	298
query11	15451	15207	14926	14926
query12	167	120	115	115
query13	1686	570	429	429
query14	10694	9236	9325	9236
query15	198	191	174	174
query16	7496	671	533	533
query17	1227	738	633	633
query18	2002	425	318	318
query19	208	204	170	170
query20	133	123	122	122
query21	218	130	112	112
query22	3910	4103	3878	3878
query23	33800	33083	32825	32825
query24	8543	2417	2423	2417
query25	606	524	457	457
query26	1228	273	161	161
query27	2764	500	343	343
query28	4406	2213	2217	2213
query29	825	616	486	486
query30	305	233	196	196
query31	915	837	727	727
query32	83	76	70	70
query33	594	386	340	340
query34	799	848	550	550
query35	841	841	741	741
query36	952	997	929	929
query37	121	103	89	89
query38	3492	3600	3519	3519
query39	1508	1436	1383	1383
query40	226	128	145	128
query41	61	62	60	60
query42	130	117	109	109
query43	502	528	488	488
query44	1240	754	742	742
query45	189	180	174	174
query46	891	1001	638	638
query47	1764	1766	1736	1736
query48	405	415	322	322
query49	781	527	416	416
query50	640	691	406	406
query51	4014	4120	3934	3934
query52	112	112	106	106
query53	243	272	203	203
query54	308	303	283	283
query55	87	85	86	85
query56	330	317	301	301
query57	1182	1207	1123	1123
query58	295	276	281	276
query59	2564	2649	2540	2540
query60	343	337	337	337
query61	159	162	164	162
query62	787	720	679	679
query63	237	197	193	193
query64	4519	1284	998	998
query65	4043	3973	3982	3973
query66	1165	462	360	360
query67	15348	14994	14871	14871
query68	5069	972	609	609
query69	532	339	300	300
query70	1334	1274	1233	1233
query71	407	347	330	330
query72	6259	5259	5048	5048
query73	679	620	368	368
query74	9199	8911	8822	8822
query75	3302	3336	2863	2863
query76	3224	1158	747	747
query77	524	431	314	314
query78	9667	9932	8876	8876
query79	2017	842	624	624
query80	730	583	507	507
query81	511	263	226	226
query82	250	162	132	132
query83	277	274	254	254
query84	260	120	94	94
query85	887	493	448	448
query86	387	299	311	299
query87	3744	3815	3615	3615
query88	2998	2303	2293	2293
query89	391	339	303	303
query90	1985	235	228	228
query91	170	177	141	141
query92	89	69	68	68
query93	2112	1025	646	646
query94	720	442	341	341
query95	421	328	324	324
query96	501	572	287	287
query97	2917	2981	2891	2891
query98	236	216	209	209
query99	1327	1417	1330	1330
Total cold run time: 271033 ms
Total hot run time: 188309 ms

@doris-robot
Copy link

doris-robot commented Nov 12, 2025

ClickBench: Total hot run time: 27.64 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c2f24cfa8c157b70809701ec3c86c45301c5acf0, data reload: false

query1	0.05	0.05	0.05
query2	0.09	0.05	0.05
query3	0.26	0.09	0.08
query4	1.60	0.12	0.12
query5	0.27	0.26	0.25
query6	1.18	0.65	0.64
query7	0.03	0.02	0.03
query8	0.05	0.05	0.04
query9	0.60	0.51	0.52
query10	0.58	0.57	0.58
query11	0.16	0.12	0.12
query12	0.15	0.12	0.12
query13	0.63	0.62	0.62
query14	1.02	1.01	1.03
query15	0.86	0.84	0.83
query16	0.39	0.40	0.41
query17	1.06	1.02	1.07
query18	0.22	0.20	0.21
query19	1.94	1.82	1.75
query20	0.02	0.01	0.01
query21	15.45	0.21	0.13
query22	5.04	0.07	0.04
query23	15.66	0.26	0.11
query24	2.26	0.54	0.93
query25	0.07	0.08	0.06
query26	0.15	0.15	0.13
query27	0.08	0.06	0.06
query28	4.33	1.14	0.94
query29	12.55	3.93	3.21
query30	0.29	0.14	0.14
query31	2.82	0.60	0.39
query32	3.23	0.55	0.46
query33	3.05	3.04	3.09
query34	15.87	5.19	4.57
query35	4.56	4.65	4.55
query36	0.67	0.52	0.49
query37	0.09	0.07	0.07
query38	0.06	0.04	0.04
query39	0.03	0.03	0.03
query40	0.17	0.15	0.14
query41	0.09	0.03	0.03
query42	0.04	0.03	0.03
query43	0.04	0.04	0.03
Total cold run time: 97.76 s
Total hot run time: 27.64 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Nov 12, 2025

BE UT Coverage Report

Increment line coverage 25.00% (1/4) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 52.81% (18279/34611)
Line Coverage 38.17% (166146/435275)
Region Coverage 33.16% (129156/389453)
Branch Coverage 33.90% (55420/163492)

@hello-stephen
Copy link
Contributor

hello-stephen commented Nov 12, 2025

BE Regression && UT Coverage Report

Increment line coverage 25.00% (1/4) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 71.53% (24327/34011)
Line Coverage 58.00% (252831/435940)
Region Coverage 53.40% (210902/394967)
Branch Coverage 54.62% (89879/164568)

@hello-stephen
Copy link
Contributor

hello-stephen commented Nov 12, 2025

BE Regression && UT Coverage Report

Increment line coverage 25.00% (1/4) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 71.53% (24329/34011)
Line Coverage 58.01% (252895/435940)
Region Coverage 53.41% (210960/394967)
Branch Coverage 54.64% (89917/164568)

liaoxin01
liaoxin01 approved these changes Nov 12, 2025
View reviewed changes
Copy link
Contributor

@liaoxin01 liaoxin01 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 12, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Nov 12, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Nov 12, 2025

PR approved by anyone and no changes requested.

dataroaring
dataroaring approved these changes Nov 12, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit 996a500 into apache:master Nov 12, 2025
30 of 31 checks passed
github-actions bot pushed a commit that referenced this pull request Nov 12, 2025
@sollhui
[fix](memtable) add check when memtable init aggregate function to av…
f0704cc 
…oid core dump (#57934)

### What problem does this PR solve?

Sometimes planner will plan error and cause `slot_desc` less than
`_num_columns` and cause be core dump, this pr add check when memtable
init aggregate function to avoid core dump.

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
@github-actions github-actions bot mentioned this pull request Nov 12, 2025
Merged
yiguolei pushed a commit that referenced this pull request Nov 13, 2025
@github-actions @sollhui
branch-4.0: [fix](memtable) add check when memtable init aggregate fu…
e15512d 
…nction to avoid core dump #57934 (#57969)

Cherry-picked from #57934

Co-authored-by: hui lai <laihui@selectdb.com>
wyxxxcat pushed a commit to wyxxxcat/doris that referenced this pull request Nov 13, 2025
@sollhui @wyxxxcat
[fix](memtable) add check when memtable init aggregate function to av…
304f147 
…oid core dump (apache#57934)

### What problem does this PR solve?

Sometimes planner will plan error and cause `slot_desc` less than
`_num_columns` and cause be core dump, this pr add check when memtable
init aggregate function to avoid core dump.

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
@sollhui sollhui mentioned this pull request Nov 13, 2025
Merged
16 tasks
wyxxxcat pushed a commit to wyxxxcat/doris that referenced this pull request Nov 18, 2025
@sollhui @wyxxxcat
[fix](memtable) add check when memtable init aggregate function to av…
76794df 
…oid core dump (apache#57934)

### What problem does this PR solve?

Sometimes planner will plan error and cause `slot_desc` less than
`_num_columns` and cause be core dump, this pr add check when memtable
init aggregate function to avoid core dump.

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
liaoxin01 pushed a commit that referenced this pull request Nov 26, 2025
@sollhui
[fix](partial update) fix partial update always failed after create r…
6122098 
…ollup/MV (#58003)

### What problem does this PR solve?

Create a rollup:
```
ALTER TABLE mow_table ADD ROLLUP rollup1(event_date, event_time, user_id, country, update_time)
```

update table after create rollup with column(city) do not exist in
rollup:
```
UPDATE mow_table SET city = "beijing" WHERE user_id = 2000
```

BE node will core dump(#57934 avoid
core dump, but still always fail)
github-actions bot pushed a commit that referenced this pull request Nov 26, 2025
@sollhui
[fix](partial update) fix partial update always failed after create r…
d18a6c9 
…ollup/MV (#58003)

### What problem does this PR solve?

Create a rollup:
```
ALTER TABLE mow_table ADD ROLLUP rollup1(event_date, event_time, user_id, country, update_time)
```

update table after create rollup with column(city) do not exist in
rollup:
```
UPDATE mow_table SET city = "beijing" WHERE user_id = 2000
```

BE node will core dump(#57934 avoid
core dump, but still always fail)
sollhui added a commit to sollhui/doris that referenced this pull request Dec 1, 2025
@sollhui
[fix](partial update) fix partial update always failed after create r…
03af7ef 
…ollup/MV (apache#58003)

Create a rollup:
```
ALTER TABLE mow_table ADD ROLLUP rollup1(event_date, event_time, user_id, country, update_time)
```

update table after create rollup with column(city) do not exist in
rollup:
```
UPDATE mow_table SET city = "beijing" WHERE user_id = 2000
```

BE node will core dump(apache#57934 avoid
core dump, but still always fail)
sollhui added a commit to sollhui/doris that referenced this pull request Dec 1, 2025
@sollhui
[fix](partial update) fix partial update always failed after create r…
959c2cb 
…ollup/MV (apache#58003)

Create a rollup:
```
ALTER TABLE mow_table ADD ROLLUP rollup1(event_date, event_time, user_id, country, update_time)
```

update table after create rollup with column(city) do not exist in
rollup:
```
UPDATE mow_table SET city = "beijing" WHERE user_id = 2000
```

BE node will core dump(apache#57934 avoid
core dump, but still always fail)
@sollhui sollhui mentioned this pull request Dec 1, 2025
Merged
sollhui added a commit to sollhui/doris that referenced this pull request Dec 1, 2025
@sollhui
[fix](memtable) add check when memtable init aggregate function to av…
57e687e 
…oid core dump (apache#57934)

Sometimes planner will plan error and cause `slot_desc` less than
`_num_columns` and cause be core dump, this pr add check when memtable
init aggregate function to avoid core dump.

None

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
nagisa-kunhah pushed a commit to nagisa-kunhah/doris that referenced this pull request Dec 14, 2025
@sollhui @nagisa-kunhah
[fix](partial update) fix partial update always failed after create r…
a7deb50 
…ollup/MV (apache#58003)

### What problem does this PR solve?

Create a rollup:
```
ALTER TABLE mow_table ADD ROLLUP rollup1(event_date, event_time, user_id, country, update_time)
```

update table after create rollup with column(city) do not exist in
rollup:
```
UPDATE mow_table SET city = "beijing" WHERE user_id = 2000
```

BE node will core dump(apache#57934 avoid
core dump, but still always fail)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@liaoxin01 liaoxin01 liaoxin01 approved these changes

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/4.0.2-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@sollhui @hello-stephen @doris-robot @liaoxin01 @dataroaring @yiguolei @morrySnow