Skip to content

[feat](es) support es flatten type #58762

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morningman merged 2 commits into from
Dec 8, 2025
Merged

[feat](es) support es flatten type #58762

morningman merged 2 commits into from
Dec 8, 2025

Conversation

@morningman
Copy link
Contributor

@morningman morningman commented Dec 5, 2025 •
edited
Loading

What problem does this PR solve?

Support ES flatten type, mapping to json

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@morningman
Copy link
Contributor Author

morningman commented Dec 5, 2025

run buildall

suxiaogang223
suxiaogang223 approved these changes Dec 5, 2025
View reviewed changes
Copy link
Contributor

@suxiaogang223 suxiaogang223 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Contributor

github-actions bot commented Dec 5, 2025

PR approved by anyone and no changes requested.

@morningman
Copy link
Contributor Author

morningman commented Dec 5, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Dec 5, 2025

TPC-H: Total hot run time: 34222 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit cd460683cab04adf42c265891e4bac4e48b65fa1, data reload: false

------ Round 1 ----------------------------------
q1	17643	5059	4905	4905
q2	2063	308	189	189
q3	10269	1302	741	741
q4	10244	909	316	316
q5	7572	2412	2135	2135
q6	197	170	139	139
q7	947	812	631	631
q8	9349	1384	1161	1161
q9	6980	5275	5343	5275
q10	6837	2217	1775	1775
q11	545	311	287	287
q12	342	379	246	246
q13	17770	3707	3012	3012
q14	231	241	219	219
q15	592	519	511	511
q16	893	871	803	803
q17	713	832	471	471
q18	7764	7349	7155	7155
q19	1105	939	607	607
q20	380	348	228	228
q21	3940	3134	2467	2467
q22	1034	1008	949	949
Total cold run time: 107410 ms
Total hot run time: 34222 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4910	4903	5015	4903
q2	334	393	307	307
q3	2141	2683	2265	2265
q4	1331	1753	1309	1309
q5	4231	4422	4578	4422
q6	223	187	137	137
q7	2076	1868	1867	1867
q8	2713	2556	2543	2543
q9	7496	7554	7466	7466
q10	3047	3289	2920	2920
q11	596	525	507	507
q12	768	890	593	593
q13	3426	4057	3329	3329
q14	294	326	295	295
q15	565	609	576	576
q16	971	927	859	859
q17	1169	1522	1390	1390
q18	7786	7739	7465	7465
q19	883	859	872	859
q20	2005	1948	1840	1840
q21	4576	4261	4205	4205
q22	1056	1030	1015	1015
Total cold run time: 52597 ms
Total hot run time: 51072 ms

@doris-robot
Copy link

doris-robot commented Dec 5, 2025

TPC-DS: Total hot run time: 179967 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit cd460683cab04adf42c265891e4bac4e48b65fa1, data reload: false

query5	4788	671	502	502
query6	354	231	212	212
query7	4664	481	277	277
query8	308	243	239	239
query9	8769	2641	2665	2641
query10	514	315	282	282
query11	15273	14801	14533	14533
query12	178	117	115	115
query13	1683	491	381	381
query14	6092	3283	3006	3006
query14_1	2928	2862	2971	2862
query15	214	195	183	183
query16	7529	490	450	450
query17	1235	720	622	622
query18	2031	444	345	345
query19	212	190	171	171
query20	134	123	119	119
query21	224	142	117	117
query22	4127	4201	3982	3982
query23	16545	16153	16022	16022
query23_1	16027	15948	16012	15948
query24	7238	1617	1227	1227
query24_1	1239	1244	1241	1241
query25	673	516	461	461
query26	1263	290	172	172
query27	2891	473	317	317
query28	4394	2177	2166	2166
query29	879	597	478	478
query30	325	241	217	217
query31	859	711	648	648
query32	90	72	70	70
query33	675	363	310	310
query34	905	872	538	538
query35	784	828	725	725
query36	890	908	835	835
query37	123	95	82	82
query38	3843	3829	3745	3745
query39	770	733	703	703
query39_1	698	697	686	686
query40	228	136	118	118
query41	67	63	65	63
query42	126	94	97	94
query43	442	415	403	403
query44	1308	776	767	767
query45	194	198	177	177
query46	907	960	592	592
query47	1659	1688	1634	1634
query48	397	316	235	235
query49	791	460	356	356
query50	693	316	237	237
query51	3903	3854	3926	3854
query52	122	96	86	86
query53	240	231	177	177
query54	324	252	238	238
query55	95	79	75	75
query56	336	284	299	284
query57	1167	1138	1105	1105
query58	288	298	257	257
query59	2282	2430	2308	2308
query60	350	311	311	311
query61	165	156	159	156
query62	772	667	660	660
query63	228	179	181	179
query64	4560	1170	894	894
query65	4035	3966	3951	3951
query66	1227	444	334	334
query67	15193	14918	15006	14918
query68	8587	945	666	666
query69	514	298	266	266
query70	1094	1006	1014	1006
query71	462	292	272	272
query72	5950	4939	4880	4880
query73	669	553	299	299
query74	8804	8763	8770	8763
query75	3278	3024	2498	2498
query76	3436	1143	751	751
query77	591	406	312	312
query78	9433	9717	8844	8844
query79	1798	849	599	599
query80	711	538	472	472
query81	512	270	237	237
query82	196	128	106	106
query83	281	273	253	253
query84	261	118	106	106
query85	918	504	453	453
query86	382	307	282	282
query87	4091	4077	3977	3977
query88	3629	2131	2150	2131
query89	383	318	285	285
query90	2120	164	156	156
query91	171	171	135	135
query92	89	68	66	66
query93	2100	1027	685	685
query94	757	316	282	282
query95	560	381	321	321
query96	527	508	218	218
query97	2601	2672	2624	2624
query98	248	201	204	201
query99	1358	1320	1257	1257
Total cold run time: 270772 ms
Total hot run time: 179967 ms

@doris-robot
Copy link

doris-robot commented Dec 5, 2025

ClickBench: Total hot run time: 27.38 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit cd460683cab04adf42c265891e4bac4e48b65fa1, data reload: false

query1	0.05	0.05	0.05
query2	0.10	0.05	0.05
query3	0.25	0.08	0.08
query4	1.61	0.11	0.12
query5	0.28	0.24	0.28
query6	1.16	0.64	0.63
query7	0.03	0.02	0.03
query8	0.06	0.05	0.04
query9	0.58	0.52	0.50
query10	0.57	0.56	0.56
query11	0.16	0.10	0.11
query12	0.16	0.12	0.12
query13	0.62	0.60	0.61
query14	1.01	0.98	0.98
query15	0.81	0.79	0.81
query16	0.38	0.42	0.40
query17	1.01	0.99	1.02
query18	0.24	0.21	0.21
query19	1.89	1.81	1.83
query20	0.02	0.02	0.01
query21	15.45	0.30	0.15
query22	4.90	0.05	0.05
query23	15.90	0.27	0.11
query24	1.53	0.66	0.37
query25	0.08	0.09	0.06
query26	0.14	0.12	0.14
query27	0.05	0.05	0.06
query28	4.94	1.23	1.01
query29	12.60	4.03	3.22
query30	0.30	0.14	0.12
query31	2.82	0.63	0.40
query32	3.23	0.56	0.46
query33	3.07	3.19	3.06
query34	16.79	5.15	4.53
query35	4.58	4.59	4.54
query36	0.69	0.51	0.49
query37	0.13	0.07	0.06
query38	0.08	0.05	0.03
query39	0.05	0.03	0.03
query40	0.17	0.13	0.14
query41	0.08	0.03	0.03
query42	0.04	0.04	0.03
query43	0.04	0.04	0.04
Total cold run time: 98.65 s
Total hot run time: 27.38 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Dec 5, 2025

FE Regression Coverage Report

Increment line coverage 100% (0/0) 🎉
Increment coverage report
Complete coverage report

@hello-stephen
Copy link
Contributor

hello-stephen commented Dec 5, 2025

FE UT Coverage Report

Increment line coverage 100% (0/0) 🎉
Increment coverage report
Complete coverage report

@morningman morningman merged commit c806e15 into apache:master Dec 8, 2025
27 of 28 checks passed
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 8, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Dec 8, 2025

PR approved by at least one committer and no changes requested.

github-actions bot pushed a commit that referenced this pull request Dec 8, 2025
@morningman
[feat](es) support es flatten type (#58762)
074b20c 
### What problem does this PR solve?

Support ES flatten type, mapping to json
@github-actions github-actions bot mentioned this pull request Dec 8, 2025
Merged
github-actions bot pushed a commit that referenced this pull request Dec 8, 2025
@morningman
[feat](es) support es flatten type (#58762)
3373d57 
### What problem does this PR solve?

Support ES flatten type, mapping to json
@github-actions github-actions bot mentioned this pull request Dec 8, 2025
Merged
morrySnow pushed a commit that referenced this pull request Dec 9, 2025
@github-actions @morningman
branch-3.1: [feat](es) support es flatten type #58762 (#58793)
c6b7311 
Cherry-picked from #58762

Co-authored-by: Mingyu Chen (Rayner) <morningman@163.com>
yiguolei pushed a commit that referenced this pull request Dec 12, 2025
@github-actions @morningman
branch-4.0: [feat](es) support es flatten type #58762 (#58794)
c11e585 
Cherry-picked from #58762

Co-authored-by: Mingyu Chen (Rayner) <morningman@163.com>
nagisa-kunhah pushed a commit to nagisa-kunhah/doris that referenced this pull request Dec 14, 2025
@morningman @nagisa-kunhah
[feat](es) support es flatten type (apache#58762)
5240d21 
### What problem does this PR solve?

Support ES flatten type, mapping to json
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@zy-kkk zy-kkk zy-kkk approved these changes

+1 more reviewer

@suxiaogang223 suxiaogang223 suxiaogang223 approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.1.4-merged dev/4.0.3-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@morningman @doris-robot @hello-stephen @suxiaogang223 @zy-kkk @yiguolei