Skip to content

branch-3.1: [feature](load) new insgestion load #45937 #53582

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morrySnow merged 3 commits into from
Jul 21, 2025
Merged

branch-3.1: [feature](load) new insgestion load #45937 #53582

morrySnow merged 3 commits into from
Jul 21, 2025

Conversation

@morningman
Copy link
Contributor

@morningman morningman commented Jul 19, 2025

bp #45937

@gnehil @morningman
[feature](load) new insgestion load (apache#45937)
7b3102a 
Problem Summary:

Ingestion Load is used to load pre-processed data into doris.

Preprocessing refers to writing the result data to an external storage
system after the data is processed according to the partitioning,
bucketing and aggregation methods defined by the doris table.

The preprocessing is completed by the external system, and then the BE
reads the data and converts it into segment files and saves it.

The basic flow is as follows:

![ingestion_load](https://github.com/apache/doris/assets/30104232/aa468cd4-90bf-4d9d-b69b-0425b66b15f4)

[feature](load) new insgestion load
@morningman morningman requested a review from morrySnow as a code owner July 19, 2025 05:49
@Thearas
Copy link
Contributor

Thearas commented Jul 19, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@morningman
Copy link
Contributor Author

morningman commented Jul 19, 2025

run buildall

@morningman
Copy link
Contributor Author

morningman commented Jul 19, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jul 19, 2025

TPC-H: Total hot run time: 40363 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a0d1b6d397516c57762476ac5e9d4207a883d81d, data reload: false

------ Round 1 ----------------------------------
q1	17619	6760	6627	6627
q2	2076	205	199	199
q3	10473	1139	1157	1139
q4	10233	763	755	755
q5	7735	2891	2902	2891
q6	240	145	148	145
q7	1034	651	652	651
q8	9406	1972	2047	1972
q9	6694	6444	6459	6444
q10	7073	2259	2326	2259
q11	490	276	282	276
q12	474	243	243	243
q13	17817	3013	3017	3013
q14	256	223	225	223
q15	555	519	540	519
q16	497	399	393	393
q17	989	578	577	577
q18	7250	6663	6630	6630
q19	1351	1126	1081	1081
q20	495	222	226	222
q21	3918	3108	3152	3108
q22	1083	996	1000	996
Total cold run time: 107758 ms
Total hot run time: 40363 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6621	6666	6623	6623
q2	338	252	243	243
q3	2963	3002	2957	2957
q4	2040	1905	1853	1853
q5	5787	5799	5791	5791
q6	220	143	142	142
q7	2248	1885	1854	1854
q8	3403	3570	3515	3515
q9	8960	8821	8867	8821
q10	3621	3577	3527	3527
q11	601	512	495	495
q12	812	603	611	603
q13	13531	3237	3167	3167
q14	306	267	289	267
q15	556	493	526	493
q16	505	462	469	462
q17	1856	1618	1622	1618
q18	8134	7774	7772	7772
q19	1811	1526	1620	1526
q20	2152	1909	1887	1887
q21	5372	5113	4994	4994
q22	1132	1043	1014	1014
Total cold run time: 72969 ms
Total hot run time: 59624 ms

@doris-robot
Copy link

doris-robot commented Jul 19, 2025

BE UT Coverage Report

Increment line coverage 0.00% (0/36) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 45.26% (12552/27735)
Line Coverage 36.15% (111673/308912)
Region Coverage 35.22% (57746/163935)
Branch Coverage 32.42% (31395/96844)

@hello-stephen
Copy link
Contributor

hello-stephen commented Jul 19, 2025

FE UT Coverage Report

Increment line coverage 1.71% (12/702) 🎉
Increment coverage report
Complete coverage report

@hello-stephen
Copy link
Contributor

hello-stephen commented Jul 19, 2025

BE Regression && UT Coverage Report

Increment line coverage 0.00% (0/34) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 76.20% (20881/27402)
Line Coverage 69.57% (214742/308675)
Region Coverage 67.63% (128449/189943)
Branch Coverage 61.25% (66900/109226)

@morningman morningman mentioned this pull request Jul 20, 2025
Closed
@morrySnow morrySnow changed the title branch-3.1: [feature](load) new insgestion load (#45937) branch-3.1: [feature](load) new insgestion load #45937 Jul 21, 2025
@morrySnow morrySnow merged commit cff2c52 into apache:branch-3.1 Jul 21, 2025
21 of 23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@morningman @Thearas @doris-robot @hello-stephen @morrySnow @gnehil