[Enhancement] Stream load/routine support load channel profile #55490

banmoy · 2025-01-27T05:35:53Z

Why I'm doing:

The previous pr #41664 adds load channel profile, but it only supports broker load/insert which use pipeline engine. stream load/routine load are still using non-pipeline engine, so we can't get load channel profile for stream load/routine load

What I'm doing:

support to get load channel profile for stream load/routine

Fixes #issue

What type of PR is this:

Does this PR entail a change in behavior?

Yes, this PR will result in a change in behavior.
No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

Interface/UI changes: syntax, type conversion, expression evaluation, display information
Parameter changes: default values, similar parameters but with different default values
Policy changes: use new policy to replace old one, functionality automatically enabled
Feature removed
Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

I have added test cases for my bug fix or my new feature
This pr needs user documentation (for new or modified features or behaviors)
- I have added documentation for my new feature or new function
This is a backport pr

Bugfix cherry-pick branch check:

be/src/runtime/fragment_mgr.cpp

be/src/exec/tablet_sink.cpp

fe/fe-core/src/main/java/com/starrocks/qe/scheduler/QueryRuntimeProfile.java

fe/fe-core/src/main/java/com/starrocks/load/streamload/StreamLoadTask.java

meegoo · 2025-02-11T02:29:43Z

be/src/exec/tablet_sink.cpp

+        if (query_options.__isset.enable_profile) {
+            _load_channel_profile_config.set_enable_profile(query_options.enable_profile);
+        }
+        if (query_options.__isset.load_profile_collect_second) {


load_profile_collect_second is different from big_query_profile_threshold_ns.

Signed-off-by: PengFei Li <lpengfei2016@gmail.com>

sonarqubecloud · 2025-02-11T16:53:40Z

Quality Gate passed

Issues
4 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

github-actions · 2025-02-11T18:28:04Z

[Java-Extensions Incremental Coverage Report]

✅ pass : 0 / 0 (0%)

github-actions · 2025-02-11T18:31:10Z

[FE Incremental Coverage Report]

✅ pass : 19 / 21 (90.48%)

file detail

	path	covered_line	new_line	coverage	not_covered_line_detail
🔵	com/starrocks/load/streamload/StreamLoadTask.java	0	1	00.00%	[1109]
🔵	com/starrocks/qe/scheduler/QueryRuntimeProfile.java	14	15	93.33%	[628]
🔵	com/starrocks/qe/scheduler/dag/JobSpec.java	5	5	100.00%	[]

github-actions · 2025-02-11T18:39:40Z

[BE Incremental Coverage Report]

✅ pass : 22 / 22 (100.00%)

file detail

	path	covered_line	new_line	coverage	not_covered_line_detail
🔵	be/src/exec/tablet_sink.cpp	15	15	100.00%	[]
🔵	be/src/runtime/plan_fragment_executor.cpp	1	1	100.00%	[]
🔵	be/src/runtime/fragment_mgr.cpp	6	6	100.00%	[]

github-actions · 2025-02-12T02:52:10Z

@Mergifyio backport branch-3.4

mergify · 2025-02-12T02:52:23Z

backport branch-3.4

✅ Backports have been created

#55822 [Enhancement] Stream load/routine support load channel profile (backport #55490) has been created for branch branch-3.4

Signed-off-by: PengFei Li <lpengfei2016@gmail.com> (cherry picked from commit ca8ee31)

…ort #55490) (#55822) Co-authored-by: PengFei Li <lpengfei2016@gmail.com>

…ocks#55490) Signed-off-by: PengFei Li <lpengfei2016@gmail.com>

…ocks#55490) Signed-off-by: PengFei Li <lpengfei2016@gmail.com> Signed-off-by: 00193254 <2513643704@qq.com>

banmoy requested review from a team as code owners January 27, 2025 05:35

mergify bot assigned banmoy Jan 27, 2025

starrocks-cr bot reviewed Jan 27, 2025

View reviewed changes

be/src/runtime/fragment_mgr.cpp Outdated Show resolved Hide resolved

starrocks-cr bot reviewed Jan 27, 2025

View reviewed changes

be/src/exec/tablet_sink.cpp Show resolved Hide resolved

starrocks-cr bot reviewed Jan 27, 2025

View reviewed changes

fe/fe-core/src/main/java/com/starrocks/qe/scheduler/QueryRuntimeProfile.java Show resolved Hide resolved

github-actions bot added the 3.4 label Jan 27, 2025

banmoy changed the title ~~[Enhancement] stream load/routine load supports load channel profile~~ [Enhancement] support to get load channel profile for stream load/routine Jan 27, 2025

banmoy changed the title ~~[Enhancement] support to get load channel profile for stream load/routine~~ [Enhancement] Support load channel profile for stream load/routine Jan 27, 2025

banmoy changed the title ~~[Enhancement] Support load channel profile for stream load/routine~~ [Enhancement] Stream load/routine support load channel profile Jan 27, 2025

banmoy force-pushed the stream_load_profile branch 2 times, most recently from ca6182e to ef64f68 Compare January 27, 2025 11:09

wyb reviewed Feb 10, 2025

View reviewed changes

fe/fe-core/src/main/java/com/starrocks/load/streamload/StreamLoadTask.java Outdated Show resolved Hide resolved

banmoy force-pushed the stream_load_profile branch from ef64f68 to 980ab11 Compare February 10, 2025 05:07

Seaven previously approved these changes Feb 10, 2025

View reviewed changes

banmoy force-pushed the stream_load_profile branch from 980ab11 to cbb9c29 Compare February 10, 2025 12:52

meegoo requested changes Feb 11, 2025

View reviewed changes

banmoy added 4 commits February 11, 2025 16:02

stream load/routine load supports load channel profile

2a6ba80

Signed-off-by: PengFei Li <lpengfei2016@gmail.com>

Add test

b6b0acc

Signed-off-by: PengFei Li <lpengfei2016@gmail.com>

fix comment

e08a7ac

Signed-off-by: PengFei Li <lpengfei2016@gmail.com>

fix profile options

323d386

Signed-off-by: PengFei Li <lpengfei2016@gmail.com>

banmoy dismissed Seaven’s stale review via 323d386 February 11, 2025 08:03

banmoy force-pushed the stream_load_profile branch from cbb9c29 to 323d386 Compare February 11, 2025 08:03

banmoy added 2 commits February 11, 2025 19:40

add tests

9b798f1

Signed-off-by: PengFei Li <lpengfei2016@gmail.com>

fix test

1d70ffa

Signed-off-by: PengFei Li <lpengfei2016@gmail.com>

banmoy force-pushed the stream_load_profile branch from 8034ab2 to 1d70ffa Compare February 11, 2025 13:33

add test coverage

599a03c

Signed-off-by: PengFei Li <lpengfei2016@gmail.com>

wyb approved these changes Feb 12, 2025

View reviewed changes

Seaven approved these changes Feb 12, 2025

View reviewed changes

mervinsr approved these changes Feb 12, 2025

View reviewed changes

meegoo approved these changes Feb 12, 2025

View reviewed changes

meegoo merged commit ca8ee31 into StarRocks:main Feb 12, 2025
56 checks passed

github-actions bot removed the 3.4 label Feb 12, 2025

mergify bot pushed a commit that referenced this pull request Feb 12, 2025

[Enhancement] Stream load/routine support load channel profile (#55490)

45a4174

Signed-off-by: PengFei Li <lpengfei2016@gmail.com> (cherry picked from commit ca8ee31)

mergify bot mentioned this pull request Feb 12, 2025

[Enhancement] Stream load/routine support load channel profile (backport #55490) #55822

Merged

18 tasks

wanpengfei-git pushed a commit that referenced this pull request Feb 12, 2025

[Enhancement] Stream load/routine support load channel profile (backp…

00f36d2

…ort #55490) (#55822) Co-authored-by: PengFei Li <lpengfei2016@gmail.com>

github-actions bot added the 3.4-merged label Feb 12, 2025

liaowanyou pushed a commit to liaowanyou/starrocks that referenced this pull request Feb 17, 2025

[Enhancement] Stream load/routine support load channel profile (StarR…

5198366

…ocks#55490) Signed-off-by: PengFei Li <lpengfei2016@gmail.com>

dyp12 pushed a commit to dyp12/starrocks that referenced this pull request Mar 5, 2025

[Enhancement] Stream load/routine support load channel profile (StarR…

0a18d51

…ocks#55490) Signed-off-by: PengFei Li <lpengfei2016@gmail.com> Signed-off-by: 00193254 <2513643704@qq.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[Enhancement] Stream load/routine support load channel profile #55490

[Enhancement] Stream load/routine support load channel profile #55490

Uh oh!

banmoy commented Jan 27, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

meegoo Feb 11, 2025

Uh oh!

banmoy Feb 11, 2025

Uh oh!

sonarqubecloud bot commented Feb 11, 2025

Uh oh!

github-actions bot commented Feb 11, 2025

Uh oh!

github-actions bot commented Feb 11, 2025

Uh oh!

github-actions bot commented Feb 11, 2025

Uh oh!

Uh oh!

github-actions bot commented Feb 12, 2025

Uh oh!

mergify bot commented Feb 12, 2025 •

edited

Loading

Uh oh!

Uh oh!

[Enhancement] Stream load/routine support load channel profile #55490

[Enhancement] Stream load/routine support load channel profile #55490

Uh oh!

Conversation

banmoy commented Jan 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why I'm doing:

What I'm doing:

What type of PR is this:

Checklist:

Bugfix cherry-pick branch check:

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

meegoo Feb 11, 2025

Choose a reason for hiding this comment

Uh oh!

banmoy Feb 11, 2025

Choose a reason for hiding this comment

Uh oh!

sonarqubecloud bot commented Feb 11, 2025

Quality Gate passed

Uh oh!

github-actions bot commented Feb 11, 2025

[Java-Extensions Incremental Coverage Report]

Uh oh!

github-actions bot commented Feb 11, 2025

[FE Incremental Coverage Report]

file detail

Uh oh!

github-actions bot commented Feb 11, 2025

[BE Incremental Coverage Report]

file detail

Uh oh!

Uh oh!

github-actions bot commented Feb 12, 2025

Uh oh!

mergify bot commented Feb 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Backports have been created

Uh oh!

Uh oh!

banmoy commented Jan 27, 2025 •

edited

Loading

mergify bot commented Feb 12, 2025 •

edited

Loading