feat: add time series statistics profile #17809

dqhl76 · 2025-04-18T10:31:30Z

I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Summary

Add time series statistics profile. Iniaitily, we support output_rows and output_bytes. This allows us to visualize and compare the data processing speed of different plan nodes.

{"query_id":"ed83c41f-1307-4b13-bd77-710e5e3128c6","plan_id":2,"stats_name":"output_rows","stats":[[1744971865000,69992448],[1744971866000,77529088],[1744971867000,86245376],[1744971868000,87425024],[1744971869000,97648640],[1744971870000,95813632],[1744971871000,105775104],[1744971872000,110493696],[1744971873000,113377280]]}

{"query_id":"ed83c41f-1307-4b13-bd77-710e5e3128c6","plan_id":2,"stats_name":"output_bytes","stats":[[1744971865000,559939584],[1744971866000,620232704],[1744971867000,689963008],[1744971868000,699400192],[1744971869000,781189120],[1744971870000,766509056],[1744971871000,846200832],[1744971872000,883949568],[1744971873000,907018240]]}

{"query_id":"ed83c41f-1307-4b13-bd77-710e5e3128c6","plan_id":2,"stats_name":"output_bytes","stats":[[1744971874000,953155584]]}

{"query_id":"ed83c41f-1307-4b13-bd77-710e5e3128c6","plan_id":2,"stats_name":"output_rows","stats":[[1744971874000,119144448]]}

Tests

Unit Test
Logic Test
Benchmark Test
No Test - Explain why

Type of change

Bug Fix (non-breaking change which fixes an issue)
New Feature (non-breaking change which adds functionality)
Breaking Change (fix or feature that could cause existing functionality not to work as expected)
Documentation Update
Refactoring
Performance Improvement
Other (please describe):

This change is

github-actions · 2025-04-21T05:07:15Z

Docker Image for PR

tag: pr-17809-bf4ac89-1745211953

note: this image tag is only available for internal use.

github-actions · 2025-04-21T05:50:33Z

ClickBench Report

dqhl76 · 2025-04-21T06:31:41Z

Disabled in log by default. Enable by adding databend::log::time_series=INFO

[log.file]
level = "INFO,databend::log::time_series=INFO"

src/common/base/src/runtime/profile/profile.rs

src/common/base/src/runtime/time_series/time_series_profile.rs

github-actions · 2025-04-22T07:30:24Z

Docker Image for PR

tag: pr-17809-07566f6-1745306957

note: this image tag is only available for internal use.

github-actions · 2025-04-22T08:14:14Z

ClickBench Report

This reverts commit dc7fb59.

dqhl76 · 2025-04-22T08:34:59Z

@zhang2014 Benchmark looks good. Please take a look again

zhang2014 · 2025-04-24T01:55:59Z

src/common/base/src/runtime/profile/profile.rs

-                profile.statistics[name as usize].fetch_add(value, Ordering::SeqCst);
+                let previous_value =
+                    profile.statistics[name.clone() as usize].fetch_add(value, Ordering::SeqCst);
+                if let Some(time_series_profile) = &profile.time_series {


add record_timeseries_profile function is better?

zhang2014 · 2025-04-24T01:58:46Z

src/common/base/src/runtime/time_series/time_series_profile.rs

+    }
+
+    pub fn finish(&self) {
+        for item in self.items.iter() {


Maybe need record last point?

zhang2014 · 2025-04-24T01:59:43Z

src/common/base/src/runtime/time_series/time_series_profile.rs

+        let mut items = Vec::with_capacity(len);
+        for i in 0..len {
+            items.push(TimeSeriesProfileItem {
+                queue: ConcurrentQueue::unbounded(),


Suggested change

queue: ConcurrentQueue::unbounded(),

queue: ConcurrentQueue::bounded(DEFAULT_BATCH_SIZE),

zhang2014 · 2025-04-24T02:00:18Z

src/common/base/src/runtime/time_series/time_series_profile.rs

+                Ordering::SeqCst,
+            ) {
+                Ok(_) => {
+                    item.queue


flush if queue is full

zhang2014 · 2025-04-24T02:03:50Z

src/common/base/src/runtime/profile/profile.rs

+                let previous_value =
+                    profile.statistics[name.clone() as usize].fetch_add(value, Ordering::SeqCst);
+                if let Some(time_series_profile) = &profile.time_series {
+                    time_series_profile.record_point(name, previous_value + value);


losting data.

zhang2014 · 2025-04-24T02:53:29Z

src/common/base/src/runtime/time_series/time_series_profile.rs

+            pub stats_name: String,
+            pub stats: Vec<(usize, usize)>,
+        }
+        info!(


too many logs.

github-actions bot added the pr-feature this PR introduces a new feature to the codebase label Apr 18, 2025

dqhl76 force-pushed the time-series-profile branch 2 times, most recently from 971528e to dfdef76 Compare April 18, 2025 13:17

dqhl76 added ci-benchmark Benchmark: run all test and removed ci-benchmark Benchmark: run all test labels Apr 21, 2025

dqhl76 added 5 commits April 21, 2025 14:38

feat: add time series statistics profile

eb4d4f0

fix

ab26568

fixup

6d0950c

disable target in default

7ba649e

fixup

a0ecd35

dqhl76 force-pushed the time-series-profile branch from 3cf1865 to a0ecd35 Compare April 21, 2025 06:38

dqhl76 marked this pull request as ready for review April 21, 2025 07:36

dqhl76 requested a review from zhang2014 April 21, 2025 07:36

zhang2014 reviewed Apr 21, 2025

View reviewed changes

src/common/base/src/runtime/profile/profile.rs Outdated Show resolved Hide resolved

zhang2014 reviewed Apr 21, 2025

View reviewed changes

src/common/base/src/runtime/time_series/time_series_profile.rs Outdated Show resolved Hide resolved

dqhl76 and others added 3 commits April 21, 2025 19:47

apply review suggestions

afbad9d

Merge branch 'main' into time-series-profile

ef9cb3e

test benchmark

dc7fb59

dqhl76 added the ci-benchmark Benchmark: run all test label Apr 22, 2025

Revert "test benchmark"

5afa43f

This reverts commit dc7fb59.

dqhl76 removed the ci-benchmark Benchmark: run all test label Apr 22, 2025

zhang2014 reviewed Apr 24, 2025

View reviewed changes

src/common/base/src/runtime/time_series/time_series_profile.rs

Ordering::SeqCst,

) {

Ok(_) => {

item.queue

Copy link

Member

zhang2014 Apr 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

flush if queue is full

zhang2014 reviewed Apr 24, 2025

View reviewed changes

zhang2014 marked this pull request as draft April 24, 2025 02:52

zhang2014 reviewed Apr 24, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add time series statistics profile #17809

feat: add time series statistics profile #17809

dqhl76 commented Apr 18, 2025 •

edited

Loading

github-actions bot commented Apr 21, 2025

github-actions bot commented Apr 21, 2025

dqhl76 commented Apr 21, 2025

github-actions bot commented Apr 22, 2025

github-actions bot commented Apr 22, 2025

dqhl76 commented Apr 22, 2025

zhang2014 Apr 24, 2025

zhang2014 Apr 24, 2025 •

edited

Loading

zhang2014 Apr 24, 2025

zhang2014 Apr 24, 2025

zhang2014 Apr 24, 2025

zhang2014 Apr 24, 2025

	queue: ConcurrentQueue::unbounded(),
	queue: ConcurrentQueue::bounded(DEFAULT_BATCH_SIZE),

feat: add time series statistics profile #17809

Are you sure you want to change the base?

feat: add time series statistics profile #17809

Conversation

dqhl76 commented Apr 18, 2025 • edited Loading

Summary

Tests

Type of change

github-actions bot commented Apr 21, 2025

Docker Image for PR

github-actions bot commented Apr 21, 2025

ClickBench Report

dqhl76 commented Apr 21, 2025

github-actions bot commented Apr 22, 2025

Docker Image for PR

github-actions bot commented Apr 22, 2025

ClickBench Report

dqhl76 commented Apr 22, 2025

zhang2014 Apr 24, 2025

Choose a reason for hiding this comment

zhang2014 Apr 24, 2025 • edited Loading

Choose a reason for hiding this comment

zhang2014 Apr 24, 2025

Choose a reason for hiding this comment

zhang2014 Apr 24, 2025

Choose a reason for hiding this comment

zhang2014 Apr 24, 2025

Choose a reason for hiding this comment

zhang2014 Apr 24, 2025

Choose a reason for hiding this comment

dqhl76 commented Apr 18, 2025 •

edited

Loading

zhang2014 Apr 24, 2025 •

edited

Loading