Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Storages: Refine log #8963

Merged
merged 5 commits into from
Apr 18, 2024
Merged

Storages: Refine log #8963

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
30de067
Storages: refine log
JinheLin Apr 18, 2024
c4bc1fa
Merge branch 'master' into refine_log
JaySon-Huang Apr 18, 2024
01e63e8
Remove useless logging
JaySon-Huang Apr 18, 2024
bb7154e
Unify the naming from read_tso/max_version -> start_ts
JaySon-Huang Apr 18, 2024
d6a37cc
Unify the logging style
JaySon-Huang Apr 18, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 14 additions & 20 deletions dbms/src/Storages/DeltaMerge/DeltaMergeStore.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1197,15 +1197,6 @@ BlockInputStreams DeltaMergeStore::read(
/*try_split_task =*/!enable_read_thread);
auto log_tracing_id = getLogTracingId(*dm_context);
auto tracing_logger = log->getChild(log_tracing_id);
LOG_INFO(
tracing_logger,
"Read create segment snapshot done, keep_order={} dt_enable_read_thread={} enable_read_thread={} "
"is_fast_scan={} is_push_down_filter_empty={}",
keep_order,
db_context.getSettingsRef().dt_enable_read_thread,
enable_read_thread,
is_fast_scan,
filter == nullptr || filter->before_where == nullptr);

auto after_segment_read = [&](const DMContextPtr & dm_context_, const SegmentPtr & segment_) {
// TODO: Update the tracing_id before checkSegmentUpdate?
Expand Down Expand Up @@ -1263,7 +1254,13 @@ BlockInputStreams DeltaMergeStore::read(
}
LOG_INFO(
tracing_logger,
"Read create stream done, pool_id={} num_streams={}",
"Read create stream done, keep_order={} dt_enable_read_thread={} enable_read_thread={} "
"is_fast_scan={} is_push_down_filter_empty={} pool_id={} num_streams={}",
keep_order,
db_context.getSettingsRef().dt_enable_read_thread,
enable_read_thread,
is_fast_scan,
filter == nullptr || filter->before_where == nullptr,
read_task_pool->pool_id,
final_num_stream);

Expand Down Expand Up @@ -1306,15 +1303,6 @@ void DeltaMergeStore::read(
/*try_split_task =*/!enable_read_thread);
auto log_tracing_id = getLogTracingId(*dm_context);
auto tracing_logger = log->getChild(log_tracing_id);
LOG_INFO(
tracing_logger,
"Read create segment snapshot done, keep_order={} dt_enable_read_thread={} enable_read_thread={} "
"is_fast_scan={} is_push_down_filter_empty={}",
keep_order,
db_context.getSettingsRef().dt_enable_read_thread,
enable_read_thread,
is_fast_scan,
filter == nullptr || filter->before_where == nullptr);

auto after_segment_read = [&](const DMContextPtr & dm_context_, const SegmentPtr & segment_) {
// TODO: Update the tracing_id before checkSegmentUpdate?
Expand Down Expand Up @@ -1383,7 +1371,13 @@ void DeltaMergeStore::read(

LOG_INFO(
tracing_logger,
"Read create PipelineExec done, pool_id={} num_streams={}",
"Read create PipelineExec done, keep_order={} dt_enable_read_thread={} enable_read_thread={} "
"is_fast_scan={} is_push_down_filter_empty={} pool_id={} num_streams={}",
keep_order,
db_context.getSettingsRef().dt_enable_read_thread,
enable_read_thread,
is_fast_scan,
filter == nullptr || filter->before_where == nullptr,
read_task_pool->pool_id,
final_num_stream);
}
Expand Down
4 changes: 2 additions & 2 deletions dbms/src/Storages/DeltaMerge/SegmentReadTaskPool.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -117,7 +117,7 @@ class SegmentReadTaskPool
Int64 num_streams_,
const String & res_group_name_);

~SegmentReadTaskPool()
~SegmentReadTaskPool() override
{
auto [pop_times, pop_empty_times, max_queue_size] = q.getStat();
auto pop_empty_ratio = pop_times > 0 ? pop_empty_times * 1.0 / pop_times : 0.0;
Expand All @@ -126,7 +126,7 @@ class SegmentReadTaskPool
auto blk_avg_bytes = total_count > 0 ? total_bytes / total_count : 0;
auto approximate_max_pending_block_bytes = blk_avg_bytes * max_queue_size;
auto total_rows = blk_stat.totalRows();
LOG_DEBUG(
LOG_INFO(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. Is there only 1 SegmentReadTaskPool for one TableScan?
  2. If it helps locate the performance issue of ReadThreads, consider adding it to ScanContext?

Copy link
Contributor Author

@JinheLin JinheLin Apr 18, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. Yes
  2. It can be added to ScanContext, but not add all of them directly, need some considerations.

@JaySon-Huang

log,
"Done. pool_id={} pop={} pop_empty={} pop_empty_ratio={} "
"max_queue_size={} blk_avg_bytes={} approximate_max_pending_block_bytes={:.2f}MB "
Expand Down