Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Revert "Storages: Update the length of Block queue and the number of read threads (#8702)" #8732

Merged
merged 1 commit into from
Jan 25, 2024
Merged

Revert "Storages: Update the length of Block queue and the number of read threads (#8702)" #8732

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion dbms/src/Interpreters/Settings.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -320,7 +320,7 @@ struct Settings
\
M(SettingDouble, dt_page_gc_threshold, 0.5, "Max valid rate of deciding to do a GC in PageStorage") \
M(SettingBool, dt_enable_read_thread, true, "Enable storage read thread or not") \
M(SettingDouble, dt_read_thread_count_scale, 2.0, "Number of read thread = number of logical cpu cores * dt_read_thread_count_scale. Only has meaning at server startup.") \
M(SettingDouble, dt_read_thread_count_scale, 1.0, "Number of read thread = number of logical cpu cores * dt_read_thread_count_scale. Only has meaning at server startup.") \
M(SettingUInt64, dt_max_sharing_column_bytes_for_all, 2048 * Constant::MB, "Memory limitation for data sharing of all requests. 0 means disable data sharing") \
M(SettingUInt64, dt_max_sharing_column_count, 5, "ColumnPtr object limitation for data sharing of each DMFileReader::Stream. 0 means disable data sharing") \
\
Expand Down
7 changes: 3 additions & 4 deletions dbms/src/Storages/DeltaMerge/SegmentReadTaskPool.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -185,10 +185,9 @@ class SegmentReadTaskPool : private boost::noncopyable
, exception_happened(false)
, mem_tracker(current_memory_tracker == nullptr ? nullptr : current_memory_tracker->shared_from_this())
// If the queue is too short, only 1 in the extreme case, it may cause the computation thread
// to encounter empty queues frequently, resulting in too much waiting and thread context switching.
// We limit the length of block queue to be 1.5 times of `num_streams_`, and in the extreme case,
// when `num_streams_` is 1, `block_slot_limit` is at least 2.
, block_slot_limit(std::ceil(num_streams_ * 1.5))
// to encounter empty queues frequently, resulting in too much waiting and thread context
// switching, so we limit the lower limit to 3, which provides two blocks of buffer space.
, block_slot_limit(std::max(num_streams_, 3))
// Limiting the minimum number of reading segments to 2 is to avoid, as much as possible,
// situations where the computation may be faster and the storage layer may not be able to keep up.
, active_segment_limit(std::max(num_streams_, 2))
Expand Down