[opt](cloud) Reduce empty rowset pressure on meta service #54395
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
liaoxin01
requested review from
dataroaring,
gavinchou and
w41ter
as code owners
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
liaoxin01
force-pushed
the
opt_empty_rowset
branch
from
e63fa9c to
30745b0
Compare
liaoxin01
changed the title
Contributor
Author
|
run buildall |
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
dataroaring
reviewed
Aug 6, 2025
be/src/cloud/cloud_meta_mgr.cpp
Outdated
| // missing versions are those that are not in the existing_versions | ||
| if (version.first > last_version + 1) { | ||
| // there is a hole between versions | ||
| auto prev_rowset = tablet->get_rowset_by_version(version); |
dataroaring
reviewed
Aug 6, 2025
be/src/cloud/cloud_meta_mgr.cpp
Outdated
| for (int64_t ver = last_version + 1; ver < version.first; ++ver) { | ||
| RowsetSharedPtr hole_rowset; | ||
| RETURN_IF_ERROR(create_empty_rowset_for_hole( | ||
| tablet, ver, prev_rowset->rowset_meta(), &hole_rowset)); |
Contributor
There was a problem hiding this comment.
We should use rowset meta from previous rowset. e.g. dropped column may happens.
dataroaring
reviewed
Aug 6, 2025
be/src/cloud/cloud_delta_writer.cpp
Outdated
| RETURN_IF_ERROR(_rowset_builder->init()); | ||
| RETURN_IF_ERROR(_rowset_builder->build_rowset()); | ||
| if (config::skip_writing_empty_rowset_metadata) { | ||
| return Status::OK(); |
Contributor
There was a problem hiding this comment.
what about prepare record?
Contributor
|
Testing advice:
|
Contributor
FE UT Coverage ReportIncrement line coverage `` 🎉 |
TPC-H: Total hot run time: 33914 ms |
TPC-DS: Total hot run time: 161817 ms |
ClickBench: Total hot run time: 35.16 s |
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
Author
|
run buildall |
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
liaoxin01
force-pushed
the
opt_empty_rowset
branch
2 times, most recently
from
d7a4356 to
e64d79b
Compare
Contributor
Author
|
run buildall |
Contributor
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
TPC-H: Total hot run time: 33753 ms |
Contributor
FE UT Coverage ReportIncrement line coverage `` 🎉 |
TPC-DS: Total hot run time: 161206 ms |
ClickBench: Total hot run time: 35.12 s |
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
gavinchou
approved these changes
Aug 22, 2025
github-actions
bot
added
the
approved
Contributor
|
PR approved by at least one committer and no changes requested. |
Contributor
|
PR approved by anyone and no changes requested. |
16 tasks
16 tasks
dataroaring
pushed a commit
that referenced
this pull request
Sep 3, 2025
…rowset_metadata (#55604) Related PR: #54395 Problem Summary: The _rs_metas and _rs_version_map information in tmp_tablet meta are inconsistent, causing the attempt to fetch rowset by version to fail (getting null pointer). The tmp_tablet meta was copied from new tablet, and its rowset information is actually useless since the real rowset data will be obtained later through sync rowset. The sync rowset operation failed to remove the old rowsets, resulting in this inconsistency. We need to first clean up the obsolete rowsets in tmp_tablet meta. *** SIGSEGV address not mapped to object (@0x38) received by PID 2824014 (TID 2824488 OR 0x7f59e8eff640) from PID 56; stack trace: *** 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /home/zcp/repo_center/doris_master/doris/be/src/common/signal_handler.h:420 1# PosixSignals::chained_handler(int, siginfo*, void*) [clone .part.0] in /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so 2# JVM_handle_linux_signal in /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so 3# 0x00007F5B3AC5F520 in /lib/x86_64-linux-gnu/libc.so.6 4# doris::cloud::CloudMetaMgr::fill_version_holes(doris::CloudTablet*, long, std::unique_lock<std::shared_mutex>&) at /home/zcp/repo_center/doris_master/doris/be/src/cloud/cloud_meta_mgr.cpp:1650 5# doris::cloud::CloudMetaMgr::sync_tablet_rowsets_unlocked(doris::CloudTablet*, std::unique_lock<bthread::Mutex>&, doris::SyncOptions const&, doris::SyncRowsetStats*) in /mnt/hdd01/PERFORMANCE_ENV/be/lib/doris_be 6# doris::cloud::CloudMetaMgr::sync_tablet_rowsets(doris::CloudTablet*, doris::SyncOptions const&, doris::SyncRowsetStats*) at /home/zcp/repo_center/doris_master/doris/be/src/cloud/cloud_meta_mgr.cpp:477 7# doris::CloudSchemaChangeJob::_process_delete_bitmap(long, long, long) at /home/zcp/repo_center/doris_master/doris/be/src/cloud/cloud_schema_change_job.cpp:519 8# doris::CloudSchemaChangeJob::_convert_historical_rowsets(doris::SchemaChangeParams const&, doris::cloud::TabletJobInfoPB&) at /home/zcp/repo_center/doris_master/doris/be/src/cloud/cloud_schema_change_job.cpp:424 9# doris::CloudSchemaChangeJob::process_alter_tablet(doris::TAlterTabletReqV2 const&) in /mnt/hdd01/PERFORMANCE_ENV/be/lib/doris_be 10# doris::alter_cloud_tablet_callback(doris::CloudStorageEngine&, doris::TAgentTaskRequest const&) at /home/zcp/repo_center/doris_master/doris/be/src/agent/task_worker_pool.cpp:2176 11# std::_Function_handler<void (), doris::TaskWorkerPool::submit_task(doris::TAgentTaskRequest const&)::$_0::operator()<doris::TAgentTaskRequest const&>(doris::TAgentTaskRequest const&) const::{lambda()#1}>::_M_invoke(std::_Any_data const&) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:292 12# doris::ThreadPool::dispatch_thread() at /home/zcp/repo_center/doris_master/doris/be/src/util/threadpool.cpp:621 13# doris::Thread::supervise_thread(void*) at /home/zcp/repo_center/doris_master/doris/be/src/util/thread.cpp:461
uchenily
pushed a commit
to uchenily/doris
that referenced
this pull request
Sep 5, 2025
…rowset_metadata (apache#55604) Related PR: apache#54395 Problem Summary: The _rs_metas and _rs_version_map information in tmp_tablet meta are inconsistent, causing the attempt to fetch rowset by version to fail (getting null pointer). The tmp_tablet meta was copied from new tablet, and its rowset information is actually useless since the real rowset data will be obtained later through sync rowset. The sync rowset operation failed to remove the old rowsets, resulting in this inconsistency. We need to first clean up the obsolete rowsets in tmp_tablet meta. *** SIGSEGV address not mapped to object (@0x38) received by PID 2824014 (TID 2824488 OR 0x7f59e8eff640) from PID 56; stack trace: *** 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /home/zcp/repo_center/doris_master/doris/be/src/common/signal_handler.h:420 1# PosixSignals::chained_handler(int, siginfo*, void*) [clone .part.0] in /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so 2# JVM_handle_linux_signal in /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so 3# 0x00007F5B3AC5F520 in /lib/x86_64-linux-gnu/libc.so.6 4# doris::cloud::CloudMetaMgr::fill_version_holes(doris::CloudTablet*, long, std::unique_lock<std::shared_mutex>&) at /home/zcp/repo_center/doris_master/doris/be/src/cloud/cloud_meta_mgr.cpp:1650 5# doris::cloud::CloudMetaMgr::sync_tablet_rowsets_unlocked(doris::CloudTablet*, std::unique_lock<bthread::Mutex>&, doris::SyncOptions const&, doris::SyncRowsetStats*) in /mnt/hdd01/PERFORMANCE_ENV/be/lib/doris_be 6# doris::cloud::CloudMetaMgr::sync_tablet_rowsets(doris::CloudTablet*, doris::SyncOptions const&, doris::SyncRowsetStats*) at /home/zcp/repo_center/doris_master/doris/be/src/cloud/cloud_meta_mgr.cpp:477 7# doris::CloudSchemaChangeJob::_process_delete_bitmap(long, long, long) at /home/zcp/repo_center/doris_master/doris/be/src/cloud/cloud_schema_change_job.cpp:519 8# doris::CloudSchemaChangeJob::_convert_historical_rowsets(doris::SchemaChangeParams const&, doris::cloud::TabletJobInfoPB&) at /home/zcp/repo_center/doris_master/doris/be/src/cloud/cloud_schema_change_job.cpp:424 9# doris::CloudSchemaChangeJob::process_alter_tablet(doris::TAlterTabletReqV2 const&) in /mnt/hdd01/PERFORMANCE_ENV/be/lib/doris_be 10# doris::alter_cloud_tablet_callback(doris::CloudStorageEngine&, doris::TAgentTaskRequest const&) at /home/zcp/repo_center/doris_master/doris/be/src/agent/task_worker_pool.cpp:2176 11# std::_Function_handler<void (), doris::TaskWorkerPool::submit_task(doris::TAgentTaskRequest const&)::$_0::operator()<doris::TAgentTaskRequest const&>(doris::TAgentTaskRequest const&) const::{lambda()apache#1}>::_M_invoke(std::_Any_data const&) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:292 12# doris::ThreadPool::dispatch_thread() at /home/zcp/repo_center/doris_master/doris/be/src/util/threadpool.cpp:621 13# doris::Thread::supervise_thread(void*) at /home/zcp/repo_center/doris_master/doris/be/src/util/thread.cpp:461
16 tasks
liaoxin01
added a commit
to liaoxin01/doris
that referenced
this pull request
Sep 12, 2025
liaoxin01
added a commit
to liaoxin01/doris
that referenced
this pull request
Sep 12, 2025
…a to false and add it to fuzzy (apache#55171) Related PR: apache#54395
liaoxin01
added a commit
to liaoxin01/doris
that referenced
this pull request
Sep 12, 2025
…rowset_metadata (apache#55604) Related PR: apache#54395 Problem Summary: The _rs_metas and _rs_version_map information in tmp_tablet meta are inconsistent, causing the attempt to fetch rowset by version to fail (getting null pointer). The tmp_tablet meta was copied from new tablet, and its rowset information is actually useless since the real rowset data will be obtained later through sync rowset. The sync rowset operation failed to remove the old rowsets, resulting in this inconsistency. We need to first clean up the obsolete rowsets in tmp_tablet meta. *** SIGSEGV address not mapped to object (@0x38) received by PID 2824014 (TID 2824488 OR 0x7f59e8eff640) from PID 56; stack trace: *** 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /home/zcp/repo_center/doris_master/doris/be/src/common/signal_handler.h:420 1# PosixSignals::chained_handler(int, siginfo*, void*) [clone .part.0] in /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so 2# JVM_handle_linux_signal in /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so 3# 0x00007F5B3AC5F520 in /lib/x86_64-linux-gnu/libc.so.6 4# doris::cloud::CloudMetaMgr::fill_version_holes(doris::CloudTablet*, long, std::unique_lock<std::shared_mutex>&) at /home/zcp/repo_center/doris_master/doris/be/src/cloud/cloud_meta_mgr.cpp:1650 5# doris::cloud::CloudMetaMgr::sync_tablet_rowsets_unlocked(doris::CloudTablet*, std::unique_lock<bthread::Mutex>&, doris::SyncOptions const&, doris::SyncRowsetStats*) in /mnt/hdd01/PERFORMANCE_ENV/be/lib/doris_be 6# doris::cloud::CloudMetaMgr::sync_tablet_rowsets(doris::CloudTablet*, doris::SyncOptions const&, doris::SyncRowsetStats*) at /home/zcp/repo_center/doris_master/doris/be/src/cloud/cloud_meta_mgr.cpp:477 7# doris::CloudSchemaChangeJob::_process_delete_bitmap(long, long, long) at /home/zcp/repo_center/doris_master/doris/be/src/cloud/cloud_schema_change_job.cpp:519 8# doris::CloudSchemaChangeJob::_convert_historical_rowsets(doris::SchemaChangeParams const&, doris::cloud::TabletJobInfoPB&) at /home/zcp/repo_center/doris_master/doris/be/src/cloud/cloud_schema_change_job.cpp:424 9# doris::CloudSchemaChangeJob::process_alter_tablet(doris::TAlterTabletReqV2 const&) in /mnt/hdd01/PERFORMANCE_ENV/be/lib/doris_be 10# doris::alter_cloud_tablet_callback(doris::CloudStorageEngine&, doris::TAgentTaskRequest const&) at /home/zcp/repo_center/doris_master/doris/be/src/agent/task_worker_pool.cpp:2176 11# std::_Function_handler<void (), doris::TaskWorkerPool::submit_task(doris::TAgentTaskRequest const&)::$_0::operator()<doris::TAgentTaskRequest const&>(doris::TAgentTaskRequest const&) const::{lambda()apache#1}>::_M_invoke(std::_Any_data const&) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:292 12# doris::ThreadPool::dispatch_thread() at /home/zcp/repo_center/doris_master/doris/be/src/util/threadpool.cpp:621 13# doris::Thread::supervise_thread(void*) at /home/zcp/repo_center/doris_master/doris/be/src/util/thread.cpp:461
liaoxin01
added a commit
to liaoxin01/doris
that referenced
this pull request
Sep 12, 2025
… when enable skip_writing_empty_rowset_metadata (apache#55742) Issue Number: close #xxx Related PR: apache#54395
Closed
16 tasks
liaoxin01
added a commit
that referenced
this pull request
Jan 13, 2026
…owsets (#59710) ### What problem does this PR solve? Related PR: #54395 Problem Summary: When skip_writing_empty_rowset_metadata is enabled, empty rowsets are not committed to meta-service. Previously, set_txn_related_delete_bitmap() would either: 1. Store full rowset info in cache (causing memory leak since sync_tablet_delete_bitmap_by_cache() never cleans it up), or 2. Skip storing entirely (causing CalcDeleteBitmapTask to fail with NOT_FOUND error) This fix introduces a lightweight marker mechanism: - For empty rowsets, store only a TxnKey marker (~16 bytes) instead of full rowset info - CalcDeleteBitmapTask checks for marker via is_empty_rowset() and returns success if found (marker is NOT removed to support task retry) - Cleanup is handled by expiration-based removal in remove_expired_tablet_txn_info() - Expiration time is consistent with set_tablet_txn_info()
github-actions bot
pushed a commit
that referenced
this pull request
Jan 13, 2026
…owsets (#59710) ### What problem does this PR solve? Related PR: #54395 Problem Summary: When skip_writing_empty_rowset_metadata is enabled, empty rowsets are not committed to meta-service. Previously, set_txn_related_delete_bitmap() would either: 1. Store full rowset info in cache (causing memory leak since sync_tablet_delete_bitmap_by_cache() never cleans it up), or 2. Skip storing entirely (causing CalcDeleteBitmapTask to fail with NOT_FOUND error) This fix introduces a lightweight marker mechanism: - For empty rowsets, store only a TxnKey marker (~16 bytes) instead of full rowset info - CalcDeleteBitmapTask checks for marker via is_empty_rowset() and returns success if found (marker is NOT removed to support task retry) - Cleanup is handled by expiration-based removal in remove_expired_tablet_txn_info() - Expiration time is consistent with set_tablet_txn_info()
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
This pull request introduces several improvements and new features to the cloud storage engine, focusing on handling empty rowsets and filling version holes in tablets. The changes enhance data consistency, optimize metadata operations, and improve compaction statistics by introducing configuration options and new logic for managing rowset versions.
Handling empty rowsets and version holes:
skip_writing_empty_rowset_metadataandenable_fill_version_holesto control whether empty rowset metadata is written to the meta service and whether missing version holes are automatically filled during tablet synchronization.CloudMetaMgrto detect and fill version holes by creating deterministic empty rowsets for missing versions, ensuring continuous versioning and tracking the number of holes filled.CloudDeltaWriterandCloudRowsetBuilderto support skipping metadata writes for empty rowsets, including batch initialization and commit logic based on the new configuration.Rowsetand related classes to distinguish hole rowsets from normal rowsets, preventing them from affecting statistics and compaction counts.These changes collectively improve the robustness of version management in the cloud storage engine, reduce unnecessary metadata operations for empty rowsets, and maintain accurate tablet statistics.
Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)