feat: implement irreversible vacuum drop table protection #18809
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add retention guard mechanism to prevent undrop operations after vacuum has started, ensuring data consistency by blocking restoration of tables whose data may have been partially or fully cleaned up. Key changes: - Add VacuumRetention metadata with monotonic timestamp semantics - Implement fetch_set_vacuum_timestamp API with correct CAS behavior - Integrate retention checks in vacuum drop table workflow - Add retention guard validation in undrop table operations - Include comprehensive error handling and user-friendly messages - Add protobuf serialization support with v151 compatibility - Provide full integration test coverage Fixes data integrity issue where undrop could succeed on tables with incomplete S3 data after vacuum cleanup has begun. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
github-actions
bot
added
the
pr-feature
Make error message more user-friendly by: - Using clear language about why undrop is blocked - Including vacuum start timestamp for better context - Removing technical jargon like 'retention guard' and 'precedes' 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
- Rename VacuumRetention to VacuumWatermark for clarity - Remove unnecessary fields: updated_by, updated_at, version - Keep only essential 'time' field for monotonic timestamp tracking - Update protobuf conversion and tests accordingly - Maintain API compatibility and retention guard functionality 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
- Move vacuum timestamp setting from gc_drop_tables to VacuumDropTablesInterpreter::execute2 - Use actual retention settings instead of hardcoded 7 days - Set timestamp before vacuum operation starts for better timing - Simplify gc_drop_tables to focus only on metadata cleanup - Improve separation of concerns between business logic and cleanup operations 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
- Only set vacuum timestamp when NOT in dry run mode - Maintains consistency with existing dry run behavior for metadata cleanup - Dry run should not modify any state including vacuum watermark - Preserves read-only nature of dry run operations 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/
Summary
Fixes data integrity issue where undrop could succeed on tables with incomplete S3 data after vacuum cleanup has begun.
🤖 Generated with Claude Code
Add retention guard mechanism to prevent undrop operations after vacuum has started, ensuring data consistency by blocking restoration of tables whose data may have been partially or fully cleaned up.
Key changes:
Tests
Type of change
This change is