sql: TTL v1 meta issue #75428
Description
This is a meta issue for the v1 TTL meta issue.
⛔ = blocked
✔️ = in review
♻️ = in progress
Done
- RFC rfc: row level TTL #75189
- ALTER TABLE ... SET syntax sql: implement the ALTER TABLE ... SET (...) syntax #75262
- ALTER TABLE ... RESET syntax parser: implement ALTER TABLE ... RESET (...) #75429
- Persist metadata sql: persist TTL metadata to the descriptor #75602
- handle each options
- Placeholder for scheduled job
- (optional) look into protecting scheduled job from being cancelled
- Implement deletion job main logic
- CREATE TABLE ... WITH
- cron management for TTL
- (stretch?) add jitter to scheduled jobs framework - (cancelled) https://cockroachlabs.slack.com/archives/C02DSDS9TM1/p1644461478718559
- DROP TABLE/DATABASE/SCHEMA should also drop the scheduled job. (sql: implement ADD/DROP schema changes for row-level TTL #76216)
- ALTER TABLE ... SET setting up the automatic column and scheduled job for the first time (sql: implement ADD/DROP schema changes for row-level TTL #76216)
- ALTER TABLE ... SET modifying TTL expire_after's default expressions (sql: implement ADD/DROP schema changes for row-level TTL #76216)
- ALTER TABLE ... RESET dropping the TTL automatic column and scheduled job (sql: implement ADD/DROP schema changes for row-level TTL #76216)
- Break up the TTL job by ranges & add concurrency. (ttljob: wire up range_concurrency and cleanup query construction #76487)
- Cancel deletion job if schema change relating to a PK change or TTL parameter change. (ttljob: wire up range_concurrency and cleanup query construction #76487)
- Block FKs into a TTL table (sql: block FKs to/from TTL tables #76409)
- implement desired metrics on the TTL table (ttljob: add metrics to row level TTL #76701)
- Implement rate limiting for deletes (ttljob: implement a delete based rate limit #76615)
- backup and restore should re-instate the TTL schedule (in review: backupccl: handle RESTORE of TTL tables #76725)
- range scans are currently unpaginated - we need to paginate them. ( in review: ttljob: paginate ranges when doing TTL #76726)
- implement feature flag and pause for TTL stops (in review: ttljob: add controls to pause TTL jobs #76835)
- cleanup: create issues for all post-v1 TODOs and tag in codebase, add unimplemented issue ttl_automatic_expiration, CODEOWNERs, only allow ascending PKs, commit with a release note for TTL (ttl: finalize for v22.1 release #76918)
- ensure pg_class.reloptions matches SHOW CREATE TABLE (ttl: finalize for v22.1 release #76918)
out of scoped
- ttl_expiration_expression (maybe a stretch)
By February 25
- ✔️ Plug in admission priority for DELETE. (ttljob: add row statistics #76837)
- ✔️ implement a scan of deleted vs active rows to report as metrics: In addition to the current metrics, we need to write a loop that counts all the expired rows and existing rows in the table. For now, this will be an extra goroutine on the TTL job. This needs to be gated by a table setting and cluster setting. (in review: ttljob: add row statistics #76837)
Stability items
- ✔️ Add telemetry (ttl: add telemetry #77108)
- (happy for someone to take this) add validation builtins (see below); incorporate into
debug doctor
- Write some builtins which scan the descriptors and validate all tables with TTL have an associated scheduled job & scans row level TTL jobs and ensure they map to a valid table.
- Write some builtins which can create the scheduled jobs you might expect if they don't exist.
- Re-do performance testing of TTL
Product Questions
- Figure out how TTL priority configuration should align with
SET ADMISSION PRIORITY(i.e. using numbers versus enums) - figure out advice for "child" metrics (since we use
aggmetricfor metrics, child labels are not applied by default... see slack) - i'd advocate for calling row-level TTL "preview" / "experimental" for this release. should we?
Epic CRDB-10488
Jira issue: CRDB-12666