Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

sql: v22.2.0: cannot perform TRUNCATE due to concurrent unknown mutation #93443

Closed
cockroach-teamcity opened this issue Dec 12, 2022 · 4 comments · Fixed by #98537
Closed

sql: v22.2.0: cannot perform TRUNCATE due to concurrent unknown mutation #93443

cockroach-teamcity opened this issue Dec 12, 2022 · 4 comments · Fixed by #98537
Assignees
@chengxiong-ruan
Labels
branch-master Failures and bugs on the master branch. branch-release-23.1 Used to mark GA and release blockers, technical advisories, and bugs for 23.1 C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report. T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions)

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Dec 12, 2022 •
edited by cockroach-jira-scripts
Loading

This issue was autofiled by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry link: https://sentry.io/organizations/cockroach-labs/issues/3803757482/?referrer=webhooks_plugin

Panic message:

truncate.go:368: cannot perform TRUNCATE due to concurrent unknown mutation of type *tabledesc.mutation for mutation 1 in tabledesc.Mutable: {ID: 1993, Version: 23, ModificationTime: "1670869560.388020034,0", ParentID: 1771, ParentSchemaID: 2010, State: PUBLIC, NextColumnID: 8, Columns: [{ID: 1, TypeID: 2950, Null: false, HasDefault: true}, {ID: 2, TypeID: 20, Null: false}, {ID: 3, TypeID: 20, Null: false}, {ID: 4, TypeID: 20, Null: false}, {ID: 5, TypeID: 1184, Null: true}, {ID: 6, TypeID: 1184, Null: true}], NextFamilyID: 1, Families: [{ID: 0, Columns: [1, 2, 3, 4, 5, 6, 7]}], MutationJobs: [{MutationID: 3, JobID: 821818688580190246}], Mutations: [{MutationID: 3, Direction: ADD, State: WRITE_ONLY, Column: {ID: 7, TypeID: 1184, Null: false, Hidden: true, HasDefault: true, State: ADD, MutationID: 3}}, {MutationID: 3, Direction: ADD, State: WRITE_ONLY}], PrimaryIndex: 10, NextIndexID: 13, Indexes: [{ID: 10, Unique: true, KeyColumns: [{ID: 1, Dir: ASC}], StoreColumns: [2, 3, 4, 5, 6, 7]}, {ID: 11, Unique: false, KeyColumns: [{ID: 5, Dir: DESC}], KeySuffixColumns: [1]}]}
(1)
Wraps: (2) assertion failure
Wraps: (3) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/sql.checkTableForDisallowedMutationsWithTruncate
| github.com/cockroachdb/cockroach/pkg/sql/truncate.go:368
| github.com/cockroachdb/cockroach/pkg/sql.(*planner).truncateTable
| github.com/cockroachdb/cockroach/pkg/sql/truncate.go:173
| github.com/cockroachdb/cockroach/pkg/sql.(*truncateNode).startExec
| github.com/cockroachdb/cockroach/pkg/sql/truncate.go:128
| github.com/cockroachdb/cockroach/pkg/sql.startExec.func2
| github.com/cockroachdb/cockroach/pkg/sql/plan.go:518
| github.com/cockroachdb/cockroach/pkg/sql.(*planVisitor).visitInternal.func1
| github.com/cockroachdb/cockroach/pkg/sql/walk.go:112
| github.com/cockroachdb/cockroach/pkg/sql.(*planVisitor).visitInternal
| github.com/cockroachdb/cockroach/pkg/sql/walk.go:297
| github.com/cockroachdb/cockroach/pkg/sql.(*planVisitor).visit
| github.com/cockroachdb/cockroach/pkg/sql/walk.go:79
| github.com/cockroachdb/cockroach/pkg/sql.walkPlan
| github.com/cockroachdb/cockroach/pkg/sql/walk.go:43
| github.com/cockroachdb/cockroach/pkg/sql.startExec
| github.com/cockroachdb/cockroach/pkg/sql/plan.go:521
| github.com/cockroachdb/cockroach/pkg/sql.(*planNodeToRowSource).Start
| github.com/cockroachdb/cockroach/pkg/sql/plan_node_to_row_source.go:147
| github.com/cockroachdb/cockroach/pkg/sql/colexec.(*Columnarizer).Init
| github.com/cockroachdb/cockroach/pkg/sql/colexec/columnarizer.go:178
| github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).Init
| github.com/cockroachdb/cockroach/pkg/sql/colflow/stats.go:90
| github.com/cockroachdb/cockroach/pkg/sql/colflow.(*BatchFlowCoordinator).init.func1
| github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:247
| github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError
| github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:92
| github.com/cockroachdb/cockroach/pkg/sql/colflow.(*BatchFlowCoordinator).init
| github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:246
| github.com/cockroachdb/cockroach/pkg/sql/colflow.(*BatchFlowCoordinator).Run
| github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:291
| github.com/cockroachdb/cockroach/pkg/sql/colflow.(*vectorizedFlow).Run
| github.com/cockroachdb/cockroach/pkg/sql/colflow/vectorized_flow.go:320
| github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).Run
| github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:695
| github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRun
| github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1611
| github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRunAll
| github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1334
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithDistSQLEngine
| github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1541
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).dispatchToExecutionEngine
| github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1177
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmtInOpenState
| github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:687
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt.func1
| github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:129
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithProfiling
| github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:2382
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt
| github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:128
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execPortal
| github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:218
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd.func2
| github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1998
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd
| github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:2000
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).run
| github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1846
| github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn
| github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:828
| github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).processCommandsAsync.func1
| github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:728
Wraps: (4) cannot perform TRUNCATE due to concurrent unknown mutation of type *tabledesc.mutation for mutation 1 in tabledesc.Mutable: {ID: 1993, Version: 23, ModificationTime: "1670869560.388020034,0", ParentID: 1771, ParentSchemaID: 2010, State: PUBLIC, NextColumnID: 8, Columns: [{ID: 1, TypeID: 2950, Null: false, HasDefault: true}, {ID: 2, TypeID: 20, Null: false}, {ID: 3, TypeID: 20, Null: false}, {ID: 4, TypeID: 20, Null: false}, {ID: 5, TypeID: 1184, Null: true}, {ID: 6, TypeID: 1184, Null: true}], NextFamilyID: 1, Families: [{ID: 0, Columns: [1, 2, 3, 4, 5, 6, 7]}], MutationJobs: [{MutationID: 3, JobID: 821818688580190246}], Mutations: [{MutationID: 3, Direction: ADD, State: WRITE_ONLY, Column: {ID: 7, TypeID: 1184, Null: false, Hidden: true, HasDefault: true, State: ADD, MutationID: 3}}, {MutationID: 3, Direction: ADD, State: WRITE_ONLY}], PrimaryIndex: 10, NextIndexID: 13, Indexes: [{ID: 10, Unique: true, KeyColumns: [{ID: 1, Dir: ASC}], StoreColumns: [2, 3, 4, 5, 6, 7]}, {ID: 11, Unique: false, KeyColumns: [{ID: 5, Dir: DESC}], KeySuffixColumns: [1]}]}
Error types: (1) *colexecerror.notInternalError (2) *assert.withAssertionFailure (3) *withstack.withStack (4) *errutil.leafError
-- report composition:
*errutil.leafError: cannot perform TRUNCATE due to concurrent unknown mutation of type *tabledesc.mutation for mutation 1 in tabledesc.Mutable: {ID: 1993, Version: 23, ModificationTime: "1670869560.388020034,0", ParentID: 1771, ParentSchemaID: 2010, State: PUBLIC, NextColumnID: 8, Columns: [{ID: 1, TypeID: 2950, Null: false, HasDefault: true}, {ID: 2, TypeID: 20, Null: false}, {ID: 3, TypeID: 20, Null: false}, {ID: 4, TypeID: 20, Null: false}, {ID: 5, TypeID: 1184, Null: true}, {ID: 6, TypeID: 1184, Null: true}], NextFamilyID: 1, Families: [{ID: 0, Columns: [1, 2, 3, 4, 5, 6, 7]}], MutationJobs: [{MutationID: 3, JobID: 821818688580190246}], Mutations: [{MutationID: 3, Direction: ADD, State: WRITE_ONLY, Column: {ID: 7, TypeID: 1184, Null: false, Hidden: true, HasDefault: true, State: ADD, MutationID: 3}}, {MutationID: 3, Direction: ADD, State: WRITE_ONLY}], PrimaryIndex: 10, NextIndexID: 13, Indexes: [{ID: 10, Unique: true, KeyColumns: [{ID: 1, Dir: ASC}], StoreColumns: [2, 3, 4, 5, 6, 7]}, {ID: 11, Unique: false, KeyColumns: [{ID: 5, Dir: DESC}], KeySuffixColumns: [1]}]}
truncate.go:368: *withstack.withStack (top exception)
*assert.withAssertionFailure
*colexecerror.notInternalError

Stacktrace (expand for inline code snippets):

cockroach/pkg/sql/truncate.go

Lines 367 to 369 in 77667a1

} else {
return errors.AssertionFailedf("cannot perform TRUNCATE due to "+
"concurrent unknown mutation of type %T for mutation %d in %v", m, i, desc)
in pkg/sql.checkTableForDisallowedMutationsWithTruncate

cockroach/pkg/sql/truncate.go

Lines 172 to 174 in 77667a1

if err := checkTableForDisallowedMutationsWithTruncate(tableDesc); err != nil {
return err
in pkg/sql.(*planner).truncateTable

cockroach/pkg/sql/truncate.go

Lines 127 to 129 in 77667a1

for id, name := range toTruncate {
if err := p.truncateTable(ctx, id, tree.AsStringWithFQNames(t.n, params.Ann())); err != nil {
return err
in pkg/sql.(*truncateNode).startExec

cockroach/pkg/sql/plan.go

Lines 517 to 519 in 77667a1

}
return n.startExec(params)
},
in pkg/sql.startExec.func2

cockroach/pkg/sql/walk.go

Lines 111 to 113 in 77667a1

}
v.err = v.observer.leaveNode(name, plan)
}()
in pkg/sql.(*planVisitor).visitInternal.func1

cockroach/pkg/sql/walk.go

Lines 296 to 298 in 77667a1

}
}
in pkg/sql.(*planVisitor).visitInternal

cockroach/pkg/sql/walk.go

Lines 78 to 80 in 77667a1

}
v.visitInternal(plan, name)
return plan
in pkg/sql.(*planVisitor).visit

cockroach/pkg/sql/walk.go

Lines 42 to 44 in 77667a1

v := makePlanVisitor(ctx, observer)
v.visit(plan)
return v.err
in pkg/sql.walkPlan

cockroach/pkg/sql/plan.go

Lines 520 to 522 in 77667a1

}
return walkPlan(params.ctx, plan, o)
}
in pkg/sql.startExec

cockroach/pkg/sql/plan_node_to_row_source.go

Lines 146 to 148 in 77667a1

// This starts all of the nodes below this node.
if err := startExec(p.params, p.node); err != nil {
p.MoveToDraining(err)
in pkg/sql.(*planNodeToRowSource).Start

cockroach/pkg/sql/colexec/columnarizer.go

Lines 177 to 179 in 77667a1

ctx = c.StartInternalNoSpan(ctx)
c.input.Start(ctx)
if execStatsHijacker, ok := c.input.(execinfra.ExecStatsForTraceHijacker); ok {
in pkg/sql/colexec.(*Columnarizer).Init

cockroach/pkg/sql/colflow/stats.go

Lines 89 to 91 in 77667a1

func (bic *batchInfoCollector) Init(ctx context.Context) {
bic.Operator.Init(ctx)
bic.mu.Lock()
in pkg/sql/colflow.(*batchInfoCollector).Init

cockroach/pkg/sql/colflow/flow_coordinator.go

Lines 246 to 248 in 77667a1

return colexecerror.CatchVectorizedRuntimeError(func() {
f.input.Root.Init(ctx)
})
in pkg/sql/colflow.(*BatchFlowCoordinator).init.func1

cockroach/pkg/sql/colexecerror/error.go

Lines 91 to 93 in 77667a1

}()
operation()
return retErr
in pkg/sql/colexecerror.CatchVectorizedRuntimeError

cockroach/pkg/sql/colflow/flow_coordinator.go

Lines 245 to 247 in 77667a1

func (f *BatchFlowCoordinator) init(ctx context.Context) error {
return colexecerror.CatchVectorizedRuntimeError(func() {
f.input.Root.Init(ctx)
in pkg/sql/colflow.(*BatchFlowCoordinator).init

cockroach/pkg/sql/colflow/flow_coordinator.go

Lines 290 to 292 in 77667a1

if err := f.init(ctx); err != nil {
f.pushError(err)
in pkg/sql/colflow.(*BatchFlowCoordinator).Run

cockroach/pkg/sql/colflow/vectorized_flow.go

Lines 319 to 321 in 77667a1

log.VEvent(ctx, 1, "running the batch flow coordinator in the flow's goroutine")
f.batchFlowCoordinator.Run(ctx)
}
in pkg/sql/colflow.(*vectorizedFlow).Run

cockroach/pkg/sql/distsql_running.go

Lines 694 to 696 in 77667a1

// TODO(radu): this should go through the flow scheduler.
flow.Run(ctx, func() {})
in pkg/sql.(*DistSQLPlanner).Run

cockroach/pkg/sql/distsql_running.go

Lines 1610 to 1612 in 77667a1

recv.expectedRowsRead = int64(physPlan.TotalEstimatedScannedRows)
runCleanup := dsp.Run(ctx, planCtx, txn, physPlan, recv, evalCtx, nil /* finishedSetupFn */)
return func() {
in pkg/sql.(*DistSQLPlanner).PlanAndRun

cockroach/pkg/sql/distsql_running.go

Lines 1333 to 1335 in 77667a1

// the planner whether or not to plan remote table readers.
cleanup := dsp.PlanAndRun(
ctx, evalCtx, planCtx, planner.txn, planner.curPlan.main, recv,
in pkg/sql.(*DistSQLPlanner).PlanAndRunAll

cockroach/pkg/sql/conn_executor_exec.go

Lines 1540 to 1542 in 77667a1

}
err := ex.server.cfg.DistSQLPlanner.PlanAndRunAll(ctx, evalCtx, planCtx, planner, recv, evalCtxFactory)
return *recv.stats, err
in pkg/sql.(*connExecutor).execWithDistSQLEngine

cockroach/pkg/sql/conn_executor_exec.go

Lines 1176 to 1178 in 77667a1

ex.sessionTracing.TraceExecStart(ctx, "distributed")
stats, err = ex.execWithDistSQLEngine(
ctx, planner, stmt.AST.StatementReturnType(), res, distribute, progAtomic,
in pkg/sql.(*connExecutor).dispatchToExecutionEngine

cockroach/pkg/sql/conn_executor_exec.go

Lines 686 to 688 in 77667a1

if err := ex.dispatchToExecutionEngine(stmtCtx, p, res); err != nil {
stmtThresholdSpan.Finish()
in pkg/sql.(*connExecutor).execStmtInOpenState

cockroach/pkg/sql/conn_executor_exec.go

Lines 128 to 130 in 77667a1

err = ex.execWithProfiling(ctx, ast, prepared, func(ctx context.Context) error {
ev, payload, err = ex.execStmtInOpenState(ctx, parserStmt, prepared, pinfo, res, canAutoCommit)
return err
in pkg/sql.(*connExecutor).execStmt.func1

cockroach/pkg/sql/conn_executor_exec.go

Lines 2381 to 2383 in 77667a1

} else {
err = op(ctx)
}
in pkg/sql.(*connExecutor).execWithProfiling

cockroach/pkg/sql/conn_executor_exec.go

Lines 127 to 129 in 77667a1

case stateOpen:
err = ex.execWithProfiling(ctx, ast, prepared, func(ctx context.Context) error {
ev, payload, err = ex.execStmtInOpenState(ctx, parserStmt, prepared, pinfo, res, canAutoCommit)
in pkg/sql.(*connExecutor).execStmt

cockroach/pkg/sql/conn_executor_exec.go

Lines 217 to 219 in 77667a1

}
ev, payload, err = ex.execStmt(ctx, portal.Stmt.Statement, portal.Stmt, pinfo, stmtRes, canAutoCommit)
// Portal suspension is supported via a "side" state machine
in pkg/sql.(*connExecutor).execPortal

cockroach/pkg/sql/conn_executor.go

Lines 1997 to 1999 in 77667a1

canAutoCommit := ex.implicitTxn() && tcmd.FollowedBySync
ev, payload, err = ex.execPortal(ctx, portal, portalName, stmtRes, pinfo, canAutoCommit)
return err
in pkg/sql.(*connExecutor).execCmd.func2

cockroach/pkg/sql/conn_executor.go

Lines 1999 to 2001 in 77667a1

return err
}()
// Note: we write to ex.statsCollector.phaseTimes, instead of ex.phaseTimes,
in pkg/sql.(*connExecutor).execCmd

cockroach/pkg/sql/conn_executor.go

Lines 1845 to 1847 in 77667a1

var err error
if err = ex.execCmd(); err != nil {
if errors.IsAny(err, io.EOF, errDrainingComplete) {
in pkg/sql.(*connExecutor).run

cockroach/pkg/sql/conn_executor.go

Lines 827 to 829 in 77667a1

}(ctx, h)
return h.ex.run(ctx, s.pool, reserved, cancel)
}
in pkg/sql.(*Server).ServeConn

cockroach/pkg/sql/pgwire/conn.go

Lines 727 to 729 in 77667a1

reservedOwned = false // We're about to pass ownership away.
retErr = sqlServer.ServeConn(ctx, connHandler, reserved, cancelConn)
}()
in pkg/sql/pgwire.(*conn).processCommandsAsync.func1

pkg/sql/truncate.go in pkg/sql.checkTableForDisallowedMutationsWithTruncate at line 368
pkg/sql/truncate.go in pkg/sql.(*planner).truncateTable at line 173
pkg/sql/truncate.go in pkg/sql.(*truncateNode).startExec at line 128
pkg/sql/plan.go in pkg/sql.startExec.func2 at line 518
pkg/sql/walk.go in pkg/sql.(*planVisitor).visitInternal.func1 at line 112
pkg/sql/walk.go in pkg/sql.(*planVisitor).visitInternal at line 297
pkg/sql/walk.go in pkg/sql.(*planVisitor).visit at line 79
pkg/sql/walk.go in pkg/sql.walkPlan at line 43
pkg/sql/plan.go in pkg/sql.startExec at line 521
pkg/sql/plan_node_to_row_source.go in pkg/sql.(*planNodeToRowSource).Start at line 147
pkg/sql/colexec/columnarizer.go in pkg/sql/colexec.(*Columnarizer).Init at line 178
pkg/sql/colflow/stats.go in pkg/sql/colflow.(*batchInfoCollector).Init at line 90
pkg/sql/colflow/flow_coordinator.go in pkg/sql/colflow.(*BatchFlowCoordinator).init.func1 at line 247
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.CatchVectorizedRuntimeError at line 92
pkg/sql/colflow/flow_coordinator.go in pkg/sql/colflow.(*BatchFlowCoordinator).init at line 246
pkg/sql/colflow/flow_coordinator.go in pkg/sql/colflow.(*BatchFlowCoordinator).Run at line 291
pkg/sql/colflow/vectorized_flow.go in pkg/sql/colflow.(*vectorizedFlow).Run at line 320
pkg/sql/distsql_running.go in pkg/sql.(*DistSQLPlanner).Run at line 695
pkg/sql/distsql_running.go in pkg/sql.(*DistSQLPlanner).PlanAndRun at line 1611
pkg/sql/distsql_running.go in pkg/sql.(*DistSQLPlanner).PlanAndRunAll at line 1334
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execWithDistSQLEngine at line 1541
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).dispatchToExecutionEngine at line 1177
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execStmtInOpenState at line 687
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execStmt.func1 at line 129
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execWithProfiling at line 2382
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execStmt at line 128
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execPortal at line 218
pkg/sql/conn_executor.go in pkg/sql.(*connExecutor).execCmd.func2 at line 1998
pkg/sql/conn_executor.go in pkg/sql.(*connExecutor).execCmd at line 2000
pkg/sql/conn_executor.go in pkg/sql.(*connExecutor).run at line 1846
pkg/sql/conn_executor.go in pkg/sql.(*Server).ServeConn at line 828
pkg/sql/pgwire/conn.go in pkg/sql/pgwire.(*conn).processCommandsAsync.func1 at line 728
Tag Value
Cockroach Release v22.2.0
Cockroach SHA: 77667a1
Platform linux amd64
Distribution CCL
Environment v22.2.0
Command server
Go Version ``
# of CPUs
# of Goroutines

Jira issue: CRDB-22335
@cockroach-teamcity cockroach-teamcity added C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report. labels Dec 12, 2022
@yuzefovich yuzefovich changed the title sentry: truncate.go:368: cannot perform TRUNCATE due to concurrent unknown mutation of type *tabledesc.mutation for mutation 1 in tabledesc.Mutable: {ID: 1993, Version: 23, ModificationTime: "1670869560.38802... sql: v22.2.0: cannot perform TRUNCATE due to concurrent unknown mutation Dec 27, 2022
@blathers-crl blathers-crl bot added the T-sql-schema-deprecated Use T-sql-foundations instead label Dec 27, 2022
@postamar
Copy link
Contributor

It seems like there's this if-else block in release-22.2 which doesn't take into account RowLevelTTL mutations so a repro should be straightforward: set a schema change job pausepoint, add row-level TTL to a table, then truncate the table. We haven't tried this out yet, will do.

@postamar
Copy link
Contributor

postamar commented Jan 17, 2023 •
edited
Loading

I've confirmed that this bug exists on master as well:

CREATE TABLE t (i INT PRIMARY KEY);
SET CLUSTER SETTING jobs.debug.pausepoints = 'schemachanger.before.exec';
ALTER TABLE t SET (ttl_expire_after = '00:10:00':::INTERVAL);
TRUNCATE TABLE t;

will fail with an internal error.

@postamar postamar added branch-master Failures and bugs on the master branch. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. branch-release-22.2 Used to mark GA and release blockers, technical advisories, and bugs for 22.2 branch-release-23.1 Used to mark GA and release blockers, technical advisories, and bugs for 23.1 labels Jan 17, 2023
@ajwerner ajwerner self-assigned this Jan 17, 2023
@ajwerner ajwerner removed the branch-release-22.2 Used to mark GA and release blockers, technical advisories, and bugs for 22.2 label Jan 17, 2023
@ajwerner ajwerner removed the release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. label Mar 8, 2023
@ajwerner ajwerner removed their assignment Mar 10, 2023
@ajwerner
Copy link
Contributor

This is seemingly just about handling the row-level TTL mutation type.

@ajwerner
Copy link
Contributor

@chengxiong-ruan, can you take a look at this. there's no real urgency, but it'd be nice to not have this bug.

chengxiong-ruan added a commit to chengxiong-ruan/cockroach that referenced this issue Mar 13, 2023
@chengxiong-ruan
sql: check row level ttl change before truncating a table
7beb7ac 
Fixes: cockroachdb#93443

Release note (sql change): This commit fixed a bug where crdb
paniced wehn user tried to truncate a table which is has an
ongoing row level ttl change. We still don't support table
truncates in this scenario, but a more gentle unimplemented
error is returned instead of panic.
@chengxiong-ruan chengxiong-ruan mentioned this issue Mar 13, 2023
Merged
craig bot pushed a commit that referenced this issue Mar 14, 2023
@tbg @chengxiong-ruan
@jayshrivastava @rickystewart
Merge #98481 #98537 #98575 #98582
3b326c3 
98481: kvserver: revert recent changes to reproposals r=pavelkalinnikov a=tbg

Reverts #97606, #97564, #94825, #94633.

- Revert "kvserver: disable assertion 'finished proposal inserted'"
- Revert "kvserver: narrow down 'finishing a proposal with outstanding reproposal'"
- Revert "kvserver: fill gaps in comment near tryReproposeWithNewLeaseIndex"
- Revert "kvserver: hoist early return out of tryReproposeWithNewLeaseIndex"
- Revert "fixup! kvserver: prevent finished proposal from being present in proposals map"
- Revert "kvserver: prevent finished proposal from being present in proposals map"
- Revert "kvserver: improve reproposal assertions and documentation"

Closes #97973.

Epic: CRDB-25287
Release Note: none


98537: sql: check row level ttl change before truncating a table r=chengxiong-ruan a=chengxiong-ruan

Fixes: #93443

Release note (sql change): This commit fixed a bug where crdb paniced wehn user tried to truncate a table which is has an ongoing row level ttl change. We still don't support table truncates in this scenario, but a more gentle unimplemented error is returned instead of panic.

98575: cdc: use int64 for emitted bytes telemetry r=miretskiy a=jayshrivastava

Previously, the stored `emitted_bytes` field was an int32, which can hold a maximum value of 2.1GB. This value is too small because the logging period is 24h and changefeeds can emit much more than 2.1GB in 24h. This change updates the field to be an int64, which solves this problem.

Epic: None
Release note: None

98582: ci: allow-list `BUILD_VCS_NUMBER` env var in cloud unit tests r=jlinder a=rickystewart

This job was filing issues linking to the wrong commit.

Epic: none
Release note: None

Co-authored-by: Tobias Grieger <tobias.b.grieger@gmail.com>
Co-authored-by: Chengxiong Ruan <chengxiongruan@gmail.com>
Co-authored-by: Jayant Shrivastava <jayants@cockroachlabs.com>
Co-authored-by: Ricky Stewart <rickybstewart@gmail.com>
@craig craig bot closed this as completed in 05f5bf2 Mar 14, 2023
chengxiong-ruan added a commit to chengxiong-ruan/cockroach that referenced this issue Mar 14, 2023
@chengxiong-ruan
sql: check row level ttl change before truncating a table
8707c8c 
Fixes: cockroachdb#93443

Release note (sql change): This commit fixed a bug where crdb
paniced wehn user tried to truncate a table which is has an
ongoing row level ttl change. We still don't support table
truncates in this scenario, but a more gentle unimplemented
error is returned instead of panic.
@chengxiong-ruan chengxiong-ruan mentioned this issue Mar 14, 2023
Merged
This was referenced Mar 15, 2023
Closed
Open
@exalate-issue-sync exalate-issue-sync bot added T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions) and removed T-sql-schema-deprecated Use T-sql-foundations instead labels May 10, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@chengxiong-ruan chengxiong-ruan

Labels
branch-master Failures and bugs on the master branch. branch-release-23.1 Used to mark GA and release blockers, technical advisories, and bugs for 23.1 C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report. T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions)
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

sql: check row level ttl change before truncating a table chengxiong-ruan/cockroach
4 participants
@ajwerner @postamar @chengxiong-ruan @cockroach-teamcity