Bug Report: possible panic on closed channel in Online DDL's rename phase

[shlomi-noach]

Overview of the Issue

In endtoend's onlineddl_vrepl_stress tests, we caught a panic caused by send on closed channel:

goroutine 612856 [running]:
panic({0x1ee2880, 0x29d3000})
        runtime/panic.go:987 +0x3bb fp=0xc001387ec8 sp=0xc001387e08 pc=0x438f5b
runtime.chansend(0xc001f59ec0, 0xc001387fc0, 0x1, 0xc001387f50?)
        runtime/chan.go:283 +0x5e6 fp=0xc001387f50 sp=0xc001387ec8 pc=0x406346
runtime.chansend1(0xc001334750?, 0x29f44b0?)
        runtime/chan.go:145 +0x1d fp=0xc001387f80 sp=0xc001387f50 pc=0x405d3d
vitess.io/vitess/go/vt/vttablet/onlineddl.(*Executor).cutOverVReplMigration.func7()
        vitess.io/vitess/go/vt/vttablet/onlineddl/executor.go:973 +0x5e fp=0xc001387fe0 sp=0xc001387f80 pc=0x16692de
runtime.goexit()
        runtime/asm_amd64.s:1598 +0x1 fp=0xc001387fe8 sp=0xc001387fe0 pc=0x470a01
created by vitess.io/vitess/go/vt/vttablet/onlineddl.(*Executor).cutOverVReplMigration
        vitess.io/vitess/go/vt/vttablet/onlineddl/executor.go:971 +0x12e5

Caused by this line:

vitess/go/vt/vttablet/onlineddl/executor.go

Line 976 in 4d9d94c

renameCompleteChan <- err

Problem is, this writer runs in a goroutine, and the channel is closed by this deferred function:

https://github.com/vitessio/vitess/blob/4d9d94c10e3db475523e2f45ea3e852051dcd619/go/vt/vttablet/onlineddl/executor.go#L865C2-L866

Which means any time the goroutine executes after the main function cutOverVReplMigration() completes, there will be (race condition permitting) a panic.

Reproduction Steps

Binary Version

-

Operating System and Environment details

-

Log Fragments

No response