more than one tikv panic repeatedly and service is unavailable when run workload and add index

[tangenta]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

tidb_enable_dist_task='on'
1、run sysbench
2、add index for one table
3、rolling restart pd

2. What did you expect to see? (Required)

Add index can success and no panic.

3. What did you see instead (Required)

tikv panic repeatedly.

{"container":"tikv","pod":"tc-tikv-2","level":"FATAL","namespace":"endless-ha-test-add-index-tps-4710408-1-283","log":"[lib.rs:510] ["assertion failed: !last_change_ts.is_zero()"] [backtrace=" 0: tikv_util::set_panic_hook::{{closure}}
 at home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/components/tikv_util/src/lib.rs:509:18
 1: <alloc::boxed::Box<F,A> as core::ops::function::Fn>::call
 at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:2032:9
 std::panicking::rust_panic_with_hook
 at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:692:13
 2: std::panicking::begin_panic_handler::{{closure}}
 at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:577:13
 3: std::sys_common::backtrace::__rust_end_short_backtrace
 at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:137:18
 4: rust_begin_unwind
 at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:575:5
 5: core::panicking::panic_fmt
 at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/panicking.rs:65:14
 6: core::panicking::panic
 at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/panicking.rs:114:5
 7: tikv::storage::txn::actions::common::next_last_change_info
 8: tikv::storage::txn::actions::prewrite::PrewriteMutation::check_for_newer_version
 at home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/src/storage/txn/actions/prewrite.rs:437:21
 9: tikv::storage::txn::actions::prewrite::prewrite
 at home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/src/storage/txn/actions/prewrite.rs:86:10
 10: tikv::storage::txn::commands::prewrite::Prewriter::prewrite
 at home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/src/storage/txn/commands/prewrite.rs:647:35
 tikv::storage::txn::commands::prewrite::Prewriter::process_write
 at home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/src/storage/txn/commands/prewrite.rs:535:19
 11: <tikv::storage::txn::commands::prewrite::Prewrite as tikv::storage::txn::commands::WriteCommand<S,L>>::process_write
 at home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/src/storage/txn/commands/prewrite.rs:250:9
 tikv::storage::txn::commands::Command::process_write
 at home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/src/storage/txn/commands/mod.rs:691:37
 12: tikv::storage::txn::scheduler::TxnScheduler<E,L>::process_write::{{closure}}::{{closure}}
 at home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/src/storage/txn/scheduler.rs:1287:21
 tikv::storage::metrics::with_perf_context::{{closure}}
 at home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/src/storage/metrics.rs:360:19
 std:🧵:local::LocalKey::try_with
 at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/local.rs:446:16
 std:🧵:local::LocalKey::with
 at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/local.rs:422:9
 tikv::storage::metrics::with_perf_context
 at home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/src/storage/metrics.rs:351:5
 tikv::storage::txn::scheduler::TxnScheduler<E,L>::process_write::{{closure}}
 at home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/src/storage/txn/scheduler.rs:1286:17
 13: <core::future::from_generator::GenFuture as core::future::future::Future>::poll
 at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/future/mod.rs:91:19
 tikv::storage::txn::scheduler::TxnScheduler<E,L>::process::{{closure}}::{{closure}}
 at home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/src/storage/txn/scheduler.rs:1177:71
 <core::future::from_generator::GenFuture as core::future::future::Future>::poll
 at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/future/mod.rs:91:19
 <resource_metering::InTags as core::future::future::Future>::poll
 at home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/components/resource_metering/src/lib.rs:266:9
 tikv::storage::txn::scheduler::TxnScheduler<E,L>::process::{{closure}}
 at home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/src/storage/txn/scheduler.rs:1191:9
 <core::future::from_generator::GenFuture as core::future::future::Future>::poll
 at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/future/mod.rs:91:19
 tikv::storage::txn::scheduler::TxnScheduler<E,L>::execute::{{closure}}
 at home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/src/storage/txn/scheduler.rs:748:54
 <core::future::from_generator::GenFuture as core::future::future::Future>::poll
 at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/future/mod.rs:91:19
 14: <tracker::tls::TrackedFuture as core::future::future::Future>::poll::{{closure}}
 at home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/components/tracker/src/tls.rs:64:23
 std:🧵:local::LocalKey::try_with
 at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/local.rs:446:16
 std:🧵:local::LocalKey::with
 at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/local.rs:422:9
 <tracker::tls::TrackedFuture as core::future::future::Future>::poll
 at home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/components/tracker/src/tls.rs:62:9
 tikv_util::yatp_pool::future_pool::PoolInner::spawn::{{closure}}
 at home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/components/tikv_util/src/yatp_pool/future_pool.rs:178:27
 <core::future::from_generator::GenFuture as core::future::future::Future>::poll
 at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/future/mod.rs:91:19
 yatp::task::future::RawTask::poll
 at rust/git/checkouts/yatp-e704b73c3ee279b6/5572a78/src/task/future.rs:59:9
 15: yatp::task::future::TaskCell::poll
 at rust/git/checkouts/yatp-e704b73c3ee279b6/5572a78/src/task/future.rs:103:9
 <yatp::task::future::Runner as yatp::pool:🏃:Runner>::handle
 at rust/git/checkouts/yatp-e704b73c3ee279b6/5572a78/src/task/future.rs:387:20
 16: <tikv_util::yatp_pool::YatpPoolRunner as yatp::pool:🏃:Runner>::handle
 at home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/components/tikv_util/src/yatp_pool/mod.rs:199:24
 17: yatp::pool::worker::WorkerThread<T,R>::run
 at rust/git/checkouts/yatp-e704b73c3ee279b6/5572a78/src/pool/worker.rs:48:13
 yatp::pool::builder::LazyBuilder::build::{{closure}}
 at rust/git/checkouts/yatp-e704b73c3ee279b6/5572a78/src/pool/builder.rs:114:25
 std::sys_common::backtrace::rust_begin_short_backtrace
 at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:121:18
 18: std:🧵:Builder::spawn_unchecked::{{closure}}::{{closure}}
 at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:551:17
 <core::panic::unwind_safe::AssertUnwindSafe as core::ops::function::FnOnce<()>>::call_once
 at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/panic/unwind_safe.rs:271:9
 std::panicking::try::do_call
 at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:483:40
 std::panicking::try
 at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:447:19
 std::panic::catch_unwind
 at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:137:14
 std:🧵:Builder::spawn_unchecked::{{closure}}
 at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:550:30
 core::ops::function::FnOnce::call_once{{vtable.shim}}
 at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:513:5
 19: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce>::call_once
 at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:2000:9
 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce>::call_once
 at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:2000:9
 std::sys::unix:🧵:Thread:🆕:thread_start
 at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:108:17
 20: start_thread
 21: __clone3
"] [location=components/txn_types/src/types.rs:611] [thread_name=sched-worker-pool-1] [thread_id=0x5]"}

4. What is your TiDB version? (Required)

9062834

[tangenta]

This issue is from tikv/tikv#16030.