resolve lock meets EpochNotMatch report newRegion's range key is not encoded

[lysu]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. What did you do?

run concurrent resolve lock and trigger tikv change region epoch at the same time.

2. What did you expect to see?

retry and execute success

3. What did you see instead?

err="newRegion's range key is not encoded: id:2 start_key:\"t\\200\\000\\000\\000\\000\\000\\001\\017\" region_epoch:<conf_ver:1 version:94 > peers:<id:3 store_id:1 > , invalid marker byte, group bytes \"t\\x80\\x00\\x00\\x00\\x00\\x00\\x01\\x0f\

github.com/pingcap/tidb/store/tikv.(*RegionCache).OnRegionEpochNotMatch
	/home/jenkins/agent/workspace/tidb_master/go/src/github.com/pingcap/tidb/store/tikv/region_cache.go:1056
github.com/pingcap/tidb/store/tikv.(*RegionRequestSender).onRegionError
	/home/jenkins/agent/workspace/tidb_master/go/src/github.com/pingcap/tidb/store/tikv/region_request.go:366
github.com/pingcap/tidb/store/tikv.(*RegionRequestSender).SendReqCtx
	/home/jenkins/agent/workspace/tidb_master/go/src/github.com/pingcap/tidb/store/tikv/region_request.go:207
github.com/pingcap/tidb/store/tikv.(*RegionRequestSender).SendReq
	/home/jenkins/agent/workspace/tidb_master/go/src/github.com/pingcap/tidb/store/tikv/region_request.go:106
github.com/pingcap/tidb/store/tikv.(*tikvStore).SendReq
	/home/jenkins/agent/workspace/tidb_master/go/src/github.com/pingcap/tidb/store/tikv/kv.go:401
github.com/pingcap/tidb/store/tikv.(*LockResolver).resolveLock
	/home/jenkins/agent/workspace/tidb_master/go/src/github.com/pingcap/tidb/store/tikv/lock_resolver.go:594
github.com/pingcap/tidb/store/tikv.(*LockResolver).resolveLocks
	/home/jenkins/agent/workspace/tidb_master/go/src/github.com/pingcap/tidb/store/tikv/lock_resolver.go:330
github.com/pingcap/tidb/store/tikv.(*LockResolver).ResolveLocks
	/home/jenkins/agent/workspace/tidb_master/go/src/github.com/pingcap/tidb/store/tikv/lock_resolver.go:285
github.com/pingcap/tidb/store/tikv.(*clientHelper).ResolveLocks
	/home/jenkins/agent/workspace/tidb_master/go/src/github.com/pingcap/tidb/store/tikv/coprocessor.go:803
github.com/pingcap/tidb/store/tikv.(*copIteratorWorker).handleCopResponse
	/home/jenkins/agent/workspace/tidb_master/go/src/github.com/pingcap/tidb/store/tikv/coprocessor.go:941
github.com/pingcap/tidb/store/tikv.(*copIteratorWorker).handleTaskOnce
	/home/jenkins/agent/workspace/tidb_master/go/src/github.com/pingcap/tidb/store/tikv/coprocessor.go:757
github.com/pingcap/tidb/store/tikv.(*copIteratorWorker).handleTask
	/home/jenkins/agent/workspace/tidb_master/go/src/github.com/pingcap/tidb/store/tikv/coprocessor.go:672
github.com/pingcap/tidb/store/tikv.(*copIteratorWorker).run
	/home/jenkins/agent/workspace/tidb_master/go/src/github.com/pingcap/tidb/store/tikv/coprocessor.go:494
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1357

4. What version of TiDB are you using? (tidb-server -V or run select tidb_version(); on TiDB)

139cb3e

[lysu]

caused by #16838

deduplicate multi-requests into one response, but EpochNotMatch handle logic will modify the response

so two goroutines modify the same struct and decode key twice and last one will meet this error

[lysu]

this question does NOT affact on any released version now.