Restore with BR or import data with TiDB-lightning may throw error

[leiysky]

Sometimes TiFlash will throw error when handling IngestSST raft command. The root cause is BR or lightning may lead to incorrect SST data, but we currently doesn't have way to reproduce this.

This issue is for tracking.

[leiysky]

/cc @solotzg

[solotzg]

Approximate logs in tiflash are like:

2021/4/22 17:33:55.044 region 37227 begin to ingest default cf at [term 7, index 11][thread 15]
2021/4/22 17:33:55.232 region 37227 ingest default cf kv count 51612 [thread 15]

2021/4/22 17:33:55.241 region 37227 begin to ingest default cf at [term 7, index 12][thread 15]
2021/4/22 17:33:55.252 region 37227 meet exception: found existing key in hex: .....

Lighting/BR ingested default cf into one region twice(because tikv server is down once), and tiflash thrown exception because found existing key. This behavior is legal in kv storage, but for tiflash, we can write date into storage layer only when there are matched data in default & write cf.

[solotzg]

We shall discuss solution here: https://docs.google.com/document/d/1aL5nsIUJhWSbzXWCrxCCZAtO0u5tWHsQHunE6Se5PXQ/edit

[solotzg]

It shall be fixed by pingcap/br#1089

[solotzg]

release-5.0
pingcap/br#1150
tikv/tikv#10202