lightning local backend panic when manually stopped

[glorv]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. What did you do? If possible, provide a recipe for reproducing the error.
   Use lightning local backend to load data. Before finish running, manually stop lightning by kill lightning-pid.

There are panic logs in lightning_:

unexpected fault address 0x7f30e009b1cc
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x1 addr=0x7f30e009b1cc pc=0x191cf2a]

goroutine 16723 [running]:
runtime.throw(0x2076c94, 0x5)
        /usr/local/go/src/runtime/panic.go:1116 +0x72 fp=0xc006cd2f70 sp=0xc006cd2f40 pc=0x439bd2
runtime.sigpanic()
        /usr/local/go/src/runtime/signal_unix.go:749 +0x405 fp=0xc006cd2fa0 sp=0xc006cd2f70 pc=0x450385
github.com/cockroachdb/pebble/internal/arenaskl.(*Skiplist).findSpliceForLevel(0xc01c092530, 0xc00d7e4c4d, 0x4a, 0xf3b3, 0xe582ee801, 0x9, 0x7f30e39c5344, 0x7f30e39c5344, 0x7f30ef185ca0, 0x0)
        /root/go/pkg/mod/github.com/cockroachdb/pebble@v0.0.0-20201023120638-f1224da22976/internal/arenaskl/skl.go:413 +0xca fp=0xc006cd3018 sp=0xc006cd2fa0 pc=0x191cf2a
github.com/cockroachdb/pebble/internal/arenaskl.(*Skiplist).findSplice(0xc01c092530, 0xc00d7e4c4d, 0x4a, 0xf3b3, 0xe582ee801, 0xc006cd3350, 0xc00d7e4c99)
        /root/go/pkg/mod/github.com/cockroachdb/pebble@v0.0.0-20201023120638-f1224da22976/internal/arenaskl/skl.go:390 +0x13c fp=0xc006cd30a0 sp=0xc006cd3018 pc=0x191cc5c
github.com/cockroachdb/pebble/internal/arenaskl.(*Skiplist).addInternal(0xc01c092530, 0xc00d7e4c4d, 0x4a, 0xf3b3, 0xe582ee801, 0xc00d7e4c98, 0x1, 0xf368, 0xc006cd3350, 0x0, ...)
        /root/go/pkg/mod/github.com/cockroachdb/pebble@v0.0.0-20201023120638-f1224da22976/internal/arenaskl/skl.go:175 +0x85 fp=0xc006cd3138 sp=0xc006cd30a0 pc=0x191c225
github.com/cockroachdb/pebble/internal/arenaskl.(*Inserter).Add(...)
        /root/go/pkg/mod/github.com/cockroachdb/pebble@v0.0.0-20201023120638-f1224da22976/internal/arenaskl/skl.go:98
github.com/cockroachdb/pebble.(*memTable).apply(0xc01c092500, 0xc00343a2d0, 0xe582eb3, 0xc01c092500, 0x0)
        /root/go/pkg/mod/github.com/cockroachdb/pebble@v0.0.0-20201023120638-f1224da22976/mem_table.go:211 +0x249 fp=0xc006cd34a8 sp=0xc006cd3138 pc=0x1968a49
github.com/cockroachdb/pebble.(*DB).commitApply(0xc0009c9400, 0xc00343a2d0, 0xc01c092500, 0x0, 0xc00343a390)
        /root/go/pkg/mod/github.com/cockroachdb/pebble@v0.0.0-20201023120638-f1224da22976/db.go:579 +0x5f fp=0xc006cd34f0 sp=0xc006cd34a8 pc=0x194941f
github.com/cockroachdb/pebble.(*DB).commitApply-fm(0xc00343a2d0, 0xc01c092500, 0xc006cd3500, 0xc01c092500)
        /root/go/pkg/mod/github.com/cockroachdb/pebble@v0.0.0-20201023120638-f1224da22976/db.go:574 +0x3e fp=0xc006cd3528 sp=0xc006cd34f0 pc=0x199781e

goroutine 16202 [semacquire]:
sync.runtime_SemacquireMutex(0xc000fbe334, 0x407c00, 0x1)
        /usr/local/go/src/runtime/sema.go:71 +0x47
sync.(*Mutex).lockSlow(0xc000fbe330)
        /usr/local/go/src/sync/mutex.go:138 +0x105
sync.(*Mutex).Lock(...)
        /usr/local/go/src/sync/mutex.go:81
github.com/cockroachdb/pebble.(*commitPipeline).prepare(0xc000fbd300, 0xc00d1983c0, 0xc0009e3500, 0xc00243eef8, 0x885bc2dce25cd501, 0x880000000040f9f0)
        /root/go/pkg/mod/github.com/cockroachdb/pebble@v0.0.0-20201023120638-f1224da22976/commit.go:364 +0x205
github.com/cockroachdb/pebble.(*commitPipeline).Commit(0xc000fbd300, 0xc00d1983c0, 0x1d56800, 0x47e465, 0x27)
        /root/go/pkg/mod/github.com/cockroachdb/pebble@v0.0.0-20201023120638-f1224da22976/commit.go:253 +0x73
github.com/cockroachdb/pebble.(*DB).Apply(0xc0009c9400, 0xc00d1983c0, 0xc0009e364f, 0x4a, 0xc0183bda50)
        /root/go/pkg/mod/github.com/cockroachdb/pebble@v0.0.0-20201023120638-f1224da22976/db.go:556 +0x110
github.com/cockroachdb/pebble.(*Batch).Commit(...)
        /root/go/pkg/mod/github.com/cockroachdb/pebble@v0.0.0-20201023120638-f1224da22976/batch.go:727
github.com/pingcap/tidb-lightning/lightning/backend.(*local).WriteRows(0xc00033f600, 0x240d0e0, 0xc000f71da0, 0xd3551380c4e0e7ec, 0x3887c65e03fbd598, 0xc0004d36c0, 0x14, 0x0, 0x0, 0x0, ...)
        /shared-nvme-disks/disk0/ansible-deploy/gl/tidb-lightning/lightning/backend/local.go:1186 +0x2c8
github.com/pingcap/tidb-lightning/lightning/backend.(*OpenedEngine).WriteRows(0xc000f6bd40, 0x240d0e0, 0xc000f71da0, 0x0, 0x0, 0x0, 0x23f00e0, 0xc0252d40a0, 0x0, 0x0)
        /shared-nvme-disks/disk0/ansible-deploy/gl/tidb-lightning/lightning/backend/backend.go:273 +0x173
github.com/pingcap/tidb-lightning/lightning/restore.(*chunkRestore).deliverLoop(0xc00add1520, 0x240d0e0, 0xc000f71da0, 0xc00bc0a360, 0xc000bd9e60, 0xc00000013e, 0xc000c31880, 0xc000f6bd40, 0xc0002fa480, 0x0, ...)
        /shared-nvme-disks/disk0/ansible-deploy/gl/tidb-lightning/lightning/restore/restore.go:1713 +0xad1
github.com/pingcap/tidb-lightning/lightning/restore.(*chunkRestore).restore.func2(0xc00f7a7b60, 0xc00add1520, 0x240d0e0, 0xc000f71da0, 0xc00bc0a360, 0xc000bd9e60, 0x13e, 0xc000c31880, 0xc000f6bd40, 0xc0002fa480)
        /shared-nvme-disks/disk0/ansible-deploy/gl/tidb-lightning/lightning/restore/restore.go:1909 +0xdd
created by github.com/pingcap/tidb-lightning/lightning/restore.(*chunkRestore).restore
        /shared-nvme-disks/disk0/ansible-deploy/gl/tidb-lightning/lightning/restore/restore.go:1907 +0x245

goroutine 1 [runnable]:
github.com/cockroachdb/pebble/internal/manual._Cfunc_free(0x7f3210019c00)
        _cgo_gotypes.go:59 +0x45
github.com/cockroachdb/pebble/internal/manual.Free.func1(0x7f3210019c00)
        /root/go/pkg/mod/github.com/cockroachdb/pebble@v0.0.0-20201023120638-f1224da22976/internal/manual/manual.go:58 +0x4d
github.com/cockroachdb/pebble/internal/manual.Free(0x7f3210019c00, 0x348, 0x348)
        /root/go/pkg/mod/github.com/cockroachdb/pebble@v0.0.0-20201023120638-f1224da22976/internal/manual/manual.go:58 +0x45
github.com/cockroachdb/pebble/internal/cache.robinHoodEntries.free(...)
        /root/go/pkg/mod/github.com/cockroachdb/pebble@v0.0.0-20201023120638-f1224da22976/internal/cache/robin_hood.go:75
github.com/cockroachdb/pebble/internal/cache.(*robinHoodMap).free(0xc000d38b80)
        /root/go/pkg/mod/github.com/cockroachdb/pebble@v0.0.0-20201023120638-f1224da22976/internal/cache/robin_hood.go:159 +0x58
github.com/cockroachdb/pebble/internal/cache.(*shard).Free(0xc000d38b20)
        /root/go/pkg/mod/github.com/cockroachdb/pebble@v0.0.0-20201023120638-f1224da22976/internal/cache/clockpro.go:248 +0xb7
github.com/cockroachdb/pebble/internal/cache.(*Cache).Unref(0xc0002360f0)
        /root/go/pkg/mod/github.com/cockroachdb/pebble@v0.0.0-20201023120638-f1224da22976/internal/cache/clockpro.go:642 +0x65
github.com/cockroachdb/pebble.(*DB).Close(0xc0009c9400, 0x0, 0x0)
        /root/go/pkg/mod/github.com/cockroachdb/pebble@v0.0.0-20201023120638-f1224da22976/db.go:926 +0x3ef
github.com/pingcap/tidb-lightning/lightning/backend.(*LocalFile).Close(...)
        /shared-nvme-disks/disk0/ansible-deploy/gl/tidb-lightning/lightning/backend/local.go:110
github.com/pingcap/tidb-lightning/lightning/backend.(*local).Close.func1(0x1f469c0, 0xc0007f4a00, 0x1dc0080, 0xc000c26f00, 0x1b93801)
        /shared-nvme-disks/disk0/ansible-deploy/gl/tidb-lightning/lightning/backend/local.go:347 +0x45
sync.(*Map).Range(0xc00033f600, 0x21a2ca8)
        /usr/local/go/src/sync/map.go:345 +0x1d9
github.com/pingcap/tidb-lightning/lightning/backend.(*local).Close(0xc00033f600)
        /shared-nvme-disks/disk0/ansible-deploy/gl/tidb-lightning/lightning/backend/local.go:346 +0x51
github.com/pingcap/tidb-lightning/lightning/backend.Backend.Close(...)
        /shared-nvme-disks/disk0/ansible-deploy/gl/tidb-lightning/lightning/backend/backend.go:192
github.com/pingcap/tidb-lightning/lightning/restore.(*RestoreController).Close(0xc0002fa480)
        /shared-nvme-disks/disk0/ansible-deploy/gl/tidb-lightning/lightning/restore/restore.go:245 +0x34
github.com/pingcap/tidb-lightning/lightning.(*Lightning).run(0xc000044280, 0x240d060, 0xc000122000, 0xc000400e00, 0x0, 0x0, 0x0, 0x0)
        /shared-nvme-disks/disk0/ansible-deploy/gl/tidb-lightning/lightning/lightning.go:311 +0xcac
github.com/pingcap/tidb-lightning/lightning.(*Lightning).RunOnce(0xc000044280, 0x240d060, 0xc000122000, 0xc000400e00, 0x0, 0x0, 0x0, 0x10, 0xc000000180)
        /shared-nvme-disks/disk0/ansible-deploy/gl/tidb-lightning/lightning/lightning.go:191 +0xee
main.main.func2(0xc00116ac00, 0xc000044280, 0x0, 0xc000044280)
        /shared-nvme-disks/disk0/ansible-deploy/gl/tidb-lightning/cmd/tidb-lightning/main.go:86 +0xb1
main.main()
        /shared-nvme-disks/disk0/ansible-deploy/gl/tidb-lightning/cmd/tidb-lightning/main.go:88 +0x58f

2. What did you expect to see?

3. What did you see instead?

4. Versions of the cluster

   • TiDB-Lightning version (run tidb-lightning -V):

     (paste TiDB-Lightning version here)
     

   • TiKV-Importer version (run tikv-importer -V)

     (paste TiKV-Importer version here)
     

   • TiKV version (run tikv-server -V):

     (paste TiKV version here)
     

   • TiDB cluster version (execute SELECT tidb_version(); in a MySQL client):

     (paste TiDB cluster version here)
     

   • Other interesting information (system version, hardware config, etc):

5. Operation logs

   • Please upload tidb-lightning.log for TiDB-Lightning if possible
   • Please upload tikv-importer.log from TiKV-Importer if possible
   • Other interesting logs

6. Configuration of the cluster and the task

   • tidb-lightning.toml for TiDB-Lightning if possible
   • tikv-importer.toml for TiKV-Importer if possible
   • inventory.ini if deployed by Ansible

7. Screenshot/exported-PDF of Grafana dashboard or metrics' graph in Prometheus for TiDB-Lightning if possible