Reload sometimes cause blockchain rebuild

[ghost]

I have noticed many times that in some cases rebuild or stop and start options in lisk.sh cause blockchain to get rebuild from scratch.

lisk_user@order:~/lisk-main$ bash lisk.sh logs
[inf] 2017-01-31 14:23:25 | Block 10144988968756536807 loaded from: 162.243.246.43:8000 - height: 1856459
[inf] 2017-01-31 14:23:25 | Block 15681905685578667465 loaded from: 162.243.246.43:8000 - height: 1856460
[inf] 2017-01-31 14:23:25 | Block 11262089480305559844 loaded from: 162.243.246.43:8000 - height: 1856461
[inf] 2017-01-31 14:23:25 | Block 3862362197742471566 loaded from: 162.243.246.43:8000 - height: 1856462
[inf] 2017-01-31 14:23:25 | Block 16670909577409793353 loaded from: 162.243.246.43:8000 - height: 1856463
[inf] 2017-01-31 14:23:25 | Block 2848045563065873855 loaded from: 162.243.246.43:8000 - height: 1856464
[inf] 2017-01-31 14:23:25 | Block 6212083298738950197 loaded from: 162.243.246.43:8000 - height: 1856465
[inf] 2017-01-31 14:23:26 | Block 8350420218561425577 loaded from: 162.243.246.43:8000 - height: 1856466
[inf] 2017-01-31 14:23:26 | Block 16464451331469451528 loaded from: 162.243.246.43:8000 - height: 1856467
[inf] 2017-01-31 14:23:26 | Block 18354989758993067962 loaded from: 162.243.246.43:8000 - height: 1856468
[inf] 2017-01-31 14:23:26 | Block 6429496506066114952 loaded from: 162.243.246.43:8000 - height: 1856469
[inf] 2017-01-31 14:23:26 | Block 16387299214909636335 loaded from: 162.243.246.43:8000 - height: 1856470
[inf] 2017-01-31 14:23:26 | Block 8204557843670266747 loaded from: 162.243.246.43:8000 - height: 1856471
[inf] 2017-01-31 14:23:26 | Block 12795434197740053871 loaded from: 162.243.246.43:8000 - height: 1856472
[inf] 2017-01-31 14:23:26 | Block 1857446355567737484 loaded from: 162.243.246.43:8000 - height: 1856473
[inf] 2017-01-31 14:23:26 | Block 2332533409075918783 loaded from: 162.243.246.43:8000 - height: 1856474
[inf] 2017-01-31 14:23:27 | Block 7379693627132560299 loaded from: 162.243.246.43:8000 - height: 1856475
[inf] 2017-01-31 14:23:27 | Block 12850281702516288729 loaded from: 162.243.246.43:8000 - height: 1856476
[inf] 2017-01-31 14:23:27 | Block 2313979771120742871 loaded from: 162.243.246.43:8000 - height: 1856477
[inf] 2017-01-31 14:23:27 | Block 98424096216928182 loaded from: 162.243.246.43:8000 - height: 1856478
^C
lisk_user@order:~/lisk-main$ bash lisk.sh reload
√ Lisk stopped successfully.
√ Lisk started successfully.
√ Lisk is running as PID: 1371
Current Block Height: 1856507
lisk_user@order:~/lisk-main$ bash lisk.sh logs  
NOTICE:  relation "mem_round_round" already exists, skipping
NOTICE:  relation "mem_accounts2delegates" already exists, skipping
NOTICE:  relation "mem_accounts2delegates_accountId" already exists, skipping
NOTICE:  relation "mem_accounts2u_delegates" already exists, skipping
NOTICE:  relation "mem_accounts2u_delegates_accountId" already exists, skipping
NOTICE:  relation "mem_accounts2multisignatures" already exists, skipping
NOTICE:  relation "mem_accounts2multisignatures_accountId" already exists, skipping
NOTICE:  relation "mem_accounts2u_multisignatures" already exists, skipping
NOTICE:  relation "mem_accounts2u_multisignatures_accountId" already exists, skipping
[inf] 2017-01-31 14:23:49 | Rebuilding blockchain, current block height: 1

[karmacoma]

@karek314: Thanks for reporting. This issue should be fixed when issue #302 is closed.

[karmacoma]

This issue should be less apparent now we are switching from forever to pm2 for process management. forever was terminating the process too aggressively, and therefore block processing activities were exited midway through memory table writes.

The total solution is to ensure block processing is performed atomically. This work is being done as part of #449.

[karmacoma]

Closed by #490