Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

BCH Testnet synchronized crash #446

Closed
Droppix opened this issue Jun 22, 2020 · 7 comments
Closed

BCH Testnet synchronized crash #446

Droppix opened this issue Jun 22, 2020 · 7 comments

Comments

@Droppix
Copy link

Droppix commented Jun 22, 2020

Hi,

I try to install Blockbook Cash Testnet (bcash_testnet) :

  • DigitalOcean, server Ram 16GB, vCPU 6, 60GB + 100GB (volume for /coins) on Ubuntu 18.04 with 6GB swapfile

  • Blockchain Cash Testnet synced (1387806 blocks) -> OK

    • sudo make all-bcash_testnet
    • sudo apt install ./backend-bcash_0.21.0-satoshilabs-1_amd64.deb
    • sudo systemctl start backend-bcash-testnet

  • Install Blockbook and launch sync -> Error

    • sudo apt install ./blockbook-bcash-testnet_0.3.4_amd64.deb
    • sudo systemctl start blockbook-bcash-testnet

    to about > 1.310.000 blocks, the crash synchronization systematically
    result => Error internalState: database is in inconsistent state and cannot be used

    I try the solution (BTC & ZEC - "internalState: database is in inconsistent state and cannot be used" #147): (workers=1, dbcache=0)

      - Delete all database : rm db/* 
  - launch "sudo ./blockbook -blockchaincfg=/home/dev/coins/blockbook/bcash_testnet/config/blockchaincfg.json -datadir=/home/dev/coins/data/bcash_testnet/blockbook/db -sync -workers=1 -dbcache=0 -internal=:19031 -public=:19131 &"

  Result => Same error

    Also, I try re-create server with different configuration (4 times on 3 days), I have always same error.

    My question:

    1/ do you have any idea where the error is coming from, I feel like there's a block that's crashing the sync? (Looks like it crashed at the same block)

    2/ When there is an error (database is in inconsistent state and cannot be used), is there no way to repair the base instead of repeating the operation which is rather long?
    (I try blockbook -repair..., but without result)

Thanks for your help
@martinboehm
Copy link
Contributor

  1. is it BCH and not BCHSV, right? Because BCHSV testnet has some huge blocks (about 1GB) and I am not sure if your HW config is enough for that
  2. if the DB is in inconsistent state, there is no way to repair it. However, if you run sync with -workers=1, it does not put the DB to the inconsistent state and you should be able to continue after crash
  3. is there some more specific error in the blockbook log or syslog?

@Droppix
Copy link
Author

Droppix commented Jun 22, 2020

1/ Yes, it's BCH

2/ 3 logs in ..coins/blockbook/bcash_testnet/logs (ERROR, INFO, WARNING -> same date/hour)

  • the first file is (blockbook.ERROR) blockbook.bch-testnet.blockbook-bcash.log.ERROR.20200622-164428.30082
Running on machine: bch-testnet
Binary: Built with gc go1.14.2 for linux/amd64
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
E0622 16:44:28.854310   30082 blockbook.go:292] resyncIndex Corruption: block checksum mismatch: expected 708943542, got 2324967102  in /home/dev/coins/data/bcash_testnet/blockbook/db/007981.sst offset 25892417 size 6524644
  • 2nd file is blockbook.INFO (blockbook.bch-testnet.blockbook-bcash.log.INFO.20200622-164429.30127)
Log file created at: 2020/06/22 16:44:29
Running on machine: bch-testnet
Binary: Built with gc go1.14.2 for linux/amd64
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
I0622 16:44:29.435445   30127 blockbook.go:136] Blockbook: {Version:0.3.4 GitCommit:eb4e10a-dirty BuildTime:2020-06-20T10:10:42+00:00 GoVersion:go1.14.2 OSArch:linux/amd64}, debug mode false
I0622 16:44:29.439956   30127 bcashrpc.go:61] rpc: block chain testnet3
I0622 16:44:29.439982   30127 mempool_bitcoin_type.go:56] mempool: starting with 8*2 sync workers
I0622 16:44:29.439993   30127 rocksdb.go:148] rocksdb: opening /home/dev/coins/data/bcash_testnet/blockbook/db, required data version 5, cache size 536870912, max open files 16384
I0622 16:44:32.936604   30127 rocksdb.go:1696] loaded 600294 block times
W0622 16:44:32.936659   30127 blockbook.go:213] internalState: database was left in open state, possibly previous ungraceful shutdown
I0622 16:44:32.961434   30127 worker.go:1781] GetSystemInfo finished in 17.621172ms
I0622 16:44:32.961961   30127 internal.go:68] internal server: starting to listen on https://:19031
I0622 16:44:33.003763   30127 public.go:122] public server starting to listen on https://:19131
I0622 16:44:33.021006   30127 sync.go:111] resync: local at 600293 is behind
E0622 16:44:36.820724   30127 blockbook.go:292] resyncIndex Corruption: block checksum mismatch: expected 708943542, got 2324967102  in /home/dev/coins/data/bcash_testnet/blockbook/db/007981.sst offset 25892417 size 6524644
I0622 16:44:36.825216   30127 rocksdb.go:284] internalState: Corruption: block checksum mismatch: expected 708943542, got 2324967102  in /home/dev/coins/data/bcash_testnet/blockbook/db/007981.sst offset 25892417 size 6524644
I0622 16:44:36.825250   30127 rocksdb.go:287] rocksdb: close
  • (3rd file is blockbook.WARNING) blockbook.bch-testnet.blockbook-bcash.log.WARNING.20200622-164424.30082
Log file created at: 2020/06/22 16:44:24
Running on machine: bch-testnet
Binary: Built with gc go1.14.2 for linux/amd64
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
W0622 16:44:24.895204   30082 blockbook.go:213] internalState: database was left in open state, possibly previous ungraceful shutdown
E0622 16:44:28.854310   30082 blockbook.go:292] resyncIndex Corruption: block checksum mismatch: expected 708943542, got 2324967102  in /home/dev/coins/data/bcash_testnet/blockbook/db/007981.sst offset 25892417 size 6524644
  • Syslog
...
Jun 22 16:44:43 bch-testnet systemd[1]: blockbook-bcash-testnet.service: Main process exited, code=exited, status=255/n/a
Jun 22 16:44:43 bch-testnet systemd[1]: blockbook-bcash-testnet.service: Failed with result 'exit-code'.
Jun 22 16:44:44 bch-testnet systemd[1]: blockbook-bcash-testnet.service: Service hold-off time over, scheduling restart.
Jun 22 16:44:44 bch-testnet systemd[1]: blockbook-bcash-testnet.service: Scheduled restart job, restart counter is at 8927.
Jun 22 16:44:44 bch-testnet systemd[1]: Stopped Blockbook daemon (Bcash Testnet).
Jun 22 16:44:44 bch-testnet systemd[1]: Started Blockbook daemon (Bcash Testnet).
Jun 22 16:44:50 bch-testnet blockbook[30220]: E0622 16:44:50.355666   30220 blockbook.go:292] resyncIndex Corruption: block checksum mismatch: expected 708943542, got 2324967102  in /home/dev/coins/data/bcash_testnet/blockbook/db/007981.sst offset 25892417 size 6524644
Jun 22 16:44:50 bch-testnet systemd[1]: blockbook-bcash-testnet.service: Main process exited, code=exited, status=255/n/a
Jun 22 16:44:50 bch-testnet systemd[1]: blockbook-bcash-testnet.service: Failed with result 'exit-code'.
Jun 22 16:44:50 bch-testnet systemd[1]: blockbook-bcash-testnet.service: Service hold-off time over, scheduling restart.
Jun 22 16:44:50 bch-testnet systemd[1]: blockbook-bcash-testnet.service: Scheduled restart job, restart counter is at 8928.
Jun 22 16:44:50 bch-testnet systemd[1]: Stopped Blockbook daemon (Bcash Testnet).
Jun 22 16:44:50 bch-testnet systemd[1]: Started Blockbook daemon (Bcash Testnet).
Jun 22 16:44:57 bch-testnet blockbook[30267]: E0622 16:44:57.570922   30267 blockbook.go:292] resyncIndex Corruption: block checksum mismatch: expected 708943542, got 2324967102  in /home/dev/coins/data/bcash_testnet/blockbook/db/007981.sst offset 25892417 size 6524644
Jun 22 16:44:57 bch-testnet systemd[1]: blockbook-bcash-testnet.service: Main process exited, code=exited, status=255/n/a
Jun 22 16:44:57 bch-testnet systemd[1]: blockbook-bcash-testnet.service: Failed with result 'exit-code'.
Jun 22 16:44:57 bch-testnet systemd[1]: blockbook-bcash-testnet.service: Service hold-off time over, scheduling restart.
Jun 22 16:44:57 bch-testnet systemd[1]: blockbook-bcash-testnet.service: Scheduled restart job, restart counter is at 8929.
Jun 22 16:44:57 bch-testnet systemd[1]: Stopped Blockbook daemon (Bcash Testnet).
Jun 22 16:44:57 bch-testnet systemd[1]: Started Blockbook daemon (Bcash Testnet).
Jun 22 16:45:02 bch-testnet kernel: [83998.655501] [UFW BLOCK] IN=eth0 OUT= MAC=ae:19:72:0c:c3:ca:f0:4b:3a:4f:80:30:08:00 SRC=212.70.149.50 DST=142.93.97.22 LEN=60 TOS=0x00 PREC=
...

@martinboehm
Copy link
Contributor

It looks like a corruption of the RocksDB database. I have never seen this issue. What happens when you delete the database and start syncing again? Does the error happen again?
There is a possibility to start Blockbook with -repair switch, which can try to fix the DB issue but the results not certain.

@Droppix
Copy link
Author

Droppix commented Jun 23, 2020

1/ Yes. When I delete the database (rm db/*), I can do a new synchronization, but I have the impression that the crash occurs at the same block... (I've tried several times, with different scenarios.. -> same result)

2/ Also, I try option "-repair", but that doesn't help, the base is always in state :
"Error internalState: database is in inconsistent state and cannot be used"

I'm going to give it one last try, re-installing everything (4th time) but this time on Debian...

@Droppix
Copy link
Author

Droppix commented Jun 24, 2020

I create a new droplet (in DigitalOcean), 16GB RAM + 150GB SSD, but under Debian.
I went through the entire installation process (https://wiki.trezor.io/User_manual:Running_a_local_instance_of_Trezor_Wallet_backend_(Blockbook)
and... I have always same error.

I think that if you try to do the same installation (make all-bcash_testnet), you will certainly make the same error.
The crash is located in the vicinity of block number > 1.310.000.

@martinboehm
Copy link
Contributor

Hi, I successfully run the initial import of BCH Testnet.
It seems that above the height 1305000 there are many huge blocks (with over 80000 transactions). Memory usage went very high (my peak memory usage was 33GB, but I run the import in bulk mode).

Run the import with -workers=1 (disable bulk import) and increase the swap file size to 16GB, you may be able to get it synced.

If you get error "Error internalState: database is in inconsistent state and cannot be used", it means you were running it in bulk mode.

@Droppix
Copy link
Author

Droppix commented Jul 1, 2020

Hi,

Thank you for your time and feedback.
I started all over again, but this time I disabling the bulk mode from the beginning, and... it's worked:)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@Droppix @martinboehm