Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Chain.syncTree error – hsd doesn't start #793

Closed
handshake-enthusiast opened this issue Jan 12, 2023 · 21 comments
Closed

Chain.syncTree error – hsd doesn't start #793

handshake-enthusiast opened this issue Jan 12, 2023 · 21 comments

Comments

@handshake-enthusiast
Copy link
Contributor

hsd (version 5.0.0-rc.1) stopped working without a clear guidance how to fix.

Log:

$ ./hsd/bin/hsd --no-wallet --prefix hsd_data/
[info] (blockstore) Opening FileBlockStore...
[info] (chain) Chain is loading.
[info] (chain) Checkpoints are enabled.
[info] (chaindb) Opening ChainDB...
[debug] (chain-migrations) There are no migrations pending. last id: 3
[debug] (chain-migrations) Last migration 3, last available migration: 3
[info] (chain-migrations) There are 0 migrations.
[info] (chaindb) ChainDB successfully loaded.
[info] (chaindb) Chain State: hash=00000000000000004299bc1f4494d678190fac05070e202e11c12d4ee1d26084 tx=8547490 coin=13503258 value=532458651.308212 burned=27898345.133312.
[info] (chaindb) Tree Root: f611a3db097f2c8f2e5b325c39d180f41eded1af8f16796650df06d0f1a7cc66.
[info] (chain) Chain Height: 140060
[debug] (chain) Memory: rss=97mb, js-heap=18/25mb native-heap=71mb
[info] (chain) Synchronizing Tree with block history...
AssertionError [ERR_ASSERTION]: Assertion failed.
    at Chain.syncTree (./hsd/lib/blockchain/chain.js:198:7)
    at async Chain.open (./hsd/lib/blockchain/chain.js:108:9)
    at async FullNode.open (./hsd/lib/node/fullnode.js:293:5)
    at async ./hsd/bin/node:75:3

hsd.conf contains:

index-address: true
index-tx: true
@pinheadmz
Copy link
Member

Hm that's a pretty critical error, there are blocks missing from the database. What else can you tell us? What version were you upgrading from? Is this a bob wallet instance? Do you know if you have pruned the chain or compacted the urkel tree?

For sanity also please check that you have adequate disk space and maybe even post some of the log from the last time you successfully ran hsd in this configuration

@handshake-enthusiast
Copy link
Contributor Author

What version were you upgrading from?

4.0.2

Is this a bob wallet instance?

No, it's on Raspberry Pi which was working all the time, but may be some power outage happened.

Do you know if you have pruned the chain or compacted the urkel tree?

I haven't done anything like that on purpose for sure.

@handshake-enthusiast
Copy link
Contributor Author

For sanity also please check that you have adequate disk space

I have enough disk space for sure.

and maybe even post some of the log from the last time you successfully ran hsd in this configuration

Unfortunately, this hsd is normally running like a daemon (with --daemon setting), so there are no debug level logs like the last part:

AssertionError [ERR_ASSERTION]: Assertion failed.
    at Chain.syncTree (./hsd/lib/blockchain/chain.js:198:7)
    at async Chain.open (./hsd/lib/blockchain/chain.js:108:9)
    at async FullNode.open (./hsd/lib/node/fullnode.js:293:5)
    at async ./hsd/bin/node:75:3

Here are the latest logs.

@handshake-enthusiast
Copy link
Contributor Author

What else can you tell us?

May be some corruption was caused by a power outage because of harsh weather conditions yesterday. I haven't even noticed. As you can find in logs [D:2023-01-11T11:32:20Z] (peer) Peer has not responded to ping (50.116.35.178:12038). is the last entry for 2023-01-11.

@handshake-enthusiast
Copy link
Contributor Author

handshake-enthusiast commented Jan 12, 2023 •
edited
Loading

so there are no debug level logs like the last part

I may be technically incorrect here. As stated on https://hsd-dev.org/guides/config.html log-level is debug by default. Errors like the following are not in logs:

AssertionError [ERR_ASSERTION]: Assertion failed.
    at Chain.syncTree (./hsd/lib/blockchain/chain.js:198:7)
    at async Chain.open (./hsd/lib/blockchain/chain.js:108:9)
    at async FullNode.open (./hsd/lib/node/fullnode.js:293:5)
    at async ./hsd/bin/node:75:3

@pinheadmz
Copy link
Member

latest logs

[D:2023-01-11T11:32:20Z] (peer) Peer has not responded to ping (50.116.35.178:12038).
[I:2023-01-12T07:29:21Z] (blockstore) Opening FileBlockStore...

Is this when the power went out?

@pinheadmz
Copy link
Member

Since the v5 migration is wallet only and you are running without a wallet, you could try resetting back to hsd v4.0.2 and restarting. What kind of disk are you storing the hsd data on? If its the Pi microSD card, I'd assume right away there is data corruption. I've been running btc and hns full nodes on Pis for years, and always use a USB3 external SSD for data.

@handshake-enthusiast
Copy link
Contributor Author

Is this when the power went out?

Hm, as far as I remember – yes, likely, around that time there was a power outage for sure. I could also turn it off myself and forgot to turn it on. It should automatically be started on boot (and was normally automatically starting). As I see it tried to autostart early today, but I suppose it got that error and couldn't proceed.

@handshake-enthusiast
Copy link
Contributor Author

What kind of disk are you storing the hsd data on?

It's an external SSD.

USB3 external SSD for data

USB 2.0 in my case https://www.raspberrypi.com/products/raspberry-pi-3-model-b-plus/

@handshake-enthusiast
Copy link
Contributor Author

Since the v5 migration is wallet only and you are running without a wallet, you could try resetting back to hsd v4.0.2 and restarting.

Same error with v4.0.2.

@handshake-enthusiast
Copy link
Contributor Author

I remember having some problem when hsd didn't want to start which was resolved with deletion of some file like chain/LOCK instead of deleting the whole directories. May be this could be a solution here as well.

@pinheadmz
Copy link
Member

That is a good indication that the last shutdown was not clean

@handshake-enthusiast
Copy link
Contributor Author

Likely not the first time I get problem with "hsd doesn't start" because of something (power outage?) BTW. But the first time I haven't quickly found a solution and decided to report.

@handshake-enthusiast
Copy link
Contributor Author

That is a good indication that the last shutdown was not clean

Ah, indeed, otherwise there would be logs like "Closing FileBlockStore...". I normally stop hsd with hsd-cli rpc stop.

@handshake-enthusiast
Copy link
Contributor Author

or if your reply is related to chain/LOCK then yes, most likely.

@handshake-enthusiast
Copy link
Contributor Author

Let me know if there are any files that I can remove to fix the issue and continue running my node.

@pinheadmz
Copy link
Member

You probably have to wipe out blocks chain and tree directories and resync from genesis

@handshake-enthusiast
Copy link
Contributor Author

Ouch, it will take many days to catch up 😬 Probably because of reindexing, or 1 GB RAM, or slow listening nodes, or latency, or all together...

@pinheadmz
Copy link
Member

Yeah I know sorry. Depending on your use case you might be able to run hnsd instead

@handshake-enthusiast
Copy link
Contributor Author

I want to have an API with indexers (txs and addresses), so I suppose hsd is necessary. Anyway, thanks for your help. I'll probably try to experiment myself and report here.

@nodech
Copy link
Contributor

nodech commented Jan 29, 2023

One trick that can be used is to remove .hsd/blocks/index directory, and restart the node. It will attempt to reindex blockstore metadata from the blockstore files.

But this still needs investigation, should not be happening.
PR #688 attempts to fix that, but there's some edge case I guess.

@nodech nodech closed this as completed Aug 1, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@nodech @pinheadmz @handshake-enthusiast