Blockchain download gets stuck

[Azulan]

On Win7x64, the blockchain download stopped acquiring new blocks after block 103520 or so. Killing the process and restarting it resumed normal downloading operations.

[davecgh]

Thanks for the report. We've seen this happening every so often as well and are working to resolve it.

[Azulan]

Also, unrelated, it is crazy slow and thrashes my disk to the point of almost unusability.

[davecgh]

Can you elaborate some? Do you mean block download speed?

Keep in mind that if you're connected a slow peer, you'll have really slow download speeds. You could try ctrl+c and restart to choose a different peer. We don't yet have to code to detect "slow" peers and try to switch to syncing from a faster one, so if you get stuck with a slow peer, it would definitely have slow speeds. Is that what you're experiencing?

As far as disk trashing, none of us have experienced that, however, the next release is switching the default db back end to leveldb, which is less intensive on the disk. The SQLite db is definitely disk intensive - however it shouldn't be the point of unusability.

[Azulan]

Download speed spikes at 10s intervals to max out my connection so it's not necessarily slow peers. I suppose the bottleneck is my hard drive and the db. As it stands, I've left btcd running for a number of hours and it still has not completed the full blockchain download. The satoshi client would have finished at this point on my machine. I suspect disk activity is something particular to my system though. It has been acting wonky lately.

Thanks for all of your work on this implementation though. I'm grateful for increased diversity in the ecosystem.

[davecgh]

Thanks for the feedback. There is definitely some room for optimization (only an alpha client right now).

I'm curious how leveldb would work for you. I know that on my system, I'm able to get the entire block chain with full verification in ~18 hours with leveldb versus ~36-40 with SQLite, so it's a pretty significant difference. The main reason it isn't the default in the current release is the goleveldb driver had some bugs that we were working with the goleveldb authors to resolve.

[vessenes]

I've been downloading and processing from a local bitcoind for 24 hours, and am at block 233042. This is on an SSD with sqlite. Definitely slower than bitcoin qt. I'm excited to be done syncing!

You might consider distributing a sqlite database as a torrent, or leveldb. The chain is verifiable, so you wouldn't need to overly worry about correctness.

[vessenes]

@davecgh Can you say more about these bugs? sqlite is killing me; I'm trying to figure out if I should turn it off and recompile with leveldb support.

[davecgh]

I believe all of the bugs with leveldb are remedied. The next release (I'm guessing will be Friday) will have leveldb enabled by default. The code is already in, so if you're running from source, you can follow the directions in the README on the front page to update (it's a single command). Windows users would obviously need to wait for the MSI.

This, unfortunately does mean a redownload of the block database. Typically, I prefer for changes such as these to be automatic from a user perspective, but since it's still alpha level and SQLite is going to ultimately be removed altogether, we don't think it's worth writing a bunch of migration code. As the product matures and is finally production ready, any changes like this would automatically update as needed.

[davecgh]

Also, back on the original topic of this issue. We believe we have identified the sync issues and are reworking the code to improve things.

[dajohi]

@vessenes If you'd like to pull the latest btcd and its deps, the default is now leveldb.

[vessenes]

@dajohi Thanks!

[typeless]

Just updated and tested it, the download rate is over 3Mb/s on my openSUSE machine. (2 days ago it can't even exceed 10Kb/s)

[davecgh]

@typeless Thanks for the update. I'm glad to hear the improvements we've made already are noticeable. We have a few more fixes in the pipeline for better handling of choosing a sync peer, peer timeout handling, and a couple other things that should continue to improve things as well.

[shripadk]

Cross-posting from here: https://blog.conformal.com/btcd-not-your-moms-bitcoin-daemon/#comment-265

I had similar issues with syncing (testnet3). Took almost 5 hours to sync from a local bitcoind instance. The second time I tried it crashed with a "unexpected EOF" error. I just resynced again. Hopefully this time it won't crash

➜ ~GOPATH ./bin/btcd -C /Users/shripadk/Bitcoin-btcd/bitcoin.conf
[21:02:27 2013-10-12] [INF] [BTCD] [BMGR] Loading block database from ‘/Users/shripadk/.btcd/data/testnet/blocks_leveldb’
[21:02:27 2013-10-12] [ERR] [BTCD] unexpected EOF

Here is the log output. Any idea why the db got corrupted?

➜ ~GOPATH cat ~/.btcd/data/testnet/blocks_leveldb/LOG
=============== Oct 12, 2013 (IST) ===============
21:04:41.482844 db@open opening
21:04:41.483848 journal@recovery F·1
21:04:41.483859 journal@recovery recovering @2791
21:04:41.484765 journal@drop journal-2791 S·104B “length overflows block”

OS: MacOSX 10.7.2
Go version: go version go1.1.2 darwin/amd64

[typeless]

@shripadk I tried on MacBook Air but couldn't reproduce the problem. (No .conf file supplied)
OS: MacOSX 10.7.5
Go version: go version devel +a10c7a350ab5 Sat Oct 12 18:40:41 2013 -0400 darwin/amd64

[davecgh]

@shripadk Can you make a separate issue for this? Yours is a separate issue than this issue. I'd like to keep different issues separate if possible. Once you've done that, I'll have the main btcdb dev work with you on the issue.

[shripadk]

@davecgh Done. #18

[davecgh]

We believe this should be fixed as of commit 5a9cc91. I'm going to leave the issue open for a bit to allow some testing against it.

[PRabahy]

Is that commit rolled into the latest release (0.3.1-alpha)? I uninstalled, deleted \roaming\btcd, reinstalled the new version, and keep getting stuck syncing.

I still see PEER messages in the windows, but don't see any BMGR messages.

[davecgh]

@prab:

It is not. Here is the list of commits since the 0.3.1-alpha release: BTCD_0_3_1_ALPHA...master

The commit that fixes this is 5a9cc91 on Oct 17. 0.3.1-alpha was released on Oct 15. I will likely be rolling another release tomorrow.

[davecgh]

The fix for this is included in the btcd v0.3.2-alpha release (https://github.com/conformal/btcd/releases/tag/BTCD_0_3_2_ALPHA)

[Azulan]

I can verify that the blockchain download no longer gets stuck. It remains slow though.

[davecgh]

@Azulan Thanks for the confirmation. I'll be creating a series of issues (enhancements) for various optimizations we have planned to speed things up so everyone can track progress on that front. I'm a firm believer in making it work, then making it fast! That is to say implement a proper and stable design first, then identify and address the areas that need to be sped up through profiling.