Add support for variable length headers

[erasmospunk]

In order to support coins like Dogecoin, Namecoin (AuxPow #127, #83) and Zcash that hash a different header format.

The common pattern in the above coins is that the header is of variable size. The way I quickly solved it for Zcash using the original electrum-server and set the header size to 2048 bytes and filled the unused space with 0x00. This is not very efficient as space is wasted but it does the job.

An alternative is to use db backend to save the headers as height -> header_bytes. However this will complicate the serving the headers for the blockchain.block.get_chunk command.

One simple fix it to store the headers as before but additionally store the header offset and size in db as height -> header_offset: u32, header_size: u32.

We would need to modify the Coin::header_offset and ::header_len by adding an optional Storage instance so that the specific coin type can calculate it's own header size. I am not sure how to handle this from the async point of view as the disk access will lock the thread.

Let me know what you think.

[erasmospunk]

It seems that the LogicalFile for saving the header_offset and header_size values is the better option.

[kyuupichan]

if you're saving a linear array, which we surely are here, then yes the logical file setup is the way to go. ElectrumX uses that for tx counts by block and for headers themselves.

[kyuupichan]

I think only saving header offsets is enough. Getting two in a row to determine the size is not going to be an issue.

[erasmospunk]

@kyuupichan you mean to parse the header to determine the size? Yeah, that is a good compromise. Dogecoin that currently has ~1600000 blocks, the extra data we will store is only 6.4MiB.

Alright, will get to work.

[kyuupichan]

Maybe we're misunderstanding each other. My point was that there's no point storing an array of offsets and lengths for each header - if you have an array of offsets, you get the length by subtracting the next value from the current.

This is what I do for tx counts in ElectrumX at present.

[kyuupichan]

@erasmospunk and I store the full array in RAM; as long as it doesn't exceed 4 bytes per block I think that's quite reasonable to do for header offsets too.

[erasmospunk]

@kyuupichan now I understood what you mean, I see that header_len is calculated from header_offset only.

[erasmospunk]

@kyuupichan some initial work here that fixes Namecoin and Dogecoin. Next I will try to support Zcash.

Let me know if some things should be done differently.

ps. How to calculate TX_PER_BLOCK?

[kyuupichan]

TX_PER_BLOCK is just used to estimate sync times in logs for the user. For bitcoin I made it the average TX count of the most recent 5k blocks or something like that. Doesn't have to be correct, it will just give worse ETAs, which are only very roughly correct anyway. Bitcoin testnet is a mess and giving sane ETAs there is almost impossible.

[erasmospunk]

Just force pushed the Zcash support

[erasmospunk]

Created the pull request #140, will close this issue once merged.