[3/3] key, wallet: HD wallets

[div72]

Adds HD wallet support. Introduces two new RPCs: sethdseed and upgradewallet(previously a cli argument).

This PR does not introduce BIP39/seed phrases support.

Upgrade Instructions

Old pre-HD wallets will not automatically be upgraded and will continue to work as normal. HD can be enabled for old wallets like this:

1. Unlock wallet if locked.
2. Call upgradewallet (5040101 can be given as a second argument if the wallet is not meant to be fully upgraded; does not affect anything for now.)
3. Wait. (The generation and the flush of the keys might take a few minutes.)

Note: New wallets are automatically created as HD wallets and do not need to be updated.
Note 2: Unused keys in the key pool will be removed when doing this action. When dealing with multiple out of sync wallet.dat files caution must be taken care as losing the other files can cause loss of keys/funds.

Backup/Restore Instructions

A backup of the wallet.dat file is enough as the seed is contained in the file. Alternatively the private key associated with the seed can be backed up like this:

1. Do a walletdump. (walletdump <file>)
2. Get the key marked with hdmaster=1 or hdseed=1 in the file.
3. Backup that key.

and restored with:
sethdseed true {key}

Note: This operation can also take a few minutes.
Note 2: The wallet dump files contain unencrypted private keys and should be removed after the backup.
Note 3: If the sethdseed command is used from a shell, cleaning the shell's $HISTFILE might be a good idea.

Testing checklist

• Ability to re-create private keys from a given seed.
• Old wallets are able to generate private keys without an issue.
• Receiving and sending is not impacted.

Fixes #570.
Fixes #2325.

[jamescowens]

Do we want to try to get this in Kermit's Mom?

[div72]

It's a nice to have but it carries a non-insignificant bug risk and without upgradewallet being an RPC, encrypted wallets can't be upgraded.

[jamescowens]

In general this works really well. I don't have any comments on the coding style. Based on my testing I have the following suggestions:

1. The new commit which raises the keypool (re)fill size to 1000 should only apply for wallets that have not been upgraded. Post upgrade wallets should use a much smaller keypool size because otherwise it takes 2-4 minutes to upgrade the wallet and/or generate a new one on first wallet startup and or setting a new seed. Since the BIP32 keys are deterministic, a large keypool by default post upgrade is not necessary. (Having calls take this long with no feedback is asking for trouble as well, but cutting the keypool size down should solve this problem I think.)
2. We need to make it clear in the documentation that an upgraded wallet is a HYBRID wallet. The hdmaster key will only restore the keys associated with the hdmaster. The pre-BIP32 keys must still be retained as well. (i.e. for a hybrid wallet, the actual set of keys to be retained for restoration purposes are all important pre-BIP32 keys + the hdmaster key.)
3. A hybrid wallet should be able to be restored in one step by doing importwallet on a file containing pre BIP32 keys + the hdmaster key. Right now the restoration requires two steps. Import the wallet file, where the pre-BIP32 keys are pulled in, and then do the sethdseed with the hdmaster key. (In other words the hdmaster key in the import file is actually ignored - we should have it automatically processed and do the sethdseed behind the scenes.)

Actually the hdmaster key is not ignored, rather it is imported as a normal (loose) key, which then prevents resetting the seed to that key. We need to fix this...

[jamescowens]

ACK

[jamescowens]

I will merge this right after I do the 5.4.1.0 point release.

[jamescowens]

@div72 you want to rebase this so we can merge it?