Murisi/integrate hw masp

[murisi]

Describe your changes

Made the following changes to enable hardware wallet signing of MASP Transactions:

• Allow the outsourcing of generator randomness to the hardware wallet
• Allow the generation of MASP extended full viewing keys using the hardware wallet
• Allow the client to generate proofs using proof generation keys from the hardware wallet
• Now apply signatures generated by the hardware wallet onto client build Transactions
• The following transaction types are supported by MASP hardware wallet signing:
  • Shielding transactions (only randomness parameters are taken from hardware wallet)
  • Shielded, unshielding, and IBC transactions (randomness parameters and proof generation keys are taken from hardware wallet)
  • Combined hardware wallet and software client signing where the the shielded keys are on the hardware wallet and wrapper signer keys are on the computer, or vice-versa, or both reside on the device is supported
• Adjusted the hardware and localnet genesis balances files in order to ensure that the MASP inflation controller computations are the same for both kinds of tests
• Recomputed/recalibrated the MASP test numerical expectations to work with the new adjusted balances files
• Modified the MASP test commands to enable signing to happen on the hardware wallet when NAMADA_E2E_USE_DEVICE=true
• Moved the MASP spending and viewing keys as well as payment addresses used in the the tests crate into the localnet and hardware genesis files to enable hardware wallet signing
• Separated spending and viewing keys from their birthdays in the wallet's store.toml in order to ease the manual editing of these files (as done above)
• Removed the possibility of using the hardware wallet to sign the wrapper header without signing the raw header because the former depends on the latter in the current design
• Now ensure the removal of the Builder section in sign_tx because it contains private information and because it will be rejected on the basis of not being referred to in the wrapper signature section

This PR depends upon the following being merged:

• Murisi/box impl build params masp#88 to reduce the constraints induced by the Builder API
• Murisi/zip32 support Zondax/ledger-namada#61 to generate extended full viewing keys.
• Fix signing for non-MASP IBC Transfers heliaxdev/ledger-namada#2 to fix the signing of pure IBC Txs

Outstanding issues:

• The hardware wallet applies ZIP 32 derivations to a seed obtained from doing SLIP 10 derivation with a fixed path. (This was necessary because the hardware wallet's ZIP 32 functions are unable to directly access the root seed.) This PR currently modifies namadaw's key derivation to match the hardware wallet's to ease checking that the same extended keys are derived for a given mnemonic. However, it might be desirable to revert namadaw's derivation to pure ZIP 32 if we desire interoperability with systems implementing this standard strictly. Or perhaps both types of ZIP 32 could be supported with a namadaw CLI flag to choose which derivation is desired. See a relevant discusssion at https://zecsec.com/audits/zcash-ledger-audit-report-v2.pdf#subsection.3.2 .

Checklist before merging

• If this PR has some consensus breaking changes, I added the corresponding breaking:: labels
  • This will require 2 reviewers to approve the changes

[Fraccaman]

@murisi thanks for PR! Do u mind rebasing this on main?

[codecov]

Codecov Report

Attention: Patch coverage is 49.43820% with 135 lines in your changes missing coverage. Please review.

Project coverage is 74.49%. Comparing base (cc531ec) to head (79106c5).
Report is 106 commits behind head on main.

Files with missing lines	Patch %	Lines
crates/wallet/src/store.rs	25.00%	60 Missing ⚠️
crates/wallet/src/lib.rs	35.55%	29 Missing ⚠️
crates/wallet/src/keys.rs	32.14%	19 Missing ⚠️
crates/sdk/src/tx.rs	73.91%	6 Missing ⚠️
crates/apps_lib/src/wallet/defaults.rs	0.00%	4 Missing ⚠️
crates/sdk/src/lib.rs	0.00%	4 Missing ⚠️
crates/core/src/masp.rs	86.95%	3 Missing ⚠️
crates/tx/src/types.rs	84.21%	3 Missing ⚠️
crates/apps_lib/src/config/genesis/transactions.rs	0.00%	2 Missing ⚠️
crates/apps_lib/src/config/genesis/utils.rs	0.00%	2 Missing ⚠️
... and 2 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3797      +/-   ##
==========================================
- Coverage   74.58%   74.49%   -0.09%     
==========================================
  Files         341      341              
  Lines      107484   107642     +158     
==========================================
+ Hits        80167    80190      +23     
- Misses      27317    27452     +135     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[tzemanovic]

I'm trying to run e2e::ledger_tests::masp_txs_and_queries with these changes and 713f954:

Something is wrong with the wrapper signature. It fails on shield --source Albert --target znam1ky620tz7z658cralqt693qpvk42wvth468zp38nqvq2apmex5rfut3dfqm2asrsqv0tc7saqje7 --token BTC --amount 20 --use-device with "Mempool validation failed: WrapperTx signature verification failed: The wrapper signature is invalid". Same thing if I try --dry-run-wrapper.

Relatedly, do you recall why we disallow --use-device with --dry-run? We allow it with --dry-run-wrapper. (it was added in #3570)

[murisi]

Thanks for looking into this PR.

Relatedly, do you recall why we disallow --use-device with --dry-run? We allow it with --dry-run-wrapper. (it was added in #3570)

I believe that combining --use_device with --dry-run is disallowed because the hardware wallet does not support displaying and signing raw transactions. And I think (at least at the time when the code was written) --dry-run causes the creation of raw transactions instead of wrapper transactions.

Something is wrong with the wrapper signature. It fails on shield --source Albert --target znam1ky620tz7z658cralqt693qpvk42wvth468zp38nqvq2apmex5rfut3dfqm2asrsqv0tc7saqje7 --token BTC --amount 20 --use-device with "Mempool validation failed: WrapperTx signature verification failed: The wrapper signature is invalid". Same thing if I try --dry-run-wrapper.

If the hardware wallet produced the wrapper signature for a Tx containing a MASP Transaction, then the issue might be due to the hardware wallet's reponse. But I need to check. I'd actually been planning to add a last commit to this PR to integrate MASP hardware wallet testing with the E2E and integration tests. This will be done by Monday evening.

[murisi]

I'm trying to run e2e::ledger_tests::masp_txs_and_queries with these changes and 713f954:

Hi @tzemanovic . All the MASP tests now work (when using a variant of the Ledger app with the above PRs merged in) except the integration::masp::masp_fee_payment ones (failure probably due to simultaneous construction of Transactions requiring the hardware wallet) and integration::masp::expired_masp_tx (failure probably due to the time taken to press "approve" on the hardware wallet interferes with the logic of the test). Please try and run them again when you get the chance.

[tzemanovic]

These changes in the wallet store are breaking, but I think we could make them compatible by e.g. trying to deserialize to the current format in case of a failure and migrate to the new format

[murisi]

Thanks, good point. I've written code to deserialise the current format and migrate it to the new format. I had to slightly modify the new format to support encryptions of data in the old format (which do not seem to be easily migratable).

[tzemanovic]

I'm trying to run e2e::ledger_tests::masp_txs_and_queries with these changes and 713f954:

Hi @tzemanovic . All the MASP tests now work...

Everything works well, thanks! But there are some affected unrelated integration tests and it looks like there's maybe a breaking change for IBC/hermes.

An observation from UX perspective - a shielded transfer and unshielding transfer require multiple wallet interactions, first to access a viewing key and then 2 more to sign the transfer(s) and again to obtain the signature from the wallet. The last 2 interactions print the same information on the wallet screen, so it appears that it's repeating the same action. I'm not sure if we can do much about that in the wallet, but I think we could at least print something out in the CLI before the request to make it clear what's happening.

I'll also add emulator automation for one of the masp tests and we'll need to update the elf file we use in the CI. For local testing I just merged together all the open PRs in ledger-namada. Am I okay to use that?

[tzemanovic]

One more thing - during a shielded and unshielding transfer when the amount being transferred is lower than the balance of the source, the change that stays in the sender's balance is shown as an obscured receiver payment address so it's not apparent that the destination is the same as the sender. Is there something we can do to improve this?

[murisi]

Everything works well, thanks! But there are some affected unrelated integration tests and it looks like there's maybe a breaking change for IBC/hermes.

You're right. I just checked, and the root causes of this were the following:

• This PR changes the spending keys, viewing keys, and payment addresses used in the IBC tests from raw bech32m literals to aliases of things in the wallet. Unfortunately some of the IBC tests were still checking whether the aliases began with zvk and zsk. Furthermore literals need to be specified as receivers for IBC transactions. Both of these are now fixed.
• The second cause was my breaking changes to the Store (that you mentioned above). This PR will need to be fixed to achieve backwards compatibility with the old format (as you mentioned), but regardless Hermes will need its namada_sdk dependency updated to support the new representation I think...

[murisi]

One more thing - during a shielded and unshielding transfer when the amount being transferred is lower than the balance of the source, the change that stays in the sender's balance is shown as an obscured receiver payment address so it's not apparent that the destination is the same as the sender. Is there something we can do to improve this?

This is a very good point. I think we can open an issue at https://github.com/Zondax/ledger-namada to propose that the Namada Ledger app checks whether Sapling outputs are going to the default payment address of the signer, and if so annotates the output with an indication that it is actually change.

Done that here: https://github.com/Zondax/ledger-namada-zip32/issues/7 . Thanks again for the suggestion.