Upgrade ed25519-zebra to version 2

[oxarbitrage]

Motivation

We need to validate signatures after Canopy NU where ZIP-215 was implemented.

Solution

In this 2 tickets: #1565 and #1667 the conclusion is to just upgrade and validate everything using ed25519-zebra version 2.

This PR upgrades ed25519-zebra, all tests are passing and zebra is syncing.

I think some testing will be needed, some ideas could be the followings however not sure if totally required:

• Sync zebra up to mainnet and testnet tips, where new signatures will be present.

Review

Related Issues

• Close Enable ZIP-215: Ed25519 Validation Rules #1565
• Close Move to the post-Canopy ed25519 signature rules #1667

Follow Up Work

• get some recent block (after canopy) from zcashd and add it as a test vector into zebra Add Mainnet Canopy Block Test Vectors #1099
• check that modified signatures and modified data don't pass validation Tracking: transaction validation #196

[dconnolly]

Since this is activated at Canopy height, it would be interesting to check pre and post Canopy transactions, and these are for Sprout only, but specifically the signatures in the transactions, not whole blocks.

We should consider a stateful 'sync to Canopy, sync past Canopy' acceptance test, very similar to our Sapling stateful tests, which would be a nice, more finite, full integration test.

[oxarbitrage]

Sounds good, ill see what i can do ;)

[oxarbitrage]

Maybe we should go with the tests vector created by @hdevalence instead of the big sync for this PR: https://github.com/zcash/zcash/blob/master/src/gtest/test_consensus.cpp#L119

What do you think @dconnolly ?

[oxarbitrage]

By discussions in the chat with @dconnolly and @teor2345 :

• The test vectors from Henry, the ones we care about are already added in https://github.com/ZcashFoundation/ed25519-zebra/blob/0e7a96a267a756e642e102a28a44dd79b9c7df69/tests/small_order.rs#L12
• We want to upgrade to version 2 and validate everything with that but we want to test this, ideas:
  • Deidre: We just need to have tests in place to catch any regressions at the tx joinsplit sig levels
  • Teor: We should test at least one signature that would fail under the new rules, but passes under the old rules. And one that should fail under both sets of rules.
  • Deidre: So what we need in zebra, are tests that if we validate a joinsplit signature pre-canopy, and post-canopy, like teor said. The test vectors we need may not exist. Or rather, may not be easy to pluck. The compatibility between low level, generic Ed25519 sigs that are ZIP-215 compatible look to be clear, but yeah as teor said, we want to have a transaction with a signature over it, pre and post canopy, bascially to test our integration of ed25519
  • Deidre: we want to have a transaction with a signature over it, pre and post canopy, bascially to test our integration of ed25519

[teor2345]

We should consider a stateful 'sync to Canopy, sync past Canopy' acceptance test, very similar to our Sapling stateful tests, which would be a nice, more finite, full integration test.

Sounds like #1388.

[teor2345]

The Windows CI failure is #1769

[oxarbitrage]

We decided to do a manual cargo tree to test this PR and a full sync of the mainnet and testnet.

$ cargo tree | grep ed25519
├── ed25519-zebra v2.2.0
├── ed25519-zebra v2.2.0 (*)
$

Full output

[teor2345]

The changes here are good, we just need to run the full sync tests, then we can merge. I'll update the checklist in the ticket.

[oxarbitrage]

I synchronized a mainnet today on top of this PR.

It started at 2.08 pm local time and ended 5:04 as can be seen in the network traffic chart:

This is 2 hours 56 minutes. I am posting this because there were report of about 2 hours and a half. Unsure if this PR can be the difference or just network/hardware conditions.

I will try the testnet and post details.

[teor2345]

This is 2 hours 56 minutes. I am posting this because there were report of about 2 hours and a half. Unsure if this PR can be the difference or just network/hardware conditions.

I'm not concerned about this variance, it takes me about 4-6 hours locally, probably due to network speed/latency:
#1803 (comment)

If we got this change wrong, then we would be rejecting blocks and failing to reach the tip.

[oxarbitrage]

The testnet is getting close to block 700_000 after 3 days. Very slow but it keeps moving. I can report again when it gets into the tip, if everything goes fine i think it will take at least 1 day more.

[teor2345]

The testnet is getting close to block 700_000 after 3 days. Very slow but it keeps moving. I can report again when it gets into the tip, if everything goes fine i think it will take at least 1 day more.

Hmm 3 days is a really bad sign, I usually see testnet sync within a few hours. But I also have local testnet zcashd and Zebra nodes.

So the slowness it could mean a bunch of different things:

• testnet is all unstable
• Zebra has unrelated bugs
• this change makes Zebra really slow on testnet (which seems really unlikely, because it's mainly a cryptography change)

I'm going to put this on my list of things to double-check on testnet.

[teor2345]

I've just started an ephemeral mainnet and testnet sync for this PR, with some local peers for speed.

[teor2345]

Commit ccc5ae3 (merge 40d4b97) synced within 3 hours on testnet, and 5 hours on mainnet. That's pretty typical for me, I'm network-limited, and my local peers are a much larger proportion of testnet. (Also testnet data is much smaller.)

There were no verification issues, so I think we're good to merge this.

So it looks like your slow testnet sync was a local network issue, or a testnet stability issue. (Which we'll fix with #1222 pretty soon.)

You might also find it helps to keep a synced local Zebra or zcashd instance running, and configure your test instance talk to it. (If you have enough RAM and disk for that.)