Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

NEP-364: Efficient signature verification host functions #364

Merged
merged 24 commits into from
Oct 6, 2022
Merged

NEP-364: Efficient signature verification host functions #364

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
34a9391
NEP-364
blasrodri Jun 15, 2022
1d4dc44
correct some grammar
blasrodri Jun 15, 2022
56d08f0
add more benchmarks
blasrodri Jun 15, 2022
0129422
Update README.md
blasrodri Jul 6, 2022
8962147
incorporate comments
blasrodri Jul 12, 2022
67ac13b
comment on the number of signers
blasrodri Jul 12, 2022
055bd09
Update neps/nep-0364.md
blasrodri Jul 12, 2022
70265a5
Update neps/nep-0364.md
blasrodri Jul 12, 2022
d6c0771
add alternatives
blasrodri Jul 12, 2022
a7789b3
add comments and specs on functions
blasrodri Jul 13, 2022
1d2777f
cleanup algorithms except for ed25519
blasrodri Jul 21, 2022
479edff
rm reference to old functions
blasrodri Jul 21, 2022
5ed5b18
typos and spec conforming with wasm
blasrodri Jul 21, 2022
1e8d3df
return u64
blasrodri Jul 22, 2022
80ac231
minor adjustments
blasrodri Aug 17, 2022
f3b5c39
update cost (probably WIP)
blasrodri Aug 17, 2022
e3af2cb
update docs
blasrodri Aug 17, 2022
ba8a3d3
revert README
blasrodri Sep 6, 2022
bdf64d0
addressing comments
blasrodri Sep 7, 2022
59257ca
address comments
blasrodri Sep 22, 2022
d25d8f8
rm curly brace
blasrodri Sep 22, 2022
3dac4ea
document crate decisions
blasrodri Oct 3, 2022
73e0290
updated comments
blasrodri Oct 5, 2022
003493e
rm redundant sentence
blasrodri Oct 6, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
93 changes: 46 additions & 47 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,23 +10,22 @@ Changes to the protocol specification and standards are called NEAR Enhancement

## NEPs

|NEP # | Title | Author | Status |
|---|---|---|---|
|[0001](https://github.com/near/NEPs/blob/master/neps/nep-0001.md) | NEP Purpose and Guidelines | @jlogelin | Living |
|[0021](https://github.com/near/NEPs/blob/master/neps/nep-0021.md) | Fungible Token Standard (Deprecated) | @evgenykuzyakov | Final |
|[0141](https://github.com/near/NEPs/blob/master/neps/nep-0141.md) | Fungible Token Standard | @evgenykuzyakov @oysterpack | Final |
|[0145](https://github.com/near/NEPs/blob/master/neps/nep-0145.md) | Storage Management | @evgenykuzyakov | Final |
|[0148](https://github.com/near/NEPs/blob/master/neps/nep-0148.md) | Fungible Token Metadata | @robert-zaremba @evgenykuzyakov @oysterpack | Final |
|[0171](https://github.com/near/NEPs/blob/master/neps/nep-0171.md) | Non Fungible Token Standard | @mikedotexe @evgenykuzyakov @oysterpack | Final |
|[0177](https://github.com/near/NEPs/blob/master/neps/nep-0177.md) | Non Fungible Token Metadata | @chadoh @mikedotexe | Final |
|[0178](https://github.com/near/NEPs/blob/master/neps/nep-0178.md) | Non Fungible Token Approval Management | @chadoh @thor314 | Final |
|[0181](https://github.com/near/NEPs/blob/master/neps/nep-0181.md) | Non Fungible Token Enumeration | @chadoh @thor314 | Final |
|[0199](https://github.com/near/NEPs/blob/master/neps/nep-0199.md) | Non Fungible Token Royalties and Payouts | @thor314 @mattlockyer | Final |
|[0245](https://github.com/near/NEPs/blob/master/neps/nep-0245.md) | Multi Token Standard | @zcstarr @riqi @jriemann @marcos.sun | Review |
|[0297](https://github.com/near/NEPs/blob/master/neps/nep-0297.md) | Events Standard | @telezhnaya | Final |
|[0330](https://github.com/near/NEPs/blob/master/neps/nep-0330.md) | Source Metadata | @BenKurrek | Review |


| NEP # | Title | Author | Status |
| ----------------------------------------------------------------- | ----------------------------------------------------------------- | ------------------------------------------- | ------ |
| [0001](https://github.com/near/NEPs/blob/master/neps/nep-0001.md) | NEP Purpose and Guidelines | @jlogelin | Living |
| [0021](https://github.com/near/NEPs/blob/master/neps/nep-0021.md) | Fungible Token Standard (Deprecated) | @evgenykuzyakov | Final |
| [0141](https://github.com/near/NEPs/blob/master/neps/nep-0141.md) | Fungible Token Standard | @evgenykuzyakov @oysterpack | Final |
| [0145](https://github.com/near/NEPs/blob/master/neps/nep-0145.md) | Storage Management | @evgenykuzyakov | Final |
| [0148](https://github.com/near/NEPs/blob/master/neps/nep-0148.md) | Fungible Token Metadata | @robert-zaremba @evgenykuzyakov @oysterpack | Final |
| [0171](https://github.com/near/NEPs/blob/master/neps/nep-0171.md) | Non Fungible Token Standard | @mikedotexe @evgenykuzyakov @oysterpack | Final |
| [0177](https://github.com/near/NEPs/blob/master/neps/nep-0177.md) | Non Fungible Token Metadata | @chadoh @mikedotexe | Final |
| [0178](https://github.com/near/NEPs/blob/master/neps/nep-0178.md) | Non Fungible Token Approval Management | @chadoh @thor314 | Final |
| [0181](https://github.com/near/NEPs/blob/master/neps/nep-0181.md) | Non Fungible Token Enumeration | @chadoh @thor314 | Final |
| [0199](https://github.com/near/NEPs/blob/master/neps/nep-0199.md) | Non Fungible Token Royalties and Payouts | @thor314 @mattlockyer | Final |
| [0245](https://github.com/near/NEPs/blob/master/neps/nep-0245.md) | Multi Token Standard | @zcstarr @riqi @jriemann @marcos.sun | Review |
| [0297](https://github.com/near/NEPs/blob/master/neps/nep-0297.md) | Events Standard | @telezhnaya | Final |
| [0330](https://github.com/near/NEPs/blob/master/neps/nep-0330.md) | Source Metadata | @BenKurrek | Review |
| [0330](https://github.com/near/NEPs/blob/master/neps/nep-0330.md) | Efficient signature verification and hashing precompile functions | | Draft |
blasrodri marked this conversation as resolved.
Show resolved Hide resolved

## Specification

Expand Down Expand Up @@ -54,26 +53,26 @@ Spec changes are ultimately done via pull requests to this repository (formalize
2. The forum has comment threading which allows the community and NEAR Collective to ideate, ask questions, wrestle with approaches, etc. If more immediate responses are desired, consider bringing the conversation to [Zulip](https://near.zulipchat.com/#narrow/stream/320497-nep-standards).
3. When the governance conversations have reached a point where a clear plan is evident, create a pull request, using the instructions below.

* Clone this repository and create a branch with "my-feature".
* Update relevant content in the current specification that are affected by the proposal.
* Create a Pull request, using [nep-0000-template.md](nep-0000-template.md) to describe motivation and details of the new Contract or Protocol specification. In the document header, ensure the `Status` is marked as `Draft`, and any relevant discussion links are added to the `DiscussionsTo` section.
Use the pull request number padded with zeroes. For instance, the pull request `219` should be created as `neps/nep-0219.md`.
* Add your Draft standard to the `NEPs` section of this README.md. This helps advertise your standard via github.
* Update Docusaurus documentation under the `specs/Standards` to describe the contract standard at a high level, how to integrate it into a Dapp, and a link to the standard document (ie. `neps/nep-0123.md`). This helps advertise your standard via [nomicon](https://nomicon.io/). Any related nomicon sections should be prefixed and styled using the following snippet:

```
:::caution
This is part of proposed spec [NEP-123](https://github.com/near/NEPs/blob/master/neps/nep-0123.md) and subject to change.
:::
```

* Once complete, submit the pull request for editor review.
* The formalization dance begins:
* NEP Editors, who are unopinionated shepherds of the process, check document formatting, completeness and adherence to [NEP-0001](neps/nep-0001.md) and approve the pull request.
* Once ready, the author updates the NEP status to `Review` allowing further community participation, to address any gaps or clarifications, normally part of the Review PR.
* NEP Editors mark the NEP as `Last Call`, allowing a 14 day grace period for any final community feedback. Any unresolved show stoppers roll the state back to `Review`.
* NEP Editors mark the NEP as `Final`, marking the standard as complete. The standard should only be updated to correct errata and add non-normative clarifications.
- Clone this repository and create a branch with "my-feature".
- Update relevant content in the current specification that are affected by the proposal.
- Create a Pull request, using [nep-0000-template.md](nep-0000-template.md) to describe motivation and details of the new Contract or Protocol specification. In the document header, ensure the `Status` is marked as `Draft`, and any relevant discussion links are added to the `DiscussionsTo` section.
Use the pull request number padded with zeroes. For instance, the pull request `219` should be created as `neps/nep-0219.md`.
- Add your Draft standard to the `NEPs` section of this README.md. This helps advertise your standard via github.
- Update Docusaurus documentation under the `specs/Standards` to describe the contract standard at a high level, how to integrate it into a Dapp, and a link to the standard document (ie. `neps/nep-0123.md`). This helps advertise your standard via [nomicon](https://nomicon.io/). Any related nomicon sections should be prefixed and styled using the following snippet:

```
:::caution
This is part of proposed spec [NEP-123](https://github.com/near/NEPs/blob/master/neps/nep-0123.md) and subject to change.
:::
```

- Once complete, submit the pull request for editor review.

- The formalization dance begins:
- NEP Editors, who are unopinionated shepherds of the process, check document formatting, completeness and adherence to [NEP-0001](neps/nep-0001.md) and approve the pull request.
- Once ready, the author updates the NEP status to `Review` allowing further community participation, to address any gaps or clarifications, normally part of the Review PR.
- NEP Editors mark the NEP as `Last Call`, allowing a 14 day grace period for any final community feedback. Any unresolved show stoppers roll the state back to `Review`.
- NEP Editors mark the NEP as `Final`, marking the standard as complete. The standard should only be updated to correct errata and add non-normative clarifications.

Tip: build consensus and integrate feedback. NEPs that have broad support are much more likely to make progress than those that don't receive any comments. Feel free to reach out to the NEP assignee in particular to get help identify stakeholders and obstacles.

Expand All @@ -92,19 +91,19 @@ This repository uses [Docusaurus](https://docusaurus.io/) for the [Nomicon websi

- Run the local docs development server

```sh
# Start the site
yarn start
```
```sh
# Start the site
yarn start
```

_Expected Output_
_Expected Output_

```sh
# Website with live reload is started
Docusaurus server started on port 3000
```
```sh
# Website with live reload is started
Docusaurus server started on port 3000
```

The website for docs will open your browser locally to port `3000`
The website for docs will open your browser locally to port `3000`

2. Make changes to the docs

Expand Down
153 changes: 153 additions & 0 deletions neps/nep-0364.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,153 @@
---
NEP: 364
Title: Efficient signature verification and hashing precompile functions
Author: Blas Rodriguez Irizar <rodrigblas@gmail.com>
DiscussionsTo: https://github.com/nearprotocol/neps/pull/364
Status: Draft
Type: Runtime Spec
Category: Contract
Created: 15-Jun-2022
---

## Summary

This NEP introduces the request of adding into the NEAR runtime a set of pre-compile
functions that can help IBC compatible light clients run on-chain.

## Motivation

Signature verification and hashing are ubiquitous operations in light clients,
especially in PoS consensus mechanisms. Our rough numbers say that we’ll be
verifying ~200 signatures every minute (Polkadot’s authority set is 300 signers).
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What does "we" here refer to? How is the number 200 calculated?


Therefore, we need a mechanism to perform these operations in a cost-effective way
in terms of gas and speed. Currently, NEAR does not have any native signature verification toolbox.
This implies that a light client operating insid NEAR will have to import a library
compiled to WASM as mentioned in [Zulip](https://near.zulipchat.com/#narrow/stream/295302-general/topic/light_client).

With regards to hashing, there are already several hashing functions implemented natively.
For our use case, we need to be able to use `blake2b` has function, which is not available.
nagisa marked this conversation as resolved.
Show resolved Hide resolved

## Rationale and alternatives

Add

```rust
pub fn ed25519_verify(sig: &ed25519::Signature, msg: &[u8], pub_key: &ed25519::Public) -> bool
bowenwang1996 marked this conversation as resolved.
Show resolved Hide resolved
pub fn sr25519_verify(sig: &sr25519::Signature, msg: &[u8], pub_key: &sr25519::Public) -> bool
pub fn ecdsa_verify(sig: &ecdsa::Signature, msg: &[u8], pub_key: &ecdsa::Public) -> bool
pub fn blake2b(nn: usize, k: &[u8], data: &[u8]) -> Result<[u8; 32], Error>
```

using `blake2_rfc`, `ed25519_dalek`, `schnorrkel` and `secp256k1` crates into NEAR runtime as pre-compiled functions.

We've tried running a signature verifier smart contract on-chain using our own wasm
module powered by widespread used crypto Rust crates. The biggest pitfall of that solution
is performance and cost. Our [benchamarks](https://github.com/blasrodri/near-test) show the following results:

```log
near call sitoula-test.testnet verify_ed25519 '{"signature_p1": [145,193,203,18,114,227,14,117,33,213,121,66,130,14,25,4,36,120,46,142,226,215,7,66,122,112,97,30,249,135,61,165], "signature_p2": [221,249,252,23,105,40,56,70,31,152,236,141,154,122,207,20,75,118,79,90,168,6,221,122,213,29,126,196,216,104,191,6], "msg": [107,97,106,100,108,102,107,106,97,108,107,102,106,97,107,108,102,106,100,107,108,97,100,106,102,107,108,106,97,100,115,107], "iterations": 10}' --accountId sitoula-test.testnet --gas 300000000000000
# transaction id DZMuFHisupKW42w3giWxTRw5nhBviPu4YZLgKZ6cK4Uq

near call sitoula-test.testnet verify_schnorrkel '{"signature_p1": [106, 144, 17, 34, 142, 65, 191, 241, 233, 250, 132, 168, 204, 173, 122, 196, 118, 248, 159, 159, 254, 37, 153, 84, 248, 104, 206, 217, 168, 65, 12, 74], "signature_p2": [183, 134, 143, 30, 123, 61, 112, 153, 244, 109, 199, 195, 164, 0, 7, 55, 26, 199, 164, 219, 147, 217, 157, 239, 198, 108, 162, 246, 52, 49, 116, 132], "msg": [107,97,106,100,108,102,107,106,97,108,107,102,106,97,107,108,102,106,100,107,108,97,100,106,102,107,108,106,97,100,115,107], "iterations": 10}' --accountId sitoula-test.testnet --gas 300000000000000
# transaction id DpWpK2jnyBRgZ7fScaBKULikJyvmMjDHx7AyWMrJL5VB

near call sitoula-test.testnet verify_ecdsa '{"signature_p1": [231, 117, 17, 89, 49, 142, 111, 201, 161, 107, 167, 147, 215, 167, 196, 226, 200, 176, 184, 62, 196, 240, 210, 137, 77, 198, 90, 97, 201, 212, 96, 229], "signature_p2": [1, 31, 7, 121, 178, 247, 150, 131, 108, 250, 173, 71, 100, 192, 83, 64, 145, 85, 254, 69, 176, 7, 114, 89, 64, 205, 30, 243, 193, 78, 142, 27], "msg": [107,97,106,100,108,102,107,106,97,108,107,102,106,97,107,108,102,106,100,107,108,97,100,106,102,107,108,106,97,100,115,107], "iterations": 10}' --accountId sitoula-test.testnet --gas 300000000000000
# transaction id AKFwCuQ8g2a8t9SX7vcve348BvwoU3M42MSef7gy1kPA
```

With `iterations = 130` **all these calls return ExecutionError**: `'Exceeded the maximum amount of gas allowed to burn per contract.'`
With iterations = 50 these are the results:
Comment on lines +49 to +50
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is one iteration one signature verification?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree. For each host function, please explain why exactly they are needed and their specification

I've added a reference in the motivation section. Would you like it to be more specific, being included in the comment of each host function?


```
ed25518: tx id 6DcJYfkp9fGxDGtQLZ2m6PEDBwKHXpk7Lf5VgDYLi9vB (299 Tgas)
schnorrkel: tx id ACYJh7YC4pQM8fu7DsjmmkSb3WSvikfQkVyej3htiuDQ (304 Tgas)
ecdsa: tx id 2FhtFEbakuwQyHWcyeRoPkCDgXz4erTYSD75QkxDTb2e (185 Tgas)
```

- Performance in wall clock time when you compile the signature validation library directly from rust to native.
Here are the results on a AMD Ryzen 9 5900X 12-Core Processor machine:

```
# 10k signature verifications
ed25519: took 387ms
ecdsa: took 324ms
shnorrkel: took 411ms
```

- Performance in wall clock time when you compile the library into wasm first and then use the single-pass compiler in Wasmer 1 to then compile to native.

```
ed25519: took 9926ms
ecdsa: took 6725ms
shnorrkel: took 10220ms
```

As an extra datapoint, when passing `--enable-simd` instead of `--singlepass`

```
ed25519: took 3085ms
ecdsa: took 1895ms
shnorrkel: took 3179ms
```

Steps to reproduce:
commit: `31cf97fb2e155d238308f062c4b92bae716ac19f` in `https://github.com/blasrodri/near-test`

```sh
# wasi singlepass
cargo wasi build --bin benches --release
wasmer compile --singlepass ./target/wasm32-wasi/release/benches.wasi.wasm -o benches_singlepass
wasmer run ./benches_singlepass
```

```sh
# rust native
cargo run --bin benches --release
```

Overall: the difference between the two versions (native vs wasi + singlepass are)

```
ed25519: 25.64x slower
ecdsa: 20.75x slower
shnorrkel: 24.86x slower
```
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you also mention is the expected improvement after this NEP ? How many verifications are you able to fit in a 300 TGas transaction ?


- What is the impact of not doing this?
nagisa marked this conversation as resolved.
Show resolved Hide resolved
Costs of running IBC compatible trustless bridges would be very high. Plus, the fact that signature verification
is such an expensive operation will force the contract to split the process of batch verification of signatures
into multiple transactions.

- Why is this design the best in the space of possible designs?
nagisa marked this conversation as resolved.
Show resolved Hide resolved
Adding exisiting proved and vetted crypto crates into the runtime is a safe workaround. It will boost performance
between 20-25x according to our benchmarks. This will both reduce operation costs significantly and will also
enable the contract to verify all the signatures in one transaction, which will simplify the contract design.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't necessarily agree that it is a safe workaround. The cost of introducing a new host function is actually quite high:

  • A bug in the host function code would cause the entire network to go down
  • Once the host function is introduced, one cannot remove it as that could break contracts which already rely on the behavior of the host functions.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe "safe workaround" isn't the right way of putting it. However, these are highly adopted libraries with operations that are part of multiple services and blockchains. In that way, I was trying to convey the fact that these are no new functions that haven't been battle-tested. If you have any suggestions on how to communicate it differently I'd be happy to update it.


- What other designs have been considered and what is the rationale for not choosing them?
This has been answered before.
nagisa marked this conversation as resolved.
Show resolved Hide resolved

## Specification
nagisa marked this conversation as resolved.
Show resolved Hide resolved

Once this NEP is approved and integrated, these four functions will be available in the `near_sdk` crate in the
blasrodri marked this conversation as resolved.
Show resolved Hide resolved
`env` module.

## Security Implications (Optional)

We have chosen these crates because they are the ones already integrated in the [sp_io](https://docs.rs/sp-io/latest/sp_io/)
crate heavily used in the [substrate](https://github.com/paritytech/substrate) repository.

## Unresolved Issues (Optional)

- What parts of the design do you expect to resolve through the NEP process before this gets merged?
Both the function signatures and crates are up for discussion.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suggest that we use the library that nearcore already uses for ed25519 signatures. This would avoid introducing new dependencies and ensure that NEAR signatures themselves can be 100% verified in smart contracts

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As far as I can see the proposal is the same as the one being used today
https://github.com/near/nearcore/blob/master/core/crypto/src/signature.rs#L729-L732
https://github.com/near/NEPs/pull/364/files#diff-1d8966e006ab1c29466f5c4d5ce49a965477f992e7d16c1d032d5b0164096459R38


## Future possibilities

I currently do not envision any extension in this regard.

## Copyright

[copyright]: #copyright

Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/).