Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NEP-364: Efficient signature verification host functions #364

Merged
merged 24 commits into from
Oct 6, 2022
Merged
Show file tree
Hide file tree
Changes from 14 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
93 changes: 46 additions & 47 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,23 +10,22 @@ Changes to the protocol specification and standards are called NEAR Enhancement

## NEPs

|NEP # | Title | Author | Status |
|---|---|---|---|
|[0001](https://github.com/near/NEPs/blob/master/neps/nep-0001.md) | NEP Purpose and Guidelines | @jlogelin | Living |
|[0021](https://github.com/near/NEPs/blob/master/neps/nep-0021.md) | Fungible Token Standard (Deprecated) | @evgenykuzyakov | Final |
|[0141](https://github.com/near/NEPs/blob/master/neps/nep-0141.md) | Fungible Token Standard | @evgenykuzyakov @oysterpack | Final |
|[0145](https://github.com/near/NEPs/blob/master/neps/nep-0145.md) | Storage Management | @evgenykuzyakov | Final |
|[0148](https://github.com/near/NEPs/blob/master/neps/nep-0148.md) | Fungible Token Metadata | @robert-zaremba @evgenykuzyakov @oysterpack | Final |
|[0171](https://github.com/near/NEPs/blob/master/neps/nep-0171.md) | Non Fungible Token Standard | @mikedotexe @evgenykuzyakov @oysterpack | Final |
|[0177](https://github.com/near/NEPs/blob/master/neps/nep-0177.md) | Non Fungible Token Metadata | @chadoh @mikedotexe | Final |
|[0178](https://github.com/near/NEPs/blob/master/neps/nep-0178.md) | Non Fungible Token Approval Management | @chadoh @thor314 | Final |
|[0181](https://github.com/near/NEPs/blob/master/neps/nep-0181.md) | Non Fungible Token Enumeration | @chadoh @thor314 | Final |
|[0199](https://github.com/near/NEPs/blob/master/neps/nep-0199.md) | Non Fungible Token Royalties and Payouts | @thor314 @mattlockyer | Final |
|[0245](https://github.com/near/NEPs/blob/master/neps/nep-0245.md) | Multi Token Standard | @zcstarr @riqi @jriemann @marcos.sun | Review |
|[0297](https://github.com/near/NEPs/blob/master/neps/nep-0297.md) | Events Standard | @telezhnaya | Final |
|[0330](https://github.com/near/NEPs/blob/master/neps/nep-0330.md) | Source Metadata | @BenKurrek | Review |


| NEP # | Title | Author | Status |
| ----------------------------------------------------------------- | ----------------------------------------------------------------- | ------------------------------------------- | ------ |
| [0001](https://github.com/near/NEPs/blob/master/neps/nep-0001.md) | NEP Purpose and Guidelines | @jlogelin | Living |
| [0021](https://github.com/near/NEPs/blob/master/neps/nep-0021.md) | Fungible Token Standard (Deprecated) | @evgenykuzyakov | Final |
| [0141](https://github.com/near/NEPs/blob/master/neps/nep-0141.md) | Fungible Token Standard | @evgenykuzyakov @oysterpack | Final |
| [0145](https://github.com/near/NEPs/blob/master/neps/nep-0145.md) | Storage Management | @evgenykuzyakov | Final |
| [0148](https://github.com/near/NEPs/blob/master/neps/nep-0148.md) | Fungible Token Metadata | @robert-zaremba @evgenykuzyakov @oysterpack | Final |
| [0171](https://github.com/near/NEPs/blob/master/neps/nep-0171.md) | Non Fungible Token Standard | @mikedotexe @evgenykuzyakov @oysterpack | Final |
| [0177](https://github.com/near/NEPs/blob/master/neps/nep-0177.md) | Non Fungible Token Metadata | @chadoh @mikedotexe | Final |
| [0178](https://github.com/near/NEPs/blob/master/neps/nep-0178.md) | Non Fungible Token Approval Management | @chadoh @thor314 | Final |
| [0181](https://github.com/near/NEPs/blob/master/neps/nep-0181.md) | Non Fungible Token Enumeration | @chadoh @thor314 | Final |
| [0199](https://github.com/near/NEPs/blob/master/neps/nep-0199.md) | Non Fungible Token Royalties and Payouts | @thor314 @mattlockyer | Final |
| [0245](https://github.com/near/NEPs/blob/master/neps/nep-0245.md) | Multi Token Standard | @zcstarr @riqi @jriemann @marcos.sun | Review |
| [0297](https://github.com/near/NEPs/blob/master/neps/nep-0297.md) | Events Standard | @telezhnaya | Final |
| [0330](https://github.com/near/NEPs/blob/master/neps/nep-0330.md) | Source Metadata | @BenKurrek | Review |
| [0330](https://github.com/near/NEPs/blob/master/neps/nep-0330.md) | Efficient signature verification and hashing host functions | | Draft |
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please set the correct number

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

unrelated to this specific PR, but I see a process issue (so, cc @frol): our process for NEPs should not require modification of shared files (like readme.md), as that is very prone to conflicts.


## Specification

Expand Down Expand Up @@ -54,26 +53,26 @@ Spec changes are ultimately done via pull requests to this repository (formalize
2. The forum has comment threading which allows the community and NEAR Collective to ideate, ask questions, wrestle with approaches, etc. If more immediate responses are desired, consider bringing the conversation to [Zulip](https://near.zulipchat.com/#narrow/stream/320497-nep-standards).
3. When the governance conversations have reached a point where a clear plan is evident, create a pull request, using the instructions below.

* Clone this repository and create a branch with "my-feature".
* Update relevant content in the current specification that are affected by the proposal.
* Create a Pull request, using [nep-0000-template.md](nep-0000-template.md) to describe motivation and details of the new Contract or Protocol specification. In the document header, ensure the `Status` is marked as `Draft`, and any relevant discussion links are added to the `DiscussionsTo` section.
Use the pull request number padded with zeroes. For instance, the pull request `219` should be created as `neps/nep-0219.md`.
* Add your Draft standard to the `NEPs` section of this README.md. This helps advertise your standard via github.
* Update Docusaurus documentation under the `specs/Standards` to describe the contract standard at a high level, how to integrate it into a Dapp, and a link to the standard document (ie. `neps/nep-0123.md`). This helps advertise your standard via [nomicon](https://nomicon.io/). Any related nomicon sections should be prefixed and styled using the following snippet:

```
:::caution
This is part of proposed spec [NEP-123](https://github.com/near/NEPs/blob/master/neps/nep-0123.md) and subject to change.
:::
```

* Once complete, submit the pull request for editor review.
* The formalization dance begins:
* NEP Editors, who are unopinionated shepherds of the process, check document formatting, completeness and adherence to [NEP-0001](neps/nep-0001.md) and approve the pull request.
* Once ready, the author updates the NEP status to `Review` allowing further community participation, to address any gaps or clarifications, normally part of the Review PR.
* NEP Editors mark the NEP as `Last Call`, allowing a 14 day grace period for any final community feedback. Any unresolved show stoppers roll the state back to `Review`.
* NEP Editors mark the NEP as `Final`, marking the standard as complete. The standard should only be updated to correct errata and add non-normative clarifications.
- Clone this repository and create a branch with "my-feature".
- Update relevant content in the current specification that are affected by the proposal.
- Create a Pull request, using [nep-0000-template.md](nep-0000-template.md) to describe motivation and details of the new Contract or Protocol specification. In the document header, ensure the `Status` is marked as `Draft`, and any relevant discussion links are added to the `DiscussionsTo` section.
Use the pull request number padded with zeroes. For instance, the pull request `219` should be created as `neps/nep-0219.md`.
- Add your Draft standard to the `NEPs` section of this README.md. This helps advertise your standard via github.
- Update Docusaurus documentation under the `specs/Standards` to describe the contract standard at a high level, how to integrate it into a Dapp, and a link to the standard document (ie. `neps/nep-0123.md`). This helps advertise your standard via [nomicon](https://nomicon.io/). Any related nomicon sections should be prefixed and styled using the following snippet:

```
:::caution
This is part of proposed spec [NEP-123](https://github.com/near/NEPs/blob/master/neps/nep-0123.md) and subject to change.
:::
```

- Once complete, submit the pull request for editor review.

- The formalization dance begins:
- NEP Editors, who are unopinionated shepherds of the process, check document formatting, completeness and adherence to [NEP-0001](neps/nep-0001.md) and approve the pull request.
- Once ready, the author updates the NEP status to `Review` allowing further community participation, to address any gaps or clarifications, normally part of the Review PR.
- NEP Editors mark the NEP as `Last Call`, allowing a 14 day grace period for any final community feedback. Any unresolved show stoppers roll the state back to `Review`.
- NEP Editors mark the NEP as `Final`, marking the standard as complete. The standard should only be updated to correct errata and add non-normative clarifications.

Tip: build consensus and integrate feedback. NEPs that have broad support are much more likely to make progress than those that don't receive any comments. Feel free to reach out to the NEP assignee in particular to get help identify stakeholders and obstacles.

Expand All @@ -92,19 +91,19 @@ This repository uses [Docusaurus](https://docusaurus.io/) for the [Nomicon websi

- Run the local docs development server

```sh
# Start the site
yarn start
```
```sh
# Start the site
yarn start
```

_Expected Output_
_Expected Output_

```sh
# Website with live reload is started
Docusaurus server started on port 3000
```
```sh
# Website with live reload is started
Docusaurus server started on port 3000
```

The website for docs will open your browser locally to port `3000`
The website for docs will open your browser locally to port `3000`

2. Make changes to the docs

Expand Down
180 changes: 180 additions & 0 deletions neps/nep-0364.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,180 @@
---
NEP: 364
Title: Efficient signature verification and hashing precompile functions
Author: Blas Rodriguez Irizar <rodrigblas@gmail.com>
DiscussionsTo: https://github.com/nearprotocol/neps/pull/364
Status: Draft
Type: Runtime Spec
Category: Contract
Created: 15-Jun-2022
---

## Summary

This NEP introduces the request of adding into the NEAR runtime a pre-compiled
function used to verify signatures that can help IBC compatible light clients run on-chain.

## Motivation

Signature verification and hashing are ubiquitous operations in light clients,
especially in PoS consensus mechanisms. Based on Polkadot's consensus mechanism
there will be a need for verification of ~200 signatures every minute
(Polkadot’s authority set is ~300 signers and it may be increased in the future: https://polkadot.polkassembly.io/referendum/16).

Therefore, a mechanism to perform these operations cost-effectively in terms
of gas and speed would be highly beneficial to have. Currently, NEAR does not have any native signature verification toolbox.
This implies that a light client operating inside NEAR will have to import a library
compiled to WASM as mentioned in [Zulip](https://near.zulipchat.com/#narrow/stream/295302-general/topic/light_client).

Polkadot uses [three different cryptographic schemes](https://wiki.polkadot.network/docs/learn-keys)
for its keys/accounts, which also translates into different signature types. However, for this NEP the focus is on:

- The vanilla ed25519 implementation uses Schnorr signatures.

## Rationale and alternatives

Add a signature verification signatures function into the runtime as host functions.

- ED25519 signature verification function using `ed25519_dalek` crates into NEAR runtime as pre-compiled functions.

Benchmarks were run using a signature verifier smart contract on-chain importing the aforementioned functions from
widespread used crypto Rust crates. The biggest pitfall of these functions running wasm code instead of native
is performance and gas cost. Our [benchmarks](https://github.com/blasrodri/near-test) show the following results:

```log
near call sitoula-test.testnet verify_ed25519 '{"signature_p1": [145,193,203,18,114,227,14,117,33,213,121,66,130,14,25,4,36,120,46,142,226,215,7,66,122,112,97,30,249,135,61,165], "signature_p2": [221,249,252,23,105,40,56,70,31,152,236,141,154,122,207,20,75,118,79,90,168,6,221,122,213,29,126,196,216,104,191,6], "msg": [107,97,106,100,108,102,107,106,97,108,107,102,106,97,107,108,102,106,100,107,108,97,100,106,102,107,108,106,97,100,115,107], "iterations": 10}' --accountId sitoula-test.testnet --gas 300000000000000
# transaction id DZMuFHisupKW42w3giWxTRw5nhBviPu4YZLgKZ6cK4Uq
```

With `iterations = 130` **all these calls return ExecutionError**: `'Exceeded the maximum amount of gas allowed to burn per contract.'`
With iterations = 50 these are the results:
Comment on lines +49 to +50
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is one iteration one signature verification?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree. For each host function, please explain why exactly they are needed and their specification

I've added a reference in the motivation section. Would you like it to be more specific, being included in the comment of each host function?


```
ed25519: tx id 6DcJYfkp9fGxDGtQLZ2m6PEDBwKHXpk7Lf5VgDYLi9vB (299 Tgas)
```

- Performance in wall clock time when you compile the signature validation library directly from rust to native.
Here are the results on an AMD Ryzen 9 5900X 12-Core Processor machine:

```
# 10k signature verifications
ed25519: took 387ms
```

- Performance in wall clock time when you compile the library into wasm first and then use the single-pass compiler in Wasmer 1 to then compile to native.

```
ed25519: took 9926ms
```

As an extra data point, when passing `--enable-simd` instead of `--singlepass`

```
ed25519: took 3085ms
```

Steps to reproduce:
commit: `31cf97fb2e155d238308f062c4b92bae716ac19f` in `https://github.com/blasrodri/near-test`

```sh
# wasi singlepass
cargo wasi build --bin benches --release
wasmer compile --singlepass ./target/wasm32-wasi/release/benches.wasi.wasm -o benches_singlepass
wasmer run ./benches_singlepass
```

```sh
# rust native
cargo run --bin benches --release
```

Overall: the difference between the two versions (native vs wasi + singlepass is)

```
ed25519: 25.64x slower
```
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you also mention is the expected improvement after this NEP ? How many verifications are you able to fit in a 300 TGas transaction ?


### What is the impact of not doing this?

Costs of running IBC-compatible trustless bridges would be very high. Plus, the fact that signature verification
is such an expensive operation will force the contract to split the process of batch verification of signatures
into multiple transactions.

### Why is this design the best in the space of possible designs?

Adding existing proved and vetted crypto crates into the runtime is a safe workaround. It will boost performance
between 20-25x according to our benchmarks. This will both reduce operating costs significantly and will also
enable the contract to verify all the signatures in one transaction, which will simplify the contract design.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If Polkadot uses 3 different types of keys - are you still able to verify all the signatures in one transaction ? (for example in a scenario, where all the polkadot validators are using sr25519 ? )


### What other designs have been considered and what is the rationale for not choosing them?

One possible alternative would be to improve the runtime implementation so that it can compile WASM code to a sufficiently
fast machine code. Even when it may not be as fast as LLVM native produced code it could still be acceptable for
these types of use cases (CPU intensive functions) and will certainly avoid the need of adding host functions.
The effort of adding such a feature will be significantly higher than adding these host functions one by one.
But on the other side, it will decrease the need of including more host functions in the future.

Another alternative is to deal with the high cost of computing/verifying these signatures in some other manner.
Decreasing the overall cost of gas and increasing the limits of gas available to attach to the contract could be a possibility.
Introducing such modification for some contracts, and not for some others can be rather arbitrary
and not straightforward in the implementation, but an alternative nevertheless.

## Specification
nagisa marked this conversation as resolved.
Show resolved Hide resolved

This NEP aims to introduce the following host function:

```rust
/// Ed25519 is a public-key signature system with several attractive features
///
/// Proof of Stake Validator sets can contain different signature schemes.
/// Ed25519 is one of the most used ones across blockchains, and hence it's importance to be added.
/// For further reference, visit: https://ed25519.cr.yp.to
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if we should add more inline information about how ed25519 works, in particular how it handles some of the edge cases. An alternative to add this information inline would be to add this information to Nomicon, given ed25519 is already a cyptographic primitive used in NEAR before this NEP.

The motivation is to allow potential new client implementations to be able to build the client by only reading Nomicon and NEPs.

/// # Cost
///
/// `base + write_register_base + write_register_byte * num_bytes + ed25519_verify_base + ed25519_verify_byte * num_bytes`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where did these gas costs come from? Please see my comment on the implementation PR and update accordingly. near/nearcore#7165 (comment)

Copy link
Contributor Author

@blasrodri blasrodri Aug 10, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! Will update this as soon as the PR with the implementation reflects the right costs.

pub fn ed25519_verify(
&mut self,
sig_len: u64,
sig_ptr: u64,
msg_len: u64,
msg_ptr: u64,
pub_key_len: u64,
pub_key_ptr: u64,
) -> Result<u64>;
```

And a `rust-sdk` possible implementation could look like this:

```rs

pub fn ed25519_verify(sig: &ed25519::Signature, msg: &[u8], pub_key: &ed25519::Public) -> bool;

```

Once this NEP is approved and integrated, these functions will be available in the `near_sdk` crate in the
`env` module.

## Security Implications (Optional)

We have chosen these crates because they are the ones already integrated into the
[sp_io](https://docs.rs/sp-io/latest/sp_io/)](https://docs.rs/sp-io/latest/sp_io/)
crate heavily used in the [substrate](https://github.com/paritytech/substrate) repository.

## Unresolved Issues (Optional)

- What parts of the design do you expect to resolve through the NEP process before this gets merged?
Both the function signatures and crates are up for discussion.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suggest that we use the library that nearcore already uses for ed25519 signatures. This would avoid introducing new dependencies and ensure that NEAR signatures themselves can be 100% verified in smart contracts

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.


## Future possibilities

I currently do not envision any extension in this regard.

## Copyright

[copyright]: #copyright

Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/).

```

```