arkworks integration

[achimcc]

• What does it do?
  We provide host function calls for the elliptic curve arithmetic used by Arkworks. We need this for an efficient computation of those arithmetics in the WASM runtime. We provide host calls for bls12_381, bls12_377, ed_on_bls12_381, ed_on_bls12_77, bw6_761.
  Those arithmetics, which are provided by the host calls, are:
  • multi miller loop and final exponentiation for bls12_377, bls12_381 and bw6_761. They are required to compute a bilinear pairing between two elliptic curve points, e.g. for groth16 verification.
  • multi scalar multiplications for all curves
  • affine multiplications for all curves (projective multiplications are computed by converting them to affine first and then calling into the affine multiplication host calls).

The host functions receive their arguments serialized to Vec<u8> by ark-scale and return the result as serialized Result<Vec<u8>, ()>, which then needs to be deserialized by ark-scale.

We add the host function calls to the sp_crypto_ec_utils crate. By now they are an experimental feature which one can enable with the feature flag ec-utils-experimental. We finally decided not to hide the host calls behind a feature flag, since they are in an independent crate and not part of sp-io for now, so that they will not be added to Polkadot/Kusama automatically anyway.

We define the curves in https://github.com/paritytech/ark-substrate), and they receive the host functions by dependency injection. The reason for using dependency injection here, is that we intend to use ark-substrate in Substrate with Ring VRFs/Sassafrass in the future, which are part of the consensus mechanism and therefore they are implemented in the Substrate core repo as well. Hence, since we can't have circular dependencies, ark-substrate can't depend directly on Substrate. We have an additional library substrate-curves, which equips the curves from ark-substrate with the host function calls added here, to provide an end user implementation of curves analogous to ark-curves.

The following dependency diagram visualizes this (the dashed arrow represents dependency injection):

Dependency Diagram

• What important points should reviewers know?
  This is using the https://github.com/paritytech/ark-substrate repo, which contains the curve and model code.
  We cover tests in the arkworks-curves crate by usage of the arwkorks algebra test-templates.
  Tests are now covered by pulling in https://github.com/paritytech/ark-substrate where we instantiate the ark-substrate curves, which receive the sp-io host function implementations by dependency injection. We can re-use the arkworks-rs algebra test-templates for testing.

An overview and comparison of benchmark results can is here.

• Is there something left for follow-up PRs?
  Adding more curves and replacing more operations by host function calls could be further possible optimisations.

• Labels: You labeled the PR appropriately if you have permissions to do so:

  • A* for PR status (one required)
  • B* for changelog (one required)
  • C* for release notes (exactly one required)
  • D* for various implications/requirements
  • Github project assignment

• Related Issues: You mentioned a related issue if this PR is related to it, e.g. Fixes #228 or Related #1337.

• 2 Reviewers: You asked at least two reviewers to review. If you aren't sure, start with GH suggestions.

• Style Guide: Your PR adheres to the style guide

  • In particular, mind the maximal line length of 100 (120 in exceptional circumstances).
  • There is no commented code checked in unless necessary.
  • Any panickers in the runtime have a proof or were removed.

• Runtime Version: You bumped the runtime version if there are breaking changes in the runtime.

• Docs: You updated any rustdocs which may need to change.

• Polkadot Companion: Has the PR altered the external API or interfaces used by Polkadot?

  • If so, do you have the corresponding Polkadot PR ready?
  • Optionally: Do you have a corresponding Cumulus PR?

[davxy]

(Maybe) dumb questions:

1. why the upstream version doesn't satisfy our use case? What patches were applied to the upstream? If I understood correctly some computational intesive tasks are offloaded to host functions? What are these tasks/functions?
2. why we need to integrate all this stuff in Substrate? IMO (in general) is not the best to embed an entire lib in an (already giant) codebase
3. where all this cryptographic thing required? I mean, I see that some host functions were implemented to use these primitives. But then as far as I can see there is no usage of these functions. What is the end goal?

As a side note, we don't directly implement any cryptographic primitive in Substrate, we use external crates.

[achimcc]

(Maybe) dumb questions:

1. why the upstream version doesn't satisfy our use case? What patches were applied to the upstream? If I understood correctly some computational intesive tasks are offloaded to host functions? What are these tasks/functions?
2. why we need to integrate all this stuff in Substrate? IMO (in general) is not the best to embed an entire lib in an (already giant) codebase
3. where all this cryptographic thing required? I mean, I see that some host functions were implemented to use these primitives. But then as far as I can see there is no usage of these functions. What is the end goal?

As a side note, we don't directly implement any cryptographic primitive in Substrate, we use external crates.

1. The operations on arithmetic curves are way to slow in WebAssembly. We have to replace the costly operations by host function calls. The host function tasks are the arithmetic operations on elliptic curves which are slow in WebAssembly. Those are: computation of biliner pairings (ate pairing), multi scalar multiplication, affine and projective multiplications.
2. I'll implement it this way now. As explained above it is a lot more work since the curves have to call into host functions. If we want to make the host functions available to them, we have to pass them by dependency injection and that requires a lot of modifications on the arkworks code.
3. Those are the curves used by arkworks.rs which make all the tools from this library available to every Substrate pallet. Direct examples are Zero Knowledge Proof verification, especially verification of plonk or groth16 proofs. I created a simple example of a groth16 proof verification in a Substrate pallet which uses these curves (which is based on my PR branch of Substrate) here: https://github.com/achimcc/substrate-groth16. Other applications include RingVRF's which are interesting in their own, but will also become a basic building block of Sassafrass consensus. @burdges might comment more on use-cases.

[davxy]

@achimcc I'm very interested since I'm working on sassafras implementation :-)

1. The "prove" component will always run in native client code in order to produce ring membership proofs (to be sent onchain along with tickets).
2. The "verify" is indeed the critical component. Since it may be very computational intensive (and maybe also no no-std... but I don't know this for sure) and run by the runtime when checking for submitted tickets before saving them onchain.

But can't we offer a host function (i.e. verify) doing the overall verification job? I mean, why should we have to run part of the verification code in wasm at all?
I think that a verify host function would be a good interface in order to maintain independence from the actual library. Is there some detail I'm missing that prevents to do it?

[achimcc]

@davxy We also want to use the arkworks.rs stuff in other context. @burdges said, a complete native code implementation of a verify function is not what we want, but I've to admit that I don't know the reasoning behind it.

[davxy]

@burdges why we don't want a full native verify?

[bkchr]

@burdges why we don't want a full native verify?

AFAIK the idea is to use artworks as a building library to compose different kind of cryptography algorithms. Adding one verify for each cryptographic algorithm isn't such a great idea in the longterm.

[achimcc]

This all builds and all tests pass! If no one complains, I will just wait for the official arkworks-rs 0.4.0 release and will turn it into Ready for review then!

[burdges]

We could've a thinner substrate change @bkchr but then have an external repo that implements the runtime code for the curve models and curves themselves. I didn't ask for this initially because I figured it'd complicate doing tests, but achim said that's not really true. I agree it's a good idea now that I think about it:

It's true, we need it to be easy to translate PRs from arkworks' to our forked code. It's easy-ish to do this for curves, which live in the flat repo https://github.com/arkworks-rs/curves/ so we could've a similar flat sub-ark-curves repo perhaps.

It's trickier to do curve models as they're nested into the ark-ec crate https://github.com/arkworks-rs/algebra/tree/master/ec/src/models so I figured this winds up being roughly git diff and git apply, but afaik the curve models could live in the same sub-ark-curves repo.

We'll upgrade only intermittently I guess, so really I think sub-ark-curves being separate from substrate itself actually benefits us in that parachains upgrade sub-ark-curves and all their crpytography dependencies independently form substrate itself. We could even have testing infrastructure that uses all supported versions of sub-ark-curves.

---

Arkworks has need generous in permitting our changes into their traits, and we've tried to keep our changes in line with what gpu and fpga projects require, but arkworks success requires being kinda a "standard library". We also want zcash's traits to work via these same host calls, but they'll never permit even the slightest code smell into their traits, so..

We should think carefully about any changes to the arkworks trait hierarchy. In fact, we almost got G2Prepared serialization merged, which risks incompatiblity with some optimizations we like.

[achimcc]

We could've a thinner substrate change @bkchr but then have an external repo that implements the runtime code for the curve models and curves themselves. I didn't ask for this initially because I figured it'd complicate doing tests, but achim said that's not really true. I agree it's a good idea now that I think about it:

It's true, we need it to be easy to translate PRs from arkworks' to our forked code. It's easy-ish to do this for curves, which live in the flat repo https://github.com/arkworks-rs/curves/ so we could've a similar flat sub-ark-curves repo perhaps.

It's trickier to do curve models as they're nested into the ark-ec crate https://github.com/arkworks-rs/algebra/tree/master/ec/src/models so I figured this winds up being roughly git diff and git apply, but afaik the curve models could live in the same sub-ark-curves repo.

We'll upgrade only intermittently I guess, so really I think sub-ark-curves being separate from substrate itself actually benefits us in that parachains upgrade sub-ark-curves and all their crpytography dependencies independently form substrate itself. We could even have testing infrastructure that uses all supported versions of sub-ark-curves.

We have a forked curve repo already which contains the forked arkworks curves: https://github.com/achimcc/ark-substrate

I'm using it for this PR!

[burdges]

Ahh, you've even got the models there, cool.

[achimcc]

We have a funny testing duality: the curves repo is running the tests against the host functions in Substrate and Substrate is running the tests against the curve repo. In both cases we use the curves with host function implementations.

[bkchr]

Is there any need to have the host functions in substrate? From Substrate itself there is no need that the host functions are being part of this repo. Host functions can be defined anywhere.

[achimcc]

AFAIK Sassafrass consensus will use RingVRF's and with this also arkworks and the host functions. I was assuming that the consensus layer needs to be part of the Substrate core repo?

[bkchr]

Good point! If this is the case, we can add it here :)

A general node, the artworks stuff should go into some special "namespace/trait".

[achimcc]

One suggestion from my side: we decided not to Scale encode/decode the host function arguments, I think it makes sense to add some more doc comments instead, about the format of the received/returned arguments and also about the used (de-)serialization tool (ark-scale) @davxy ?

I added the types that each host function receives/returns in encoded Vec<u8> to their corresponding doc comments. If @burdges approves the PR, we can merge :)

[burdges]

I'm happy. :)

I'll also remark that Achim explored the differences between zcash & arkworks internal projective representations. It's be cheap if we ever need to internally adopt their representation.

[achimcc]

It seems like @burdges approval doesn't count for the CI, since he is not from Parity. @arkpar , @bkchr , can one of you help me out here with a second approval?

[achimcc]

bot merge

[tomaka]

Just to make sure: this is completely unstable, correct? And we don't want any relay chain or parachain to actually use them, right?

[achimcc]

Just to make sure: this is completely unstable, correct? And we don't want any relay chain or parachain to actually use them, right?

Yes, correct! We had a discussion about hiding it behind a feature flag up thread: #13031 (comment)

We finally decided not to do this, and to move the host functions into their own crate, instead of providing them as part of sp-io. This way, they will not get deployed on Polkadot/Kusama automatically.

Finally, they are an internal API for ark-substrate and we have a warning there, that we not recommend using this for a live deployment on Polkadot/Kusama at the moment.

[bkchr]

But we will probably before see this on some test network, like Rococo or Westend, but nothing was decided so far on this!

[burdges]

I've posted design notes in https://hackmd.io/@rgbPIkIdTwSICPuAq67Jbw/rkipBanIn