Commitment scheme abstraction #77

kilic · 2022-06-02T15:28:40Z

Having pse/halo2curves which reimplements pse/pairing with upstream traits and also reexports zcash/pasta_curves it is now possible to keep original commitment scheme rather than replacing it.

Also since we won't have to deal with mimicking traits in pse/pasta_curves at pse/pairing there must be less diffs in this fork and won't require any maintenance effort regarding those traits.

This PR introduces an abstraction technique for commitment schemes. It keeps the original IPA and adds two KZG variants and allows circuit developer to specify a commitment scheme for a specific circuit without conditional compilation.

Regarding the changes at constraint system side most of them are for boundary definitions of functions which mostly specifies the commitment scheme. I don't think this abstraction change would make its way to the upstream. However it can be a discussion base for a working group if idea is found useful somehow.

For further rebasing and maintenance concerns:

`mod poly`:

30 files changed, 3365 insertions(+), 1411 deletions(-)

This is admittedly a huge diff. However current replacement approach change for �mod poly is also not small.

Integration for `halo2_proofs` except `mod poly`:

35 files changed, 623 insertions(+), 578 deletions(-)

There are some formatting/folding related differences that should reduce sense of this diff. There is also a ~100 liner block diff w/o an actual change. I think it is for an unrecognized line shift.

Integration for halo2_gadgets:

37 files changed, 117 insertions(+), 88 deletions(-)

Which looks ok.

CPerezz · 2022-06-06T09:16:26Z

@kilic will take a look into this.

I'll try to see if there's a way on which we can do the same work you did here but resulting in less conflicts when merging upstream changes.

I'll need a day or two as I'll parallelize this with my keccak work :)

han0110

Finished KZG part and abstraction on CommitmentScheme looks really great! Nice work!

Will review IPA next.

halo2_gadgets/benches/sha256.rs

halo2_proofs/src/plonk/prover.rs

halo2_proofs/src/poly/kzg/multiopen/gwc.rs

halo2_proofs/src/poly/kzg/multiopen/shplonk.rs

halo2_proofs/tests/plonk_api.rs

halo2_proofs/src/poly/query.rs

CPerezz

Missing gwc/verifier, multiopen and shplonk.

Will do them later today!

halo2_proofs/src/poly/strategy.rs

halo2_proofs/src/poly/ipa/commitment.rs

halo2_proofs/src/poly/ipa/strategy.rs

halo2_proofs/src/poly/kzg/commitment.rs

CPerezz · 2022-06-08T08:55:18Z

halo2_proofs/src/poly/kzg/commitment.rs

+
+#[cfg(test)]
+mod test {
+    // TODO: duplicate is in IPA side


I belive this is already done. So let's remove the comment.

Actually contrary. It supposed to mean make these tests generic and use it across schemes. So still a TODO :)

Hmm that's weird. I saw all the tests under ipa.rs. But maybe weren't the same ones.

So if the sentence meant something else then my bad.

If we want to abstract the tests over commitment scheme, here is a suggestion on how we do it at ZKGarage: https://github.com/ZK-Garage/plonk/blob/79dffa1bacbe73ab42e2d7e48194efe5c0070bd6/plonk-core/src/test.rs#L84

halo2_proofs/src/poly/kzg/multiopen/gwc.rs

halo2_proofs/src/poly/kzg/multiopen/gwc/prover.rs

halo2_proofs/src/poly/kzg/multiopen/shplonk.rs

CPerezz

LGTM.

Just a nit/suggestion.

halo2_proofs/src/poly/kzg/commitment.rs

rust-toolchain

CPerezz · 2022-06-16T07:32:53Z

Hey @kilic why did you rebase upstream head here??? I thought this was being done in another PR.

See: #64

kilic · 2022-06-16T12:31:58Z

@CPerezz

I thought In the end this PR should be on top of recent upstream and also under other changes that we introduce? It was already started 3 or 5 commits behind of the current upstream head.

CPerezz · 2022-06-16T12:50:10Z

@kilic makes sense to me. But we should ping @noctrlz since I think is working on it.

[book] Add `U` to `BCMS` comparison table

Add `BatchVerifier::finalize_and_return_rng`

This means we only need to `Debug`-format the `PinnedVerificationKey` once on construction, instead of once per proof.

This means we only compute the degree in a verification context during construction, instead of twice per proof in the permutation argument.

Cache values in `VerifyingKey` that can be computed on construction

…ests. Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Reduce memory overhead of MSM

MSM optimizations

`BatchVerifier` now manages the entire batch verification process. Individual proofs are verified on a threadpool, and the resulting MSMs are then batch-checked as before. The addition of parallelism here couples with zcash#608 to make parallelism less fine-grained and reduce the overhead of multi-threading.

Remove book ci

Apply suggestions: Use `Vec` instead of `BTreeMap` Co-authored-by: Han <tinghan0110@gmail.com>

Apply suggestions. Use `Vec` instead of `BTreeMap` Co-authored-by: Han <tinghan0110@gmail.com>

…alo2 benches with main

CPerezz · 2023-01-16T23:41:16Z

I belive this can be closed right @kilic @han0110 ??

…tions#77 (#6) * Resolve taikoxyz/zkevm-circuits#77 * Please Clippy

…17) * Resolve Prover optimization: memory reduction privacy-scaling-explorations#77 (#6) * Resolve taikoxyz/zkevm-circuits#77 * Please Clippy * fix: recursive FFT for lengths not `2^k, 2^extended_k` Also fixed some issue when multicore feature is not on. * fix: evaluation (cherry-pick https://github.com/scroll-tech/halo2-gpu/blob/a3019b047ad7d7119103f4d9df3fd33ce1429f95/halo2_proofs/src/plonk/evaluation.rs) * fix: `g_to_lagrange` uses inverse fft * Bump version since this is a breaking change to pk * chore: update halo2curves version --------- Co-authored-by: einar-taiko <126954546+einar-taiko@users.noreply.github.com>

CPerezz self-requested a review June 6, 2022 09:15

han0110 reviewed Jun 8, 2022

View reviewed changes

CPerezz reviewed Jun 8, 2022

View reviewed changes

kilic force-pushed the abstraction branch 4 times, most recently from d13862e to 04ffc45 Compare June 10, 2022 10:25

han0110 reviewed Jun 13, 2022

View reviewed changes

halo2_proofs/src/poly/kzg/multiopen/shplonk.rs Outdated Show resolved Hide resolved

CPerezz approved these changes Jun 13, 2022

View reviewed changes

halo2_proofs/src/poly/kzg/commitment.rs Show resolved Hide resolved

rust-toolchain Outdated Show resolved Hide resolved

[book] Add U to BCMS comparison table

69a980f

kilic force-pushed the abstraction branch from 8cbf9b6 to e112ee4 Compare June 15, 2022 16:13

Merge pull request zcash#601 from zcash/book/update-comparison

c9fc4c6

[book] Add `U` to `BCMS` comparison table

This was referenced Jun 21, 2022

Optional zero-knowledge by const generic #76

Closed

[RELEASE-Proposal] PRs to merge before we cut a new release for halo2 #82

Closed

str4d and others added 7 commits June 21, 2022 15:04

Add BatchVerifier::finalize_and_return_rng

22ec369

Merge pull request zcash#603 from zcash/batch-verifier-return-rng

f6efecc

Add `BatchVerifier::finalize_and_return_rng`

Cache the representative of VerifyingKey used in transcripts

4802936

This means we only need to `Debug`-format the `PinnedVerificationKey` once on construction, instead of once per proof.

Cache the constraint system degree in VerifyingKey

05f37ee

This means we only compute the degree in a verification context during construction, instead of twice per proof in the permutation argument.

Remove parallelism from within MSM scalar operations.

6939ac4

Deduplicate the extra (scalar, base) pairs in MSMs.

6b4af24

Merge pull request zcash#607 from zcash/caching

745f5d1

Cache values in `VerifyingKey` that can be computed on construction

han0110 approved these changes Jun 23, 2022

View reviewed changes

daira and others added 4 commits June 23, 2022 14:15

Reduce memory overhead of MSM by 64 bytes per "other" base, and add t…

2ed70a3

…ests. Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Merge pull request #1 from daira/msm-optimization-daira

c1159bd

Reduce memory overhead of MSM

Merge pull request zcash#608 from ebfull/msm-optimization

7087c33

MSM optimizations

kilic and others added 18 commits July 4, 2022 23:39

Update toolchain to enable asm feature

8265d8c

Remove book ci

poly/commitment/Prover: move Rng bound to create_proof function

0d9f780

Add tag to halo2curves dep

5f24873

Remove leftovers

725ad7c

Handle serialisation panics

7e82484

Fix gwc intermediate set construction

e59a83a

Apply suggestions: Use `Vec` instead of `BTreeMap` Co-authored-by: Han <tinghan0110@gmail.com>

Fix shplonk intermediate set construction

922e345

Apply suggestions. Use `Vec` instead of `BTreeMap` Co-authored-by: Han <tinghan0110@gmail.com>

Restore proptests

a8cb527

Fix pasta dependency at Value doctest

d5bf7ca

Fix rebasing residues

3f76210

Restore expected verification key check

dc27b59

Update toolchain

b188f27

Apply clippy

2976e4e

feat: use generic for CommitmentReference

8c90616

fix: compare rotation set by HashMap in shplonk

d2d35fc

feat: remove lifetime param from CommitmentScheme

43f1211

Removed bench.yml, added trigger_proverbench_dispatch.yml to allign h…

dd7c385

…alo2 benches with main

fix: integrate verification strategies

8c94550

kilic force-pushed the abstraction branch from d1332d7 to 8c94550 Compare July 5, 2022 15:37

han0110 mentioned this pull request Jul 8, 2022

Commitment scheme abstraction with all previous improvements #84

Merged

kilic closed this Jan 17, 2023

mratsim pushed a commit to taikoxyz/halo2 that referenced this pull request Jun 26, 2023

Resolve Prover optimization: memory reduction privacy-scaling-explora…

9eaccbb

…tions#77 (#6) * Resolve taikoxyz/zkevm-circuits#77 * Please Clippy

jonathanpwang pushed a commit to axiom-crypto/halo2 that referenced this pull request Oct 26, 2023

Resolve Prover optimization: memory reduction privacy-scaling-explora…

a530f5a

…tions#77 (#6) * Resolve taikoxyz/zkevm-circuits#77 * Please Clippy

mratsim mentioned this pull request Oct 19, 2023

Fast modular inverse - 9.4x acceleration privacy-scaling-explorations/halo2curves#83

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Commitment scheme abstraction #77

Commitment scheme abstraction #77

kilic commented Jun 2, 2022 •

edited

Loading

CPerezz commented Jun 6, 2022

han0110 left a comment

CPerezz left a comment

CPerezz Jun 8, 2022

kilic Jun 8, 2022 •

edited

Loading

CPerezz Jun 13, 2022

CPerezz left a comment

CPerezz commented Jun 16, 2022

kilic commented Jun 16, 2022 •

edited

Loading

CPerezz commented Jun 16, 2022

CPerezz commented Jan 16, 2023

Commitment scheme abstraction #77

Commitment scheme abstraction #77

Conversation

kilic commented Jun 2, 2022 • edited Loading

mod poly:

Integration for halo2_proofs except mod poly:

Integration for halo2_gadgets:

CPerezz commented Jun 6, 2022

han0110 left a comment

Choose a reason for hiding this comment

CPerezz left a comment

Choose a reason for hiding this comment

CPerezz Jun 8, 2022

Choose a reason for hiding this comment

kilic Jun 8, 2022 • edited Loading

Choose a reason for hiding this comment

CPerezz Jun 13, 2022

Choose a reason for hiding this comment

CPerezz left a comment

Choose a reason for hiding this comment

CPerezz commented Jun 16, 2022

kilic commented Jun 16, 2022 • edited Loading

CPerezz commented Jun 16, 2022

CPerezz commented Jan 16, 2023

kilic commented Jun 2, 2022 •

edited

Loading

`mod poly`:

Integration for `halo2_proofs` except `mod poly`:

kilic Jun 8, 2022 •

edited

Loading

kilic commented Jun 16, 2022 •

edited

Loading