bb.js is standalone prover that handles noir constraint/witness output. #471

charlielye · 2023-05-24T06:43:14Z

Description

We are moving towards standalone prover/verifier binaries to be maintained by bb team(s), and to be consumed by noir team(s) (and whoever else).
This is the first implementation of such a prover/verifier.

Wins:

Completely architecture/os agnostic. As a wasm it runs everywhere, even the browser.
Handles up to 512k gates, proving time 18s on M1 Mac (builds off prior work).
New cleaned up AcirComposer interface that's simple and sane to use.

Todo:

Actually parse some command line arguments to make it flexible, rather than running a test loop.
Publish it to npm as part of version tagged releases.
Build native versions that follow same process cli args interface. (Think like AC's rollup_cli).
Stateless and stateful runs (stateful, like rollup_cli, uses stdin/stdout).

Checklist:

…467)

* convert partially evaluated polynomials from vectors to Polynomials and rename * rename fold method to partially_evaluate * static constexpr barycentric arrays * change purported evaluations to claimed evaluations * specify relations in Flavor

…g into cl/acir_cbinds

vezenovm · 2023-05-25T18:21:43Z

cpp/src/barretenberg/dsl/acir_proofs/acir_composer.cpp

+    exact_circuit_size_ = composer_.get_num_gates();
+    total_circuit_size_ = composer_.get_total_circuit_size();
+    // Exact or total fed in here?
+    circuit_subgroup_size_ = composer_.get_circuit_subgroup_size(exact_circuit_size_);


This should be total_circuit_size

* add initial KeccakVar code * add result field * add keccak_var_constraints to fields

Allow for correct and efficient batching over identities in the Sumcheck relation

…g into cl/acir_cbinds

* Added recursion constraint into dsl + tests * fix dsl ecdsa constraint. Added ecdsa dsl tests * changed dsl recursion to pass proof/key as witnesses * recursive verification key no longer encapsulates a native key recursive verification key no longer encapsulates a reference string (neither were fundamentally needed!) * added serialization test for acir_proofs * Added serialization methods into dsl for recursive proof composition Added passing serialization tests Fixed format of RecursionConstraint to be compatible with existing ACIR formatting * added verificaiton key hash into RecursionConstraint Can be used by backend to force recursive verification key to be a specific value (it is represented via witnesses and therefore is not by-default constraint to represent a specific circuit) * fixed compiler errors in stdlib_recursion_tests * feat(dsl)!: Noir recursion updates (#379) * merge master in zw/noir-recursion * add dsl files to commit * namespace stuff and debugging * c bind functions * constraint test and comment removal * revert some changes to RecursionConstraint while debugging serialization issues * dispaly error when trying to use create_circuit_with_witness * is_recursive flag as part of new_proof and verify_proof, new RecursiveProver type in dsl, and serialization changes for recursion acir_proofs methods * remove debug output from TestSerializationWithRecursion * master merge conflicts * EnvReferenceString undefined reference to error workaround * add missing recursion_constraint field from acir_format tests * add recursion_constraints field back in acir_proofs.test inner circuit * fix inner circuit in RecursionConstraint.TestRecursionConstraint * Empty-Commit * add verify_recursive_proof method for simulating recursive verification in the ACVM * add ecc module to env package to fix linking of ennv_load_prover_crs and env_load_verifier_crs * add back reference strinng to stdlib recursion key to pass sol verifier generation tests * remove prints from recursive verifier test * fix dirty free for when serializing vk to fields, was working on macbook so not caught earlier, but need more clarity on ubuntu * fix ecdsa tests after master merge * missing keccak constraints fields in acir format and proofs tests * one more missing keccak constraint * mismatched acir format structs for gcc build * missing block constraints in acir tests * feat(dsl)!: Arbitrary depth recursion (#433) * merge conflicts and small updates to get acir_proofs test passing with arbitrary depth cahnges * inline with mv/noir-recursion and working double recursion acir_proofs serialization test * cleanup and working towards supporting full nested proofs, still some bugs in acir_proofs test * full recursive composition test working in acir_proofs * use two public inptus for inner circuit * delete commented out unnecessary acir proofs method * made dummy transcript points unique to prevent point collisions * handle nested proofs when exporting dummy transcript, export recursive proof indices when init vkey, fix full recursive comp test in acir_proofs * update bindings on verify_recursive_proof to accept num public inputs * missing acir_format field in tests * missing one more correct acir_format struct in recursion constraint test * cleanup and additional comment for recursion_constraint * fix up comment in recursion_constraint * remove unnecesary comments when we were including proof outputs as public inputs in the recursion constraint --------- Co-authored-by: zac-williamson <blorktronics@gmail.com> * move export key into dsl package * chore: remove unused export key in recursion format from main proof system classes * pr review: moved from_witness comment and renamed from_field_pt_vector * moved export transcript in recursion format to DSL package * move order of acir functions * remove ecc bb_module declaration in env package * chore: remove usage of magic numbers when slicing g1::affine_element into barretenberg::fr * introduce NUM_AGGREGATION_ELEMENTS constant and use it through recursion constraint * nit ASSERT(result != -1) in get_public_input_index * remove unused found var * cast -1 * fix up verify_recursive_proof and add a test for it * moved from tempalte for has_valid_witness_assignments to a flag * chore: add comments to AcirProofs.TestVerifyRecursiveProofPass * update prove and verify exports to handle is_recursive * remove install state * remove debug output and use total_circuit_size * remove verify_recursive_proof as we might not need it for simulation * switch how we do is_recursive * remove acir recursive simulation method * changed wasi stubs and removed cout from acir_init_proving_key * removed cout outs getting runtime error for func definition * working recursion format funcs, but strangely getting errors about trying to invert zero in the field for dummy transcript and key * most recent debug output, hitting bad alloc in compute_proving_key_base * delete old comment * bb.js cmd changes * Lint and default fixes. * cleanup info and debug comments * add comment to recursion serialization methods * match package.json on cl/acir_cbinds * don't remove EventEmitter from bb_wasm.ts * delete empty exports file * use process.exit * log * add comment for why CIRCUIT_SIZE is 2**18 * mising keccak var constraints in new acir format tests * Cleanup some serialization. * Cleanup for publishing. README. --------- Co-authored-by: zac-williamson <blorktronics@gmail.com> Co-authored-by: Charlie Lye <karl.lye@gmail.com>

…5s wasm time). (#434) * Remove foundation submodule, add build-system * Initial bindgen work. * GPT4 created rust bindgen. * Debug build preset wasm-dbg for stack traces etc. * Async call support. Refactor binds. Common is module. WASM stubs unused wasi. * Move some c_bind to module root and make use new abi. * Initial working pthreads. * Polynomial cache * Custom poly allocator. Prealloc. * Cleanup. Several parallel_for solutions. Might be worth keeping them around until threading work stabilises. * Improve mem usage by slab allocating large parts of pippenger runtime state, and allowing slab reuse by removing from pk. * Don't need binaryen. * Bump alpine to get clang 16 * Backwards compatible cbinds. * Move slab allocator to common. Init slab allocator just before creating pk. * Dont need bleeding edge fork of libc anymore * bb.js is standalone prover that handles noir constraint/witness output. (#471) * format msgpack serialization and excldue msgpack-c from clang-format (#467) * Fix horrific nasty memory bug. --------- Co-authored-by: Suyash Bagad <suyashnbagad1997@gmail.com> Co-authored-by: Maxim Vezenov <mvezenov@gmail.com> Co-authored-by: Maddiaa <47148561+Maddiaa0@users.noreply.github.com> Co-authored-by: ledwards2225 <98505400+ledwards2225@users.noreply.github.com> Co-authored-by: Innokentii Sennovskii <isennovskiy@gmail.com> Co-authored-by: kevaundray <kevtheappdev@gmail.com> Co-authored-by: zac-williamson <blorktronics@gmail.com> Co-authored-by: codygunton <codygunton@gmail.com>

…5s wasm time). (AztecProtocol/barretenberg#434) * Remove foundation submodule, add build-system * Initial bindgen work. * GPT4 created rust bindgen. * Debug build preset wasm-dbg for stack traces etc. * Async call support. Refactor binds. Common is module. WASM stubs unused wasi. * Move some c_bind to module root and make use new abi. * Initial working pthreads. * Polynomial cache * Custom poly allocator. Prealloc. * Cleanup. Several parallel_for solutions. Might be worth keeping them around until threading work stabilises. * Improve mem usage by slab allocating large parts of pippenger runtime state, and allowing slab reuse by removing from pk. * Don't need binaryen. * Bump alpine to get clang 16 * Backwards compatible cbinds. * Move slab allocator to common. Init slab allocator just before creating pk. * Dont need bleeding edge fork of libc anymore * bb.js is standalone prover that handles noir constraint/witness output. (AztecProtocol/barretenberg#471) * format msgpack serialization and excldue msgpack-c from clang-format (AztecProtocol/barretenberg#467) * Fix horrific nasty memory bug. --------- Co-authored-by: Suyash Bagad <suyashnbagad1997@gmail.com> Co-authored-by: Maxim Vezenov <mvezenov@gmail.com> Co-authored-by: Maddiaa <47148561+Maddiaa0@users.noreply.github.com> Co-authored-by: ledwards2225 <98505400+ledwards2225@users.noreply.github.com> Co-authored-by: Innokentii Sennovskii <isennovskiy@gmail.com> Co-authored-by: kevaundray <kevtheappdev@gmail.com> Co-authored-by: zac-williamson <blorktronics@gmail.com> Co-authored-by: codygunton <codygunton@gmail.com>

charlielye and others added 11 commits May 21, 2023 18:32

Initial

0038bb6

Separate thread build dir.

284eac2

Large vector slab allocator.

afef842

format msgpack serialization and excldue msgpack-c from clang-format (#…

4c39ac6

…467)

patch: temporarily remove broken solidity ci (#470)

e1318c1

progress.

efc3926

Sumcheck improvements (#455)

0e7d32f

* convert partially evaluated polynomials from vectors to Polynomials and rename * rename fold method to partially_evaluate * static constexpr barycentric arrays * change purported evaluations to claimed evaluations * specify relations in Flavor

working. but cant compute pk independently :/

0e0b04d

Fixed.

b2635b6

Better key construction separation.

2a7e1d0

Merge remote-tracking branch 'origin/master' into cl/acir_cbinds

6477f82

charlielye changed the base branch from master to cl/wasm_pthread May 24, 2023 06:44

charlielye and others added 13 commits May 24, 2023 07:23

Clean

f108b22

tests pass

130c944

Commander

72e4c98

Fix.

b600d99

Fix

e51ab64

Fix serve.

14acc50

Fix

e660b55

More commands.

d3278fd

Fixed a bug in biggroup tests (#478)

ad28215

Add back custom libc++ build.

ada5d9d

Fix.

f815187

write_vk

0fee875

Merge branch 'cl/acir_cbinds' of github.com:aztecprotocol/barretenber…

fa7984a

…g into cl/acir_cbinds

vezenovm reviewed May 25, 2023

View reviewed changes

kevaundray and others added 4 commits May 25, 2023 21:17

DSL: Add KeccakVar opcode (#476)

0834791

* add initial KeccakVar code * add result field * add keccak_var_constraints to fields

Multi-constraint Relations (#444)

4d851dd

Allow for correct and efficient batching over identities in the Sumcheck relation

Clean up srs package. Add a global crs that the wasm can init once.

dff766b

Add back vk serialization for a3

97a2fdf

charlielye added 4 commits May 27, 2023 02:26

Clean

04b6024

Merge remote-tracking branch 'origin/master' into cl/acir_cbinds

cf4b674

Merge branch 'cl/acir_cbinds' of github.com:aztecprotocol/barretenber…

7065b1f

…g into cl/acir_cbinds

Exclude env from lib.

7cfc53a

vezenovm mentioned this pull request May 30, 2023

feat(dsl)!: DSL recursion changes w/ bb.js #485

Merged

11 tasks

charlielye changed the base branch from cl/wasm_pthread to master May 31, 2023 15:10

charlielye changed the base branch from master to cl/wasm_pthread June 1, 2023 11:26

charlielye merged commit f9bfbe7 into cl/wasm_pthread Jun 1, 2023

charlielye deleted the cl/acir_cbinds branch June 1, 2023 11:27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

bb.js is standalone prover that handles noir constraint/witness output. #471

bb.js is standalone prover that handles noir constraint/witness output. #471

charlielye commented May 24, 2023

vezenovm May 25, 2023

bb.js is standalone prover that handles noir constraint/witness output. #471

bb.js is standalone prover that handles noir constraint/witness output. #471

Conversation

charlielye commented May 24, 2023

Description

Checklist:

vezenovm May 25, 2023

Choose a reason for hiding this comment