feat(dsl)!: DSL recursion changes w/ bb.js (#485)

* Added recursion constraint into dsl + tests

* fix dsl ecdsa constraint. Added ecdsa dsl tests

* changed dsl recursion to pass proof/key as witnesses

* recursive verification key no longer encapsulates a native key

recursive verification key no longer encapsulates a reference string

(neither were fundamentally needed!)

* added serialization test for acir_proofs

* Added serialization methods into dsl for recursive proof composition

Added passing serialization tests

Fixed format of RecursionConstraint to be compatible with existing ACIR formatting

* added verificaiton key hash into RecursionConstraint

Can be used by backend to force recursive verification key to be a specific value (it is represented via witnesses and therefore is not by-default constraint to represent a specific circuit)

* fixed compiler errors in stdlib_recursion_tests

* feat(dsl)!: Noir recursion updates (#379)

* merge master in zw/noir-recursion

* add dsl files to commit

* namespace stuff and debugging

* c bind functions

* constraint test and comment removal

* revert some changes to RecursionConstraint while debugging serialization issues

* dispaly error when trying to use create_circuit_with_witness

* is_recursive flag as part of new_proof and verify_proof, new RecursiveProver type in dsl, and serialization changes for recursion acir_proofs methods

* remove debug output from TestSerializationWithRecursion

* master merge conflicts

* EnvReferenceString undefined reference to  error workaround

* add missing recursion_constraint field from acir_format tests

* add recursion_constraints field back in acir_proofs.test inner circuit

* fix inner circuit in RecursionConstraint.TestRecursionConstraint

* Empty-Commit

* add verify_recursive_proof method for simulating recursive verification in the ACVM

* add ecc module to env package to fix linking of ennv_load_prover_crs and env_load_verifier_crs

* add back reference strinng to stdlib recursion key to pass sol verifier generation tests

* remove prints from recursive verifier test

* fix dirty free for when serializing vk to fields, was working on macbook so not caught earlier, but need more clarity on ubuntu

* fix ecdsa tests after master merge

* missing keccak constraints fields in acir format and proofs tests

* one more missing keccak constraint

* mismatched acir format structs for gcc build

* missing block constraints in acir tests

* feat(dsl)!: Arbitrary depth recursion (#433)

* merge conflicts and small updates to get acir_proofs test passing with arbitrary depth cahnges

* inline with mv/noir-recursion and working double recursion acir_proofs serialization test

* cleanup and working towards supporting full nested proofs, still some bugs in acir_proofs test

* full recursive composition test working in acir_proofs

* use two public inptus for inner circuit

* delete commented out unnecessary acir proofs method

* made dummy transcript points unique to prevent point collisions

* handle nested proofs when exporting dummy transcript, export recursive proof indices when init vkey, fix full recursive comp test in acir_proofs

* update bindings on verify_recursive_proof to accept num public inputs

* missing acir_format field in tests

* missing one more correct acir_format struct in recursion constraint test

* cleanup and additional comment for recursion_constraint

* fix up comment in recursion_constraint

* remove unnecesary comments when we were including proof outputs as public inputs in the recursion constraint

---------

Co-authored-by: zac-williamson <blorktronics@gmail.com>

* move export key into dsl package

* chore: remove unused export key in recursion format from main proof system classes

* pr review: moved from_witness comment and renamed from_field_pt_vector

* moved export transcript in recursion format to DSL package

* move order of acir functions

* remove ecc bb_module declaration in env package

* chore: remove usage of magic numbers when slicing g1::affine_element into barretenberg::fr

* introduce NUM_AGGREGATION_ELEMENTS constant and use it through recursion constraint

* nit ASSERT(result != -1) in get_public_input_index

* remove unused found var

* cast -1

* fix up verify_recursive_proof and add a test for it

* moved from tempalte for has_valid_witness_assignments to a flag

* chore: add comments to AcirProofs.TestVerifyRecursiveProofPass

* update prove and verify exports to handle is_recursive

* remove install state

* remove debug output and use total_circuit_size

* remove verify_recursive_proof as we might not need it for simulation

* switch how we do is_recursive

* remove acir recursive simulation method

* changed wasi stubs and removed cout from acir_init_proving_key

* removed cout outs getting runtime error for func definition

* working recursion format funcs, but strangely getting errors about trying to invert zero in the field for dummy transcript and key

* most recent debug output, hitting bad alloc in compute_proving_key_base

* delete old comment

* bb.js cmd changes

* Lint and default fixes.

* cleanup info and debug comments

* add comment to recursion serialization methods

* match package.json on cl/acir_cbinds

* don't remove EventEmitter from bb_wasm.ts

* delete empty exports file

* use process.exit

* log

* add comment for why CIRCUIT_SIZE is 2**18

* mising keccak var constraints in new acir format tests

* Cleanup some serialization.

* Cleanup for publishing. README.

---------

Co-authored-by: zac-williamson <blorktronics@gmail.com>
Co-authored-by: Charlie Lye <karl.lye@gmail.com>

cpp/src/barretenberg/common/slab_allocator.cpp

@@ -195,9 +195,9 @@ SlabAllocator allocator;
 } // namespace
 
 namespace barretenberg {
-void init_slab_allocator(size_t circuit_size)
+void init_slab_allocator(size_t circuit_subgroup_size)
 {
-    allocator.init(circuit_size);
+    allocator.init(circuit_subgroup_size);
 }
 
 // auto init = ([]() {

cpp/src/barretenberg/common/slab_allocator.hpp

@@ -25,7 +25,7 @@ namespace barretenberg {
  * TODO: De-globalise. Init the allocator and pass around. Use a PolynomialFactory (PolynomialStore?).
  * TODO: Consider removing, but once due-dilligence has been done that we no longer have memory limitations.
  */
-void init_slab_allocator(size_t circuit_size);
+void init_slab_allocator(size_t circuit_subgroup_size);
 
 /**
  * Returns a slab from the preallocated pool of slabs, or fallback to a new heap allocation (32 byte aligned).

cpp/src/barretenberg/dsl/acir_format/acir_format.cpp

@@ -96,6 +96,21 @@ void create_circuit(Composer& composer, acir_format const& constraint_system)
     for (const auto& constraint : constraint_system.block_constraints) {
         create_block_constraints(composer, constraint);
     }
+
+    // Add recursion constraints
+    for (size_t i = 0; i < constraint_system.recursion_constraints.size(); ++i) {
+        auto& constraint = constraint_system.recursion_constraints[i];
+        create_recursion_constraints(composer, constraint);
+
+        // make sure the verification key records the public input indices of the final recursion output
+        // (N.B. up to the ACIR description to make sure that the final output aggregation object wires are public
+        // inputs!)
+        if (i == constraint_system.recursion_constraints.size() - 1) {
+            std::vector<uint32_t> proof_output_witness_indices(constraint.output_aggregation_object.begin(),
+                                                               constraint.output_aggregation_object.end());
+            composer.set_recursive_proof(proof_output_witness_indices);
+        }
+    }
 }
 
 Composer create_circuit(const acir_format& constraint_system,
@@ -120,6 +135,7 @@ Composer create_circuit(const acir_format& constraint_system,
             composer.add_variable(0);
         }
     }
+
     // Add arithmetic gates
     for (const auto& constraint : constraint_system.constraints) {
         composer.create_poly_gate(constraint);
@@ -189,6 +205,21 @@ Composer create_circuit(const acir_format& constraint_system,
         create_block_constraints(composer, constraint);
     }
 
+    // Add recursion constraints
+    for (size_t i = 0; i < constraint_system.recursion_constraints.size(); ++i) {
+        auto& constraint = constraint_system.recursion_constraints[i];
+        create_recursion_constraints(composer, constraint);
+
+        // make sure the verification key records the public input indices of the final recursion output
+        // (N.B. up to the ACIR description to make sure that the final output aggregation object wires are public
+        // inputs!)
+        if (i == constraint_system.recursion_constraints.size() - 1) {
+            std::vector<uint32_t> proof_output_witness_indices(constraint.output_aggregation_object.begin(),
+                                                               constraint.output_aggregation_object.end());
+            composer.set_recursive_proof(proof_output_witness_indices);
+        }
+    }
+
     return composer;
 }
 
@@ -286,6 +317,21 @@ Composer create_circuit_with_witness(acir_format const& constraint_system,
         create_block_constraints(composer, constraint);
     }
 
+    // Add recursion constraints
+    for (size_t i = 0; i < constraint_system.recursion_constraints.size(); ++i) {
+        auto& constraint = constraint_system.recursion_constraints[i];
+        create_recursion_constraints(composer, constraint, true);
+
+        // make sure the verification key records the public input indices of the final recursion output
+        // (N.B. up to the ACIR description to make sure that the final output aggregation object wires are public
+        // inputs!)
+        if (i == constraint_system.recursion_constraints.size() - 1) {
+            std::vector<uint32_t> proof_output_witness_indices(constraint.output_aggregation_object.begin(),
+                                                               constraint.output_aggregation_object.end());
+            composer.set_recursive_proof(proof_output_witness_indices);
+        }
+    }
+
     return composer;
 }
 Composer create_circuit_with_witness(const acir_format& constraint_system, WitnessVector const& witness)
@@ -380,6 +426,21 @@ Composer create_circuit_with_witness(const acir_format& constraint_system, Witne
         create_block_constraints(composer, constraint);
     }
 
+    // Add recursion constraints
+    for (size_t i = 0; i < constraint_system.recursion_constraints.size(); ++i) {
+        auto& constraint = constraint_system.recursion_constraints[i];
+        create_recursion_constraints(composer, constraint, true);
+
+        // make sure the verification key records the public input indices of the final recursion output
+        // (N.B. up to the ACIR description to make sure that the final output aggregation object wires are public
+        // inputs!)
+        if (i == constraint_system.recursion_constraints.size() - 1) {
+            std::vector<uint32_t> proof_output_witness_indices(constraint.output_aggregation_object.begin(),
+                                                               constraint.output_aggregation_object.end());
+            composer.set_recursive_proof(proof_output_witness_indices);
+        }
+    }
+
     return composer;
 }
 void create_circuit_with_witness(Composer& composer, acir_format const& constraint_system, WitnessVector const& witness)
@@ -471,6 +532,21 @@ void create_circuit_with_witness(Composer& composer, acir_format const& constrai
     for (const auto& constraint : constraint_system.block_constraints) {
         create_block_constraints(composer, constraint);
     }
+
+    // Add recursion constraints
+    for (size_t i = 0; i < constraint_system.recursion_constraints.size(); ++i) {
+        auto& constraint = constraint_system.recursion_constraints[i];
+        create_recursion_constraints(composer, constraint, true);
+
+        // make sure the verification key records the public input indices of the final recursion output
+        // (N.B. up to the ACIR description to make sure that the final output aggregation object wires are public
+        // inputs!)
+        if (i == constraint_system.recursion_constraints.size() - 1) {
+            std::vector<uint32_t> proof_output_witness_indices(constraint.output_aggregation_object.begin(),
+                                                               constraint.output_aggregation_object.end());
+            composer.set_recursive_proof(proof_output_witness_indices);
+        }
+    }
 }
 
 } // namespace acir_format

cpp/src/barretenberg/dsl/acir_format/acir_format.hpp

@@ -9,6 +9,7 @@
 #include "schnorr_verify.hpp"
 #include "ecdsa_secp256k1.hpp"
 #include "compute_merkle_root_constraint.hpp"
+#include "recursion_constraint.hpp"
 #include "block_constraint.hpp"
 #include "pedersen.hpp"
 #include "hash_to_field.hpp"
@@ -35,6 +36,7 @@ struct acir_format {
     std::vector<PedersenConstraint> pedersen_constraints;
     std::vector<ComputeMerkleRootConstraint> compute_merkle_root_constraints;
     std::vector<BlockConstraint> block_constraints;
+    std::vector<RecursionConstraint> recursion_constraints;
     // A standard plonk arithmetic constraint, as defined in the poly_triple struct, consists of selector values
     // for q_M,q_L,q_R,q_O,q_C and indices of three variables taking the role of left, right and output wire
     // This could be a large vector so use slab allocator, we don't expect the blackbox implementations to be so large.
@@ -81,6 +83,7 @@ template <typename B> inline void read(B& buf, acir_format& data)
     read(buf, data.pedersen_constraints);
     read(buf, data.hash_to_field_constraints);
     read(buf, data.fixed_base_scalar_mul_constraints);
+    read(buf, data.recursion_constraints);
     read(buf, data.constraints);
     read(buf, data.block_constraints);
 }
@@ -102,6 +105,7 @@ template <typename B> inline void write(B& buf, acir_format const& data)
     write(buf, data.pedersen_constraints);
     write(buf, data.hash_to_field_constraints);
     write(buf, data.fixed_base_scalar_mul_constraints);
+    write(buf, data.recursion_constraints);
     write(buf, data.constraints);
     write(buf, data.block_constraints);
 }

cpp/src/barretenberg/dsl/acir_format/acir_format.test.cpp

@@ -5,6 +5,51 @@
 #include "barretenberg/common/streams.hpp"
 #include "barretenberg/serialize/test_helper.hpp"
 #include "ecdsa_secp256k1.hpp"
+
+TEST(acir_format, test_a_single_constraint_no_pub_inputs)
+{
+
+    poly_triple constraint{
+        .a = 1,
+        .b = 2,
+        .c = 3,
+        .q_m = 0,
+        .q_l = 1,
+        .q_r = 1,
+        .q_o = -1,
+        .q_c = 0,
+    };
+
+    acir_format::acir_format constraint_system{
+        .varnum = 4,
+        .public_inputs = {},
+        .fixed_base_scalar_mul_constraints = {},
+        .logic_constraints = {},
+        .range_constraints = {},
+        .schnorr_constraints = {},
+        .ecdsa_constraints = {},
+        .sha256_constraints = {},
+        .blake2s_constraints = {},
+        .keccak_constraints = {},
+        .keccak_var_constraints = {},
+        .hash_to_field_constraints = {},
+        .pedersen_constraints = {},
+        .compute_merkle_root_constraints = {},
+        .block_constraints = {},
+        .recursion_constraints = {},
+        .constraints = { constraint },
+    };
+
+    auto composer = acir_format::create_circuit_with_witness(constraint_system, { 0, 0, 1 });
+
+    auto prover = composer.create_ultra_with_keccak_prover();
+    auto proof = prover.construct_proof();
+
+    auto verifier = composer.create_ultra_with_keccak_verifier();
+
+    EXPECT_EQ(verifier.verify_proof(proof), false);
+}
+
 TEST(acir_format, msgpack_logic_constraint)
 {
     auto [actual, expected] = msgpack_roundtrip(acir_format::LogicConstraint{});
@@ -97,6 +142,7 @@ TEST(acir_format, test_logic_gate_from_noir_circuit)
         .pedersen_constraints = {},
         .compute_merkle_root_constraints = {},
         .block_constraints = {},
+        .recursion_constraints = {},
         .constraints = { expr_a, expr_b, expr_c, expr_d },
     };
 
@@ -162,6 +208,7 @@ TEST(acir_format, test_schnorr_verify_pass)
         .pedersen_constraints = {},
         .compute_merkle_root_constraints = {},
         .block_constraints = {},
+        .recursion_constraints = {},
         .constraints = { poly_triple{
             .a = schnorr_constraint.result,
             .b = schnorr_constraint.result,
@@ -234,6 +281,7 @@ TEST(acir_format, test_schnorr_verify_small_range)
         .pedersen_constraints = {},
         .compute_merkle_root_constraints = {},
         .block_constraints = {},
+        .recursion_constraints = {},
         .constraints = { poly_triple{
             .a = schnorr_constraint.result,
             .b = schnorr_constraint.result,

cpp/src/barretenberg/dsl/acir_format/block_constraint.test.cpp

@@ -118,6 +118,7 @@ TEST(up_ram, TestBlockConstraint)
         .pedersen_constraints = {},
         .compute_merkle_root_constraints = {},
         .block_constraints = { block },
+        .recursion_constraints = {},
         .constraints = {},
     };
 

cpp/src/barretenberg/dsl/acir_format/ecdsa_secp256k1.test.cpp

@@ -99,6 +99,7 @@ TEST(ECDSASecp256k1, TestECDSAConstraintSucceed)
         .pedersen_constraints = {},
         .compute_merkle_root_constraints = {},
         .block_constraints = {},
+        .recursion_constraints = {},
         .constraints = {},
     };
 
@@ -136,6 +137,7 @@ TEST(ECDSASecp256k1, TestECDSACompilesForVerifier)
         .pedersen_constraints = {},
         .compute_merkle_root_constraints = {},
         .block_constraints = {},
+        .recursion_constraints = {},
         .constraints = {},
     };
     auto crs_factory = std::make_unique<barretenberg::srs::factories::CrsFactory>();
@@ -170,6 +172,7 @@ TEST(ECDSASecp256k1, TestECDSAConstraintFail)
         .pedersen_constraints = {},
         .compute_merkle_root_constraints = {},
         .block_constraints = {},
+        .recursion_constraints = {},
         .constraints = {},
     };